New research - how does a software development team make the transition to agility?

I am excited to see Rashina Hoda has a new paper published at this week's ICSE conference.

Rashina is an academic at Auckland University in New Zealand; I was a colleague of Rashina's as she was finishing her PhD at Victoria University of Wellington just as I was starting out on mine in 2010.

At Auckland Rashina has continued her research into agile software development. This latest paper, co-authored with James Noble, is "Becoming Agile: A Grounded Theory of Agile Transitions in Practice"; the conference, ICSE (International Conference in Software Engineering) is the most highly ranked and prestigious software engineering research conference in the world, and is being held in Buenos Aires this week.

As the paper title suggests, her current research is focused on what actually happens when a software development team starts along the path to agility. Through interviewing a number of agile team members and using the Grounded Theory method for analysis, she has found that there are five dimensions to becoming agile:

1. development practices
2. team practices
3. management approach
4. reflective practices
5. culture

Rashina found becoming agile is a continuous process of transforming the team across each of these dimensions as they become more experienced and proficient in agile development.

Development practices refer to the engineering and development practices that teams use, it includes using an iterative and incremental approach with frequent reviews and releases, technical practices such as pair programming and testing, as well as user stories,  daily stand-up meetings and artifacts. Teams that are new to agile methods focus on the iterative and incremental delivery model, while retaining many traditional practices. As they become more familiar with agile methods, they migrate to using more agile practices, while the most experienced agile teams are solely - and effectively - using recognised agile practices.

Team practices refer to project and task management practices - who drives practices such as requirements specification, prioritisation, estimation and collaboration with the customer. Teams new to agile are more manager-driven, teams transitioning to agile development are manager-assisted, and more experienced teams are team-driven.

The management approach (including project managers and team leads) is all about task assignment, problem solving and pushing collaboration with the business (requirements elicitation and clarification). A team new to agile relies on the manager to do this for the team, while in an agile team the manager is all about empowering the team to do these tasks for itself. In between, a transitioning team's manager helps the team adapt to an agile environment.

Reflective practices are all about reflection and learning. A new agile team is limited in its reflective practices; as teams become more agile the practices become more focused, until in experienced teams the practices are embedded in the team's regular activities.

The final dimension is culture. This includes organisational culture, team culture and individual traits that impact the work culture of the team. A new team's culture is likely to be hierarchical - information and decisions flow down to the team from senior management via subject manager experts and/or tech leads. As the team becomes more experienced, it evolves towards a more open and inclusive culture. The most agile teams can communicate and collaborate directly with customers, make their own decisions and control the work they do.

Rashina then describes the relationship between the dimensions - for example, how progress along one dimension affects progress along another -  which she calls hypotheses. There are four hypotheses:

H1: Transition along the development practices dimension is necessary for transition along the team practices and management approach dimensions. The iterative delivery model and technical practices such as TDD and pair programming are the first step along the path to agility; team-driven project management practices and team empowerment follow.

H2: Team practices and management practices reflect and adapt to each other. The success of team practices supports the success of management processes; the self-organising roles play an important part here. Often teams start with managers playing the roles (for example) of mentor, coordinator and translator; as the team becomes more experienced, the team members themselves take on these roles.

H3: Transition along the team practices and management practice dimensions is necessary for transition along the reflective practices dimension. Reflective practices are really only effective once the team is already a long way along the development practices, team practices and the management approach dimensions.

H4: Culture (organisational, team and individual) is an overarching dimension that affects the other four. For example, a highly agile organisational culture allows the team to improve its agility along the other four dimensions, while a poor agile organisational culture will stifle a team's agility.

Rashina suggests that teams can use this research to track their progress along the five dimensions and their overall agility, and hence motivate continuous improvement. Researchers can use these findings to explain variations across dimensions and between different teams.

I believe this research is very important because it tells teams what to focus on as they strive for agility. Whether the team leaps into agile development boots-and-all by (say) adopting Scrum or by transitioning to agile development gradually over time, it tells them they should focus on the core development practices, such as iterative delivery, stand-ups, test-driven development and prioritised user stories, before trying to become experts in team decision-making, self-organisation and team empowerment. After that the team is able to focus on improving its process - not simply by adopting agile processes that they haven't yet tried, but by critically examining their own process and looking for areas of improvement - such as where they can reduce the feedback cycle, where they can reduce time to address issues or implement new high-value features. The theory of constraints is likely to be very useful here. Note this does not mean that these dimensions should be implemented in sequence, one after the other! Rather it means that the effectiveness of the dependent dimensions is reduced if the team hasn't yet evolved in the earlier dimensions.

In parallel, the culture of the team, its members and the organisation needs to evolve to be more open and sharing to maximise the effectiveness of the team's agile practices.

Rashina (and James) have written an academic paper that is very important to the practical agile world.

• Download Rashina and James's paper 

A theory of agile architecture in a nutshell (part 2)

This post presents the results of my PhD thesis in a nutshell (part 2 of 2).  The first part introduced my research and talked about agile architecture. This part discusses the theory of agile architecture.

The final version of my thesis can now be downloaded from the VUW library if you want to read it in its full academic glory.

A theory of agile architecture

I have called the findings of this research ‘a theory of agile architecture.’ The theory describes how agile teams determine how much up-front effort to put into architecture design. Most importantly, up-front architecture depends on context; this research describes six ‘forces’ that make up the context:

• F1. Requirements instability
• F2. Technical risk
• F3. Early value
• F4. Team culture
• F5. Customer agility
• F6. Experience

Teams address these forces with a number of ‘strategies’ which strategies teams use determine how much architecture planning the teams do:

• S1. Respond to change
• S2. Address risk
• S3. Emergent architecture
• S4. Big design up-front
• S5. Use frameworks and template architectures

These forces and strategies, and the relationships between them, make up the theory of agile architecture.

The agile architecture forces

The six forces are:

Requirements instability (F1) refers to the project having some or all of its requirements undefined at the start of development, or changing during development. This force is the main motivation for using agile development and for designing an agile architecture.

Technical risk (F2) is exposure to a potentially negative outcome because of uncertainty with the technology, the design or the system itself. Technical risk is often caused by architectural complexity brought about by demanding non-function requirements (or architecturally significant requirements) and is mitigated by more up-front design to reduce uncertainty.

Early value (F3) is the customer's need to derive commercial value (that is, cash flow) from the system being developed before it would otherwise be ready. To provide early value, a team reduces the planning horizon and spends less time on up-front architecture design. An early value system may also be in the form of a minimum viable product (MVP).

Team culture (F4) is a collaborative and people-focused culture based on trust: that is, an agile team culture. F4 increases the team's ability to communicate rapidly and hence reduces the time needed to respond to change.

Customer agility (F5) refers to the agility of the environment, such as the agility of the customer or the team's own organisation and any other stakeholders. As well as the team being agile, the environment must support the team's agility.

Experience (F6) refers to the team's architectural and technical experience. Experienced team members are more able to use tacit knowledge to make faster and fewer explicit decisions, and so can reduce up-front effort and respond to change faster.

F4, F5 and F6 all impact upon the team's agility, rendering it more or less able to design an agile architecture.

The agile architecture strategies

The five strategies are:

Respond to change (S1) is the key strategy for designing an agile architecture, and uses the tactics described in my previous post. The better the team is able to use these tactics, the more able it is to respond to change and reduce up-front effort.

Reduce risk (S2) is a strategy that uses more up-front design to mitigate technical risk as early as possible. S2 is in tension with S1's tactic of delaying decisions: teams have to balance mitigating risk with the architecture's ability to respond to change.

Emergent architecture (S3) is a strategy that results in as few up-front decisions as possible – which typically may be simply selecting the technology stack and the top level architectural styles and patterns. S3 is a strategy that takes S1 to the extreme, with all decisions delayed, whether or not the requirements are likely to change. S3 does not mitigate any risk up-front, and is therefore not suitable for complex systems with demanding ASRs.

Big design up-front (S4) is a strategy in which all (or most) architecture decisions are made up-front. S4 is most likely used by teams that have a non-agile customer (or manager) who requires the team to complete the design before development starts. S4 reduces the team's ability to respond to change.

Use Frameworks and template architectures (S5) reduces the effort required for architectural design by providing 'precooked' architectural solutions in the form of frameworks, templates, reference architectures and standard off-the-shelf libraries and plug-ins. Frameworks reduce complexity and hence risk, and make it easier to change subsidiary architectural decisions. This is particularly important for agile development because the reduced effort allows a team to respond to change more quickly and hence become more agile.

Relationships between the forces and strategies

Respond to change (S1): Requirements instability (F1) is a trigger for S1: if requirements are unstable, the team starts designing their architecture (and system) so that it can be changed. Team culture (F4), customer agility (F5) and experience (F6) are all success factors for S1: the more agile the team is, the better it will be at designing an agile architecture.

Implication: It is incorrect to say that the more unstable the requirements are, the less up-front design we need to do. Rather, the more agile the team is, the more able the team is to respond to change, and therefore the less up-front design it may need to do, and therefore the more unstable requirements it is able to manage.

Address risk (S2): Naturally, technical risk (F2) drives the need for S2 – largely an up-front activity, particularly for system-wide risk. Thus S2 is in tension with – and must balance – S1.

Implication: Because technical risk is caused by complexity, it is complexity and not size that determines how much effort teams put into up-front architecture design. (Although size can increase the impact of complexity.) 

Emergent architecture (S3): The need for early value (F3) motivates S3; being highly successful at S1 (and hence being highly agile with an agile-friendly customer) is crucial. Because addressing risk (S2) increases up-front effort, emergent architectures can only be designed in low-risk systems with low complexity.

Implication: An emergent architecture (or at least, less up-front architecture) may mean more architectural design later and more overall work over the life of the project – but for the start-up, that's acceptable – no, preferable – because once they have early adopters and cash flow they can pay for that extra work later. It's all about value and not necessarily cost.

Big up-front design (S4): An agile team would typically use S4 when one (or more) of the success factors are missing: agile team culture (F4), customer agility (F5) or architecture and technical experience (F6); most often this is when the customer is not agile, needing a traditional fixed price contract or needs budget approval for a particular set of functionality.

Standard frameworks and template architectures (S5) provide "precooked" architectures, reducing complexity, risk and effort, and allow the architecture to be more easily changed.

All of the relationships between the forces and strategies are summarised in this figure:

The agile architect

Architectural decisions are made either by members of the development teams or in agreement with members of the teams. No architects made decisions in isolation of the team. Decisions are either made and approved by the teams themselves – by consensus or by an architecture owner, are made by the teams and approved by external architects, or are researched by the teams and the decisions are made by the external architects. In the latter cases the external architects ensure the architectures of the teams' sub-systems met system-wide requirements or constraints that are perhaps not clearly visible to the teams themselves.

Having the whole team involved with the architecture design process ensures the whole team has an understanding of the architecture, its rationale and its importance. This understanding improves the team's ability to communicate and collaborate.

Other roles of the architect

Team members responsible for architecture decision making often perform many other roles in the team, usually informally. These roles include being the problem solver (uber developer), knowing the big picture, creating a shared mindset with the rest of the team, creating and enforcing development guidelines, and driving the development methodology. These roles show that the architect is an important member of the development team and contributes to team culture.

• Download thesis

A theory of agile architecture in a nutshell (part 1)

Today I deposited the final version of my thesis,  "Reconciling agility and architecture: a theory of agile architecture," with the university library. It is officially complete! 

This post presents the first part of a summary of my findings: my thesis in a nutshell. This part introduces my research and discusses what an agile architecture is. The second part, coming soon, discusses the theory of agile architecture itself.

If you want to read my thesis in its full academic glory, email me for a copy, or wait for it to become available at the Victoria University Library sometime next week.

The problem

The Agile Manifesto discourages too much planning ahead: if you spend too much time planning your architecture up-front, you're not agile because you're not delivering value or responding to change.

On the other hand, if you spend too little time planning your architecture up-front, your agility will probably also suffer because you'll end up spending all your time fixing architectural problems rather than delivering value.

So what is the right amount of up-front design effort, and what does it depend on?

My PhD research investigated how teams determine how much architecture to design up-front.

The research process

My research was empirical – the knowledge came from agile practitioners. These findings are all based on what agile teams actually do.

The research is qualitative –  the data is in the form of words rather than numbers. This means the findings are descriptive: the findings are an explanation of what is going on, rather than, for example, a recommendation that teams spend some percentage of their time planning.

And to really get into the methodology detail, the research strategy was Grounded Theory. Grounded Theory is a rigorous and methodical research methodology that abstracts the research participants' experiences into a theory –  hence "a theory of agile architecture." The theory is descriptive because it describes participants' experiences, rather than prescriptive –  it does not prescribe a methodology or a set of instructions or anything like that.

The findings

Defining agility

Before getting into the meat of the findings, we need to define agility: what does being agile mean? For this research, I defined agility as:

"...a software development team’s ability to create change, respond to change and learn from change so that it can better deliver value."

Of course, the most important part of this is being able to respond to change.

For the avoidance of doubt, this definition of agility is not necessarily the same thing as adhering to the philosophy of the Agile Manifesto, or following a particular agile methodology, or using a particular set of agile practices. You can be agile without following a particular set of agile practices, and just because you're using a particular methodology it doesn't make you agile. This research developed a theory of agile architecture – but first what is an agile architecture?

What is an agile architecture?

Simon Brown has an excellent definition of agile architecture: an architecture that provides agility by being flexible and adaptable. Hayim Makabee considers something similar with his adaptable design up-front. Philippe Kruchten noted that this definition of agile architecture is not necessarily the product of an agile process (that is, it could be created by a big up-front design); he compared this with a definition of agile architecture in which the architecture has emerged as part of the agile process, and noted this does not necessarily lead to a flexible architecture.

I define an agile architecture as comprising both of these definitions:

An agile architecture is first an architecture that is consistent with the definition of agility: the architecture can "respond to change" by either being easily modifiable or by being tolerant of change. Second, an agile architecture is the product of the team's agile process. 

Why does an agile architecture consist of both of these parts? Consider these five "tactics" that teams use to design an agile architecture:

1. Keep designs simple – the agile principle of simplicity: only designing for what is immediately required; no over-engineering and gold plating.
2. Prove the design with code iteratively – a tactic made possible through having design and development taking place simultaneously (i.e. not having the architecture design take place before development of features begins). Instead of testing a design through analysis, build it to see if it works.
3. Use good design practices – of course, every software design should use good design practices. That's what it means! But the benefit is amplified in agile, because we are explicitly welcoming change, and having a well designed architecture makes it easier to change.
4. Delay decision making – this tactic helps reduce rework because decisions are made when more information is known about the decision to be made, and hence more accurate decisions can be made. (Note this does not mean delay as long as possible! It simply means delay until the right time.) It also helps avoiding over-engineering by not forcing requirements to be defined prematurely.
5. Plan for options – don't close off possible future requirements. That is, you know things are likely going to change, so don't design your architecture into a corner. That's not to say spend extra time designing for those possible future requirements – that's over-engineering which contravenes the first tactic.

Keeping the design simple (#1), following good design practices (#3) and planning for options (#5) all affect the architecture itself: the use of these tactics may be determined later by inspecting the architecture. These three tactics are not dependent on an agile process. On the other hand, proving the architecture with code and delaying decisions are features of the agile process – the architecting  – and not so much the architecture itself.

The first three tactics each increase the architecture's modifiability, while the last two make it more tolerant of change (the architecture is less affected by changing requirements).

The agile architecture tactics also impact the up-front design effort: keeping the design simple, proving the design with code and delaying decisions all reduce the up-front effort.

The impact of the tactics are summarised in the following table.

Tactic	Impact on responsiveness to change	Affects architecture or architecting process?	Reduces up-front effort?
Keep designs simple	Increases modifiability	Architecture	Yes
Prove the design with code iteratively	Increases modifiability	Architecting process	Yes
Use good design practices	Increases modifiability	Architecture	No
Delay decision making	Increases tolerance of change	Architecting process	Yes
Plan for options	Increases tolerance of change	Architecture	No

That's the first half of the nutshell. Coming shortly: part 2, which describes the theory of agile architecture.