An emergent architecture is only half the story

I have heard many stories of architects supposedly working in agile environments that suggest these architects aren't maximising their agile potential. These stories often describe architects who believe that an evolving architecture is simply one that in which the emergent decisions are made in parallel with development;  they neglect continuously refactoring the architecture to ensure that the architecture always represents the best design for the product as it is understood at that time.

In my thesis I define an evolving architecture as:

"An evolving architecture consists of emergent decisions, in which the architecture emerges as architecture design decisions are made, and refactored decisions, in which previous architecture decisions (whether up-front or emergent) are changed." (page 41)

 Yes, it is difficult to refactor architecture – many definitions of architecture include the decisions that are hard to change (including my own) – but agile architects use a number of tactics to increase agility:

1. Keep designs simple 
2. Prove the design with code iteratively
3. Use good design practices
4. Delay decision making (aka "just enough anticipation")
5. Plan for options.

These tactics all make it easier to either continuously refactor the architecture or reduce the need to refactor (see my earlier post).

I wish to mention two important implications of an agile architecture here:

Firstly, agile development is all about delivering a value stream. The customer does not receive the whole product at once at the end of the project: the product is delivered in regular increments from as early as possible in development. This does not only let them deliver feedback to the team; it may also allow them to start using the product early and get income or value from it.  Even what the traditional software development lifecycle would call the maintenance phase is just another part of the value stream.

Consider: how would you design your architecture differently if you took into account what you know about the whole agile development cycle, including maintenance? If you knew that the product may change drastically over its lifetime in ways that cannot be imagined at the start? An agile architecture looks different from one that is not designed to evolve and be refactored.

For more on value streams, take a look at the "NoProjects" movement, such as Allen Kelly's recent "#NoProjects/#BeyondProjects" blog post and the related presentation. 

Secondly, note the term is value stream – not cost. Value to the customer may be at the expense of increased cost: that is, they may be prepared to suffer higher overall cost if it means more value in the form of earlier and/or more income.

Consider: would you be prepared to (and able to) continuously refactor the architecture, perhaps multiple times, as the product evolves, if it means delivering more value to the customer? In particular, this may mean initially designing a much reduced and simplified architecture that will be suitable for a much shorter period of time, but allows an initial release to be made  much sooner, and which will need to be refactored (perhaps completely) as the product matures.

See also Martin Fowler's recent blog post on Sacrificial Architecture.

Agile software development creates change

In my research, I defined agility as:

"...a software development team’s ability to create change, respond to change and learn from change so that it can better deliver value."

Someone commented to me that the "create change" part of this definition was unusual: what does it mean?

Agile software development actively encourages the customer to change the requirements of the product being built – to maximise the value they get from the product – with the development team "embracing" this change. As the customer sees (and hopefully uses) the evolving product, they become confident that they can refine and add new (high priority) requirements that maximise the value they get from the product, rather than  pulling new requirements from a bucket of preconceived ideas. Agile development gives the customer confidence that changes can be made successfully and easily. In contrast, traditional development methods discourage change by making change difficult.

 While not specifically talking about agile software development, the author Donald Gerwin defined creating change as: "...driving new change that would never have occurred were it not for the entity’s actions, e.g., creating more uncertainties for rivals, thus establishing a powerful competitive advantage." (*) 

While responding to change is the crucual part of the definition of agility, creating change and learning from change are also important.

(*) Manufacturing Flexibility: A Strategic Perspective, Donald Gerwin, 1993. (Sorry, not available for free.)

A theory of agile architecture in a nutshell (part 2)

This post presents the results of my PhD thesis in a nutshell (part 2 of 2).  The first part introduced my research and talked about agile architecture. This part discusses the theory of agile architecture.

The final version of my thesis can now be downloaded from the VUW library if you want to read it in its full academic glory.

A theory of agile architecture

I have called the findings of this research ‘a theory of agile architecture.’ The theory describes how agile teams determine how much up-front effort to put into architecture design. Most importantly, up-front architecture depends on context; this research describes six ‘forces’ that make up the context:

• F1. Requirements instability
• F2. Technical risk
• F3. Early value
• F4. Team culture
• F5. Customer agility
• F6. Experience

Teams address these forces with a number of ‘strategies’ which strategies teams use determine how much architecture planning the teams do:

• S1. Respond to change
• S2. Address risk
• S3. Emergent architecture
• S4. Big design up-front
• S5. Use frameworks and template architectures

These forces and strategies, and the relationships between them, make up the theory of agile architecture.

The agile architecture forces

The six forces are:

Requirements instability (F1) refers to the project having some or all of its requirements undefined at the start of development, or changing during development. This force is the main motivation for using agile development and for designing an agile architecture.

Technical risk (F2) is exposure to a potentially negative outcome because of uncertainty with the technology, the design or the system itself. Technical risk is often caused by architectural complexity brought about by demanding non-function requirements (or architecturally significant requirements) and is mitigated by more up-front design to reduce uncertainty.

Early value (F3) is the customer's need to derive commercial value (that is, cash flow) from the system being developed before it would otherwise be ready. To provide early value, a team reduces the planning horizon and spends less time on up-front architecture design. An early value system may also be in the form of a minimum viable product (MVP).

Team culture (F4) is a collaborative and people-focused culture based on trust: that is, an agile team culture. F4 increases the team's ability to communicate rapidly and hence reduces the time needed to respond to change.

Customer agility (F5) refers to the agility of the environment, such as the agility of the customer or the team's own organisation and any other stakeholders. As well as the team being agile, the environment must support the team's agility.

Experience (F6) refers to the team's architectural and technical experience. Experienced team members are more able to use tacit knowledge to make faster and fewer explicit decisions, and so can reduce up-front effort and respond to change faster.

F4, F5 and F6 all impact upon the team's agility, rendering it more or less able to design an agile architecture.

The agile architecture strategies

The five strategies are:

Respond to change (S1) is the key strategy for designing an agile architecture, and uses the tactics described in my previous post. The better the team is able to use these tactics, the more able it is to respond to change and reduce up-front effort.

Reduce risk (S2) is a strategy that uses more up-front design to mitigate technical risk as early as possible. S2 is in tension with S1's tactic of delaying decisions: teams have to balance mitigating risk with the architecture's ability to respond to change.

Emergent architecture (S3) is a strategy that results in as few up-front decisions as possible – which typically may be simply selecting the technology stack and the top level architectural styles and patterns. S3 is a strategy that takes S1 to the extreme, with all decisions delayed, whether or not the requirements are likely to change. S3 does not mitigate any risk up-front, and is therefore not suitable for complex systems with demanding ASRs.

Big design up-front (S4) is a strategy in which all (or most) architecture decisions are made up-front. S4 is most likely used by teams that have a non-agile customer (or manager) who requires the team to complete the design before development starts. S4 reduces the team's ability to respond to change.

Use Frameworks and template architectures (S5) reduces the effort required for architectural design by providing 'precooked' architectural solutions in the form of frameworks, templates, reference architectures and standard off-the-shelf libraries and plug-ins. Frameworks reduce complexity and hence risk, and make it easier to change subsidiary architectural decisions. This is particularly important for agile development because the reduced effort allows a team to respond to change more quickly and hence become more agile.

Relationships between the forces and strategies

Respond to change (S1): Requirements instability (F1) is a trigger for S1: if requirements are unstable, the team starts designing their architecture (and system) so that it can be changed. Team culture (F4), customer agility (F5) and experience (F6) are all success factors for S1: the more agile the team is, the better it will be at designing an agile architecture.

Implication: It is incorrect to say that the more unstable the requirements are, the less up-front design we need to do. Rather, the more agile the team is, the more able the team is to respond to change, and therefore the less up-front design it may need to do, and therefore the more unstable requirements it is able to manage.

Address risk (S2): Naturally, technical risk (F2) drives the need for S2 – largely an up-front activity, particularly for system-wide risk. Thus S2 is in tension with – and must balance – S1.

Implication: Because technical risk is caused by complexity, it is complexity and not size that determines how much effort teams put into up-front architecture design. (Although size can increase the impact of complexity.) 

Emergent architecture (S3): The need for early value (F3) motivates S3; being highly successful at S1 (and hence being highly agile with an agile-friendly customer) is crucial. Because addressing risk (S2) increases up-front effort, emergent architectures can only be designed in low-risk systems with low complexity.

Implication: An emergent architecture (or at least, less up-front architecture) may mean more architectural design later and more overall work over the life of the project – but for the start-up, that's acceptable – no, preferable – because once they have early adopters and cash flow they can pay for that extra work later. It's all about value and not necessarily cost.

Big up-front design (S4): An agile team would typically use S4 when one (or more) of the success factors are missing: agile team culture (F4), customer agility (F5) or architecture and technical experience (F6); most often this is when the customer is not agile, needing a traditional fixed price contract or needs budget approval for a particular set of functionality.

Standard frameworks and template architectures (S5) provide "precooked" architectures, reducing complexity, risk and effort, and allow the architecture to be more easily changed.

All of the relationships between the forces and strategies are summarised in this figure:

The agile architect

Architectural decisions are made either by members of the development teams or in agreement with members of the teams. No architects made decisions in isolation of the team. Decisions are either made and approved by the teams themselves – by consensus or by an architecture owner, are made by the teams and approved by external architects, or are researched by the teams and the decisions are made by the external architects. In the latter cases the external architects ensure the architectures of the teams' sub-systems met system-wide requirements or constraints that are perhaps not clearly visible to the teams themselves.

Having the whole team involved with the architecture design process ensures the whole team has an understanding of the architecture, its rationale and its importance. This understanding improves the team's ability to communicate and collaborate.

Other roles of the architect

Team members responsible for architecture decision making often perform many other roles in the team, usually informally. These roles include being the problem solver (uber developer), knowing the big picture, creating a shared mindset with the rest of the team, creating and enforcing development guidelines, and driving the development methodology. These roles show that the architect is an important member of the development team and contributes to team culture.

• Download thesis

A theory of agile architecture in a nutshell (part 1)

Today I deposited the final version of my thesis,  "Reconciling agility and architecture: a theory of agile architecture," with the university library. It is officially complete! 

This post presents the first part of a summary of my findings: my thesis in a nutshell. This part introduces my research and discusses what an agile architecture is. The second part, coming soon, discusses the theory of agile architecture itself.

If you want to read my thesis in its full academic glory, email me for a copy, or wait for it to become available at the Victoria University Library sometime next week.

The problem

The Agile Manifesto discourages too much planning ahead: if you spend too much time planning your architecture up-front, you're not agile because you're not delivering value or responding to change.

On the other hand, if you spend too little time planning your architecture up-front, your agility will probably also suffer because you'll end up spending all your time fixing architectural problems rather than delivering value.

So what is the right amount of up-front design effort, and what does it depend on?

My PhD research investigated how teams determine how much architecture to design up-front.

The research process

My research was empirical – the knowledge came from agile practitioners. These findings are all based on what agile teams actually do.

The research is qualitative –  the data is in the form of words rather than numbers. This means the findings are descriptive: the findings are an explanation of what is going on, rather than, for example, a recommendation that teams spend some percentage of their time planning.

And to really get into the methodology detail, the research strategy was Grounded Theory. Grounded Theory is a rigorous and methodical research methodology that abstracts the research participants' experiences into a theory –  hence "a theory of agile architecture." The theory is descriptive because it describes participants' experiences, rather than prescriptive –  it does not prescribe a methodology or a set of instructions or anything like that.

The findings

Defining agility

Before getting into the meat of the findings, we need to define agility: what does being agile mean? For this research, I defined agility as:

"...a software development team’s ability to create change, respond to change and learn from change so that it can better deliver value."

Of course, the most important part of this is being able to respond to change.

For the avoidance of doubt, this definition of agility is not necessarily the same thing as adhering to the philosophy of the Agile Manifesto, or following a particular agile methodology, or using a particular set of agile practices. You can be agile without following a particular set of agile practices, and just because you're using a particular methodology it doesn't make you agile. This research developed a theory of agile architecture – but first what is an agile architecture?

What is an agile architecture?

Simon Brown has an excellent definition of agile architecture: an architecture that provides agility by being flexible and adaptable. Hayim Makabee considers something similar with his adaptable design up-front. Philippe Kruchten noted that this definition of agile architecture is not necessarily the product of an agile process (that is, it could be created by a big up-front design); he compared this with a definition of agile architecture in which the architecture has emerged as part of the agile process, and noted this does not necessarily lead to a flexible architecture.

I define an agile architecture as comprising both of these definitions:

An agile architecture is first an architecture that is consistent with the definition of agility: the architecture can "respond to change" by either being easily modifiable or by being tolerant of change. Second, an agile architecture is the product of the team's agile process. 

Why does an agile architecture consist of both of these parts? Consider these five "tactics" that teams use to design an agile architecture:

1. Keep designs simple – the agile principle of simplicity: only designing for what is immediately required; no over-engineering and gold plating.
2. Prove the design with code iteratively – a tactic made possible through having design and development taking place simultaneously (i.e. not having the architecture design take place before development of features begins). Instead of testing a design through analysis, build it to see if it works.
3. Use good design practices – of course, every software design should use good design practices. That's what it means! But the benefit is amplified in agile, because we are explicitly welcoming change, and having a well designed architecture makes it easier to change.
4. Delay decision making – this tactic helps reduce rework because decisions are made when more information is known about the decision to be made, and hence more accurate decisions can be made. (Note this does not mean delay as long as possible! It simply means delay until the right time.) It also helps avoiding over-engineering by not forcing requirements to be defined prematurely.
5. Plan for options – don't close off possible future requirements. That is, you know things are likely going to change, so don't design your architecture into a corner. That's not to say spend extra time designing for those possible future requirements – that's over-engineering which contravenes the first tactic.

Keeping the design simple (#1), following good design practices (#3) and planning for options (#5) all affect the architecture itself: the use of these tactics may be determined later by inspecting the architecture. These three tactics are not dependent on an agile process. On the other hand, proving the architecture with code and delaying decisions are features of the agile process – the architecting  – and not so much the architecture itself.

The first three tactics each increase the architecture's modifiability, while the last two make it more tolerant of change (the architecture is less affected by changing requirements).

The agile architecture tactics also impact the up-front design effort: keeping the design simple, proving the design with code and delaying decisions all reduce the up-front effort.

The impact of the tactics are summarised in the following table.

Tactic	Impact on responsiveness to change	Affects architecture or architecting process?	Reduces up-front effort?
Keep designs simple	Increases modifiability	Architecture	Yes
Prove the design with code iteratively	Increases modifiability	Architecting process	Yes
Use good design practices	Increases modifiability	Architecture	No
Delay decision making	Increases tolerance of change	Architecting process	Yes
Plan for options	Increases tolerance of change	Architecture	No

That's the first half of the nutshell. Coming shortly: part 2, which describes the theory of agile architecture.