Agile software development creates change

In my research, I defined agility as:

"...a software development team’s ability to create change, respond to change and learn from change so that it can better deliver value."

Someone commented to me that the "create change" part of this definition was unusual: what does it mean?

Agile software development actively encourages the customer to change the requirements of the product being built – to maximise the value they get from the product – with the development team "embracing" this change. As the customer sees (and hopefully uses) the evolving product, they become confident that they can refine and add new (high priority) requirements that maximise the value they get from the product, rather than  pulling new requirements from a bucket of preconceived ideas. Agile development gives the customer confidence that changes can be made successfully and easily. In contrast, traditional development methods discourage change by making change difficult.

 While not specifically talking about agile software development, the author Donald Gerwin defined creating change as: "...driving new change that would never have occurred were it not for the entity’s actions, e.g., creating more uncertainties for rivals, thus establishing a powerful competitive advantage." (*) 

While responding to change is the crucual part of the definition of agility, creating change and learning from change are also important.

(*) Manufacturing Flexibility: A Strategic Perspective, Donald Gerwin, 1993. (Sorry, not available for free.)

A theory of agile architecture in a nutshell (part 2)

This post presents the results of my PhD thesis in a nutshell (part 2 of 2).  The first part introduced my research and talked about agile architecture. This part discusses the theory of agile architecture.

The final version of my thesis can now be downloaded from the VUW library if you want to read it in its full academic glory.

A theory of agile architecture

I have called the findings of this research ‘a theory of agile architecture.’ The theory describes how agile teams determine how much up-front effort to put into architecture design. Most importantly, up-front architecture depends on context; this research describes six ‘forces’ that make up the context:

• F1. Requirements instability
• F2. Technical risk
• F3. Early value
• F4. Team culture
• F5. Customer agility
• F6. Experience

Teams address these forces with a number of ‘strategies’ which strategies teams use determine how much architecture planning the teams do:

• S1. Respond to change
• S2. Address risk
• S3. Emergent architecture
• S4. Big design up-front
• S5. Use frameworks and template architectures

These forces and strategies, and the relationships between them, make up the theory of agile architecture.

The agile architecture forces

The six forces are:

Requirements instability (F1) refers to the project having some or all of its requirements undefined at the start of development, or changing during development. This force is the main motivation for using agile development and for designing an agile architecture.

Technical risk (F2) is exposure to a potentially negative outcome because of uncertainty with the technology, the design or the system itself. Technical risk is often caused by architectural complexity brought about by demanding non-function requirements (or architecturally significant requirements) and is mitigated by more up-front design to reduce uncertainty.

Early value (F3) is the customer's need to derive commercial value (that is, cash flow) from the system being developed before it would otherwise be ready. To provide early value, a team reduces the planning horizon and spends less time on up-front architecture design. An early value system may also be in the form of a minimum viable product (MVP).

Team culture (F4) is a collaborative and people-focused culture based on trust: that is, an agile team culture. F4 increases the team's ability to communicate rapidly and hence reduces the time needed to respond to change.

Customer agility (F5) refers to the agility of the environment, such as the agility of the customer or the team's own organisation and any other stakeholders. As well as the team being agile, the environment must support the team's agility.

Experience (F6) refers to the team's architectural and technical experience. Experienced team members are more able to use tacit knowledge to make faster and fewer explicit decisions, and so can reduce up-front effort and respond to change faster.

F4, F5 and F6 all impact upon the team's agility, rendering it more or less able to design an agile architecture.

The agile architecture strategies

The five strategies are:

Respond to change (S1) is the key strategy for designing an agile architecture, and uses the tactics described in my previous post. The better the team is able to use these tactics, the more able it is to respond to change and reduce up-front effort.

Reduce risk (S2) is a strategy that uses more up-front design to mitigate technical risk as early as possible. S2 is in tension with S1's tactic of delaying decisions: teams have to balance mitigating risk with the architecture's ability to respond to change.

Emergent architecture (S3) is a strategy that results in as few up-front decisions as possible – which typically may be simply selecting the technology stack and the top level architectural styles and patterns. S3 is a strategy that takes S1 to the extreme, with all decisions delayed, whether or not the requirements are likely to change. S3 does not mitigate any risk up-front, and is therefore not suitable for complex systems with demanding ASRs.

Big design up-front (S4) is a strategy in which all (or most) architecture decisions are made up-front. S4 is most likely used by teams that have a non-agile customer (or manager) who requires the team to complete the design before development starts. S4 reduces the team's ability to respond to change.

Use Frameworks and template architectures (S5) reduces the effort required for architectural design by providing 'precooked' architectural solutions in the form of frameworks, templates, reference architectures and standard off-the-shelf libraries and plug-ins. Frameworks reduce complexity and hence risk, and make it easier to change subsidiary architectural decisions. This is particularly important for agile development because the reduced effort allows a team to respond to change more quickly and hence become more agile.

Relationships between the forces and strategies

Respond to change (S1): Requirements instability (F1) is a trigger for S1: if requirements are unstable, the team starts designing their architecture (and system) so that it can be changed. Team culture (F4), customer agility (F5) and experience (F6) are all success factors for S1: the more agile the team is, the better it will be at designing an agile architecture.

Implication: It is incorrect to say that the more unstable the requirements are, the less up-front design we need to do. Rather, the more agile the team is, the more able the team is to respond to change, and therefore the less up-front design it may need to do, and therefore the more unstable requirements it is able to manage.

Address risk (S2): Naturally, technical risk (F2) drives the need for S2 – largely an up-front activity, particularly for system-wide risk. Thus S2 is in tension with – and must balance – S1.

Implication: Because technical risk is caused by complexity, it is complexity and not size that determines how much effort teams put into up-front architecture design. (Although size can increase the impact of complexity.) 

Emergent architecture (S3): The need for early value (F3) motivates S3; being highly successful at S1 (and hence being highly agile with an agile-friendly customer) is crucial. Because addressing risk (S2) increases up-front effort, emergent architectures can only be designed in low-risk systems with low complexity.

Implication: An emergent architecture (or at least, less up-front architecture) may mean more architectural design later and more overall work over the life of the project – but for the start-up, that's acceptable – no, preferable – because once they have early adopters and cash flow they can pay for that extra work later. It's all about value and not necessarily cost.

Big up-front design (S4): An agile team would typically use S4 when one (or more) of the success factors are missing: agile team culture (F4), customer agility (F5) or architecture and technical experience (F6); most often this is when the customer is not agile, needing a traditional fixed price contract or needs budget approval for a particular set of functionality.

Standard frameworks and template architectures (S5) provide "precooked" architectures, reducing complexity, risk and effort, and allow the architecture to be more easily changed.

All of the relationships between the forces and strategies are summarised in this figure:

The agile architect

Architectural decisions are made either by members of the development teams or in agreement with members of the teams. No architects made decisions in isolation of the team. Decisions are either made and approved by the teams themselves – by consensus or by an architecture owner, are made by the teams and approved by external architects, or are researched by the teams and the decisions are made by the external architects. In the latter cases the external architects ensure the architectures of the teams' sub-systems met system-wide requirements or constraints that are perhaps not clearly visible to the teams themselves.

Having the whole team involved with the architecture design process ensures the whole team has an understanding of the architecture, its rationale and its importance. This understanding improves the team's ability to communicate and collaborate.

Other roles of the architect

Team members responsible for architecture decision making often perform many other roles in the team, usually informally. These roles include being the problem solver (uber developer), knowing the big picture, creating a shared mindset with the rest of the team, creating and enforcing development guidelines, and driving the development methodology. These roles show that the architect is an important member of the development team and contributes to team culture.

• Download thesis

A theory of agile architecture in a nutshell (part 1)

Today I deposited the final version of my thesis,  "Reconciling agility and architecture: a theory of agile architecture," with the university library. It is officially complete! 

This post presents the first part of a summary of my findings: my thesis in a nutshell. This part introduces my research and discusses what an agile architecture is. The second part, coming soon, discusses the theory of agile architecture itself.

If you want to read my thesis in its full academic glory, email me for a copy, or wait for it to become available at the Victoria University Library sometime next week.

The problem

The Agile Manifesto discourages too much planning ahead: if you spend too much time planning your architecture up-front, you're not agile because you're not delivering value or responding to change.

On the other hand, if you spend too little time planning your architecture up-front, your agility will probably also suffer because you'll end up spending all your time fixing architectural problems rather than delivering value.

So what is the right amount of up-front design effort, and what does it depend on?

My PhD research investigated how teams determine how much architecture to design up-front.

The research process

My research was empirical – the knowledge came from agile practitioners. These findings are all based on what agile teams actually do.

The research is qualitative –  the data is in the form of words rather than numbers. This means the findings are descriptive: the findings are an explanation of what is going on, rather than, for example, a recommendation that teams spend some percentage of their time planning.

And to really get into the methodology detail, the research strategy was Grounded Theory. Grounded Theory is a rigorous and methodical research methodology that abstracts the research participants' experiences into a theory –  hence "a theory of agile architecture." The theory is descriptive because it describes participants' experiences, rather than prescriptive –  it does not prescribe a methodology or a set of instructions or anything like that.

The findings

Defining agility

Before getting into the meat of the findings, we need to define agility: what does being agile mean? For this research, I defined agility as:

"...a software development team’s ability to create change, respond to change and learn from change so that it can better deliver value."

Of course, the most important part of this is being able to respond to change.

For the avoidance of doubt, this definition of agility is not necessarily the same thing as adhering to the philosophy of the Agile Manifesto, or following a particular agile methodology, or using a particular set of agile practices. You can be agile without following a particular set of agile practices, and just because you're using a particular methodology it doesn't make you agile. This research developed a theory of agile architecture – but first what is an agile architecture?

What is an agile architecture?

Simon Brown has an excellent definition of agile architecture: an architecture that provides agility by being flexible and adaptable. Hayim Makabee considers something similar with his adaptable design up-front. Philippe Kruchten noted that this definition of agile architecture is not necessarily the product of an agile process (that is, it could be created by a big up-front design); he compared this with a definition of agile architecture in which the architecture has emerged as part of the agile process, and noted this does not necessarily lead to a flexible architecture.

I define an agile architecture as comprising both of these definitions:

An agile architecture is first an architecture that is consistent with the definition of agility: the architecture can "respond to change" by either being easily modifiable or by being tolerant of change. Second, an agile architecture is the product of the team's agile process. 

Why does an agile architecture consist of both of these parts? Consider these five "tactics" that teams use to design an agile architecture:

1. Keep designs simple – the agile principle of simplicity: only designing for what is immediately required; no over-engineering and gold plating.
2. Prove the design with code iteratively – a tactic made possible through having design and development taking place simultaneously (i.e. not having the architecture design take place before development of features begins). Instead of testing a design through analysis, build it to see if it works.
3. Use good design practices – of course, every software design should use good design practices. That's what it means! But the benefit is amplified in agile, because we are explicitly welcoming change, and having a well designed architecture makes it easier to change.
4. Delay decision making – this tactic helps reduce rework because decisions are made when more information is known about the decision to be made, and hence more accurate decisions can be made. (Note this does not mean delay as long as possible! It simply means delay until the right time.) It also helps avoiding over-engineering by not forcing requirements to be defined prematurely.
5. Plan for options – don't close off possible future requirements. That is, you know things are likely going to change, so don't design your architecture into a corner. That's not to say spend extra time designing for those possible future requirements – that's over-engineering which contravenes the first tactic.

Keeping the design simple (#1), following good design practices (#3) and planning for options (#5) all affect the architecture itself: the use of these tactics may be determined later by inspecting the architecture. These three tactics are not dependent on an agile process. On the other hand, proving the architecture with code and delaying decisions are features of the agile process – the architecting  – and not so much the architecture itself.

The first three tactics each increase the architecture's modifiability, while the last two make it more tolerant of change (the architecture is less affected by changing requirements).

The agile architecture tactics also impact the up-front design effort: keeping the design simple, proving the design with code and delaying decisions all reduce the up-front effort.

The impact of the tactics are summarised in the following table.

Tactic	Impact on responsiveness to change	Affects architecture or architecting process?	Reduces up-front effort?
Keep designs simple	Increases modifiability	Architecture	Yes
Prove the design with code iteratively	Increases modifiability	Architecting process	Yes
Use good design practices	Increases modifiability	Architecture	No
Delay decision making	Increases tolerance of change	Architecting process	Yes
Plan for options	Increases tolerance of change	Architecture	No

That's the first half of the nutshell. Coming shortly: part 2, which describes the theory of agile architecture.

The effects of complexity and value on up-front architecture planning

Complexity versus size... and value versus cost.

What are the effects of each of these on up-front planning in agile development? I am presenting a paper on these results from my research at the XP2013 conference in Vienna next week.

Abstract:

A key feature of agile software development is its prioritisation of responding to changing requirements over planning ahead. If an agile development team spends too much time planning and designing architecture then responding to change will be extremely costly, while not doing enough architectural design puts the project at risk of failure. Striking the balance depends heavily on the context of the system being built, the environment and the development teams. This Grounded Theory research into how much architecture agile teams design up-front has identified system complexity as an important factor in determining how much planning a team does up-front, while system size, although related to complexity, has a much less direct impact. Furthermore, when determining how much design to do up-front, value to the customer can be a more important factor than overall development cost. Understanding these factors can help agile teams to determine how much up-front planning is appropriate for the systems they develop.

Submitted version

The final publication will be available at link.springer.com

The effects of size and complexity on architecture planning

Shaw and Garland opened their seminal book "Software Architecture: Perspectives on an Emerging Discipline" with this sentence:

"As the size and complexity of software systems increase, the design and specification of overall system structure become more significant issues than the choice of algorithms and data structures of computation".

That is, the bigger and more complex the system, the greater the proportion of time that needs to be spent nutting out the architecture.

Barry Boehm used his COCOMO II model to determine the effects of size of architectural effort in "Architected agile solutions for Software-Reliant Systems": the more time you spend architecting, the more time you add to the overall schedule (naturally!); conversely, the less time you spend architecting, the more time you have to spend on rework and fixing up architectural problems. Add those two curves together and you get a U-shape, with a sweet spot at the minimum (see figure 5 on page 7). The cost of rework is higher with larger systems, so your sweet spot moves to the right - that is, more architectural planning.

But do we see this relationship in modern systems being built using agile methods? Not necessarily, it appears. Architecture is changing: for teams using modern development frameworks (such as .NET, Hibernate, Ruby on Rails), architectural decisions aren't nearly as intractable as they were in the old days -- they can be changed more easily during development.  In fact, some decisions that used to be considered architectural may now be considered design decisions.

Frameworks do this by providing standard solutions to problems and therefore greatly reducing the complexity (and risk) of a system.  In effect, they are decoupling the relationship between complexity and size: a system may be large but implemented entirely using predefined components and libraries, therefore greatly reducing the complexity and architectural effort required. On the other hand, a small system may not be easily built using predefined framework components and libraries, and needs significant architectural effort.

So while size and complexity are related, it is not an linear relationship and it is complexity that is the true driver of architectural effort, not size.

And what's the corollary of this? Perhaps it is modern frameworks are an important enabler for reducing the up-front effort required in agile software development.

The relationship between requirements stability and up-front planning

Is there a relationship between how stable a system's requirements are and how much up-front planning teams do?

If agile means delaying architectural effort till the last sensible moment so you can better respond to change and have less rework, then perhaps we can assume that the more stable your requirements are the more up-front planning you can do, so you can get the benefits of an complete architecture design that has been better thought through.

But is this the case?

It doesn't seem to be. A highly agile team will still try to minimise its up-front effort, defining requirements later and making architectural decisions as close as possible to the time that they're needed. Even when they have fairly stable requirements, such as when doing a functional migration.

This isn't because teams are looking for an excuse to avoid up-front effort, but rather it allows them to focus on the high-value functionality rather than get bogged down in the detail and minutiae of defining a complete set of requirements at the beginning and losing sight of the big picture.

And anyway, even if we have fairly stable requirements, who knows exactly what's going to change in advance, and hence what can be left till later?

Modeling or experimentation?

Scott Ambler, of Ambysoft and Agile Modeling fame, has just had an article published in Dr Dobbs about disciplined agile architecture.

(This material is apparently based on his and Mark Lines' new book Disciplined Agile Delivery: A practitioner's Guide to Agile Software Delivery in the Enterprise, though I haven't read it yet.)

In this article Scott presents a number of choices that need to be made when determining an initial technical strategy about architecture in agile development. The choices are the level of model detail (detailed end-to-end, detailed interface, high-level overview, or none), the view types (technology, business architecture, or user interface), and the modeling strategy (formal modeling sessions, informal modeling sessions, single candidate architecture, multiple candidate architecture).

Interestingly, Scott doesn't mention experimentation at all either in this article or in related articles. In my research, experiments – whether they’re prototypes/proofs of concept, spikes or A/B tests – are far more popular up-front activities than analysis and modeling.

Participants in my research frequently talk about problems that can't be solved through analysis, particularly when using new or unknown technologies where analysis or modeling can't be used, when technologies don’t work as expected, when there are unexpected interactions between technologies, and when there is risk that needs to be mitigated.

Without presenting a full analysis of the research results, here are half a dozen quotes from participants (referred to by their code numbers – P1 to P30) that explain their views on analysis and modeling versus experimentation.

P7 (business analyst) was in a team that was working with unfamiliar technology and building unique systems:

“In the kind of work we do, which is kind-of more cutting edge, more complicated business problems and more complicated technical environments, then it’s just natural that everybody’s feeling their way around a bit, and so [up-front analysis] is very difficult ... yeah, so it’s the process of learning and understanding and realising ... you have to get your feet wet, your hands dirty.”

and similarly P6 (development manager):

“Sometimes you have no choice but to go and write a few tests, to write a test program to explore how something works in practice because if you have a new technology you might not know.” 

P19 (development manager):

“These are problems that we are going to have to [solve]... not because we didn’t think about it, but because they would only become evident once you start digging into the code ... you really have to do it to catch the problems.” 

P10 (agile coach) said that building the system was the only way to truly tell if the system could meet its required load:

“You can estimate till the cows come home before actually trying it out in the flesh, and when you try it out in the flesh then you learn uh-oh, all sorts of things are going on ... you can’t afford to try and do everything as a thought experiment because the criticality of the system is such that you really need to know if you have enough headroom to support peak load.” 

P4 (director of architecture) was one of many who talked about the role of experimentation in risk management:

“We wouldn’t try to work out what the risk would be when we could actually try it and see.” 

Finally, P17 (team manager) and his team built a system whose load was higher than expected. P17 noted that even if they had done more analysis up-front and had a better understanding of the load on the system they wouldn’t have done anything differently:

“The only thing analysis might have helped with was some of the performance stuff. I’m not sure we would have made that many different decisions ... The only thing it would have given us is a bit more forewarning [of performance problems].” 

So that’s just a small random selection of what my research participants have been telling me about experimentation, but it’s very clear that experimentation plays a very important role in the up-front architecture planning and design activities.