21 May 2017

New research - how does a software development team make the transition to agility?

I am excited to see Rashina Hoda has a new paper published at this week's ICSE conference.

Rashina is an academic at Auckland University in New Zealand; I was a colleague of Rashina's as she was finishing her PhD at Victoria University of Wellington just as I was starting out on mine in 2010.

At Auckland Rashina has continued her research into agile software development. This latest paper, co-authored with James Noble, is "Becoming Agile: A Grounded Theory of Agile Transitions in Practice"; the conference, ICSE (International Conference in Software Engineering) is the most highly ranked and prestigious software engineering research conference in the world, and is being held in Buenos Aires this week.

As the paper title suggests, her current research is focused on what actually happens when a software development team starts along the path to agility. Through interviewing a number of agile team members and using the Grounded Theory method for analysis, she has found that there are five dimensions to becoming agile:
  1. development practices
  2. team practices
  3. management approach
  4. reflective practices
  5. culture
Rashina found becoming agile is a continuous process of transforming the team across each of these dimensions as they become more experienced and proficient in agile development.

Development practices refer to the engineering and development practices that teams use, it includes using an iterative and incremental approach with frequent reviews and releases, technical practices such as pair programming and testing, as well as user stories,  daily stand-up meetings and artifacts. Teams that are new to agile methods focus on the iterative and incremental delivery model, while retaining many traditional practices. As they become more familiar with agile methods, they migrate to using more agile practices, while the most experienced agile teams are solely - and effectively - using recognised agile practices.

Team practices refer to project and task management practices - who drives practices such as requirements specification, prioritisation, estimation and collaboration with the customer. Teams new to agile are more manager-driven, teams transitioning to agile development are manager-assisted, and more experienced teams are team-driven.

The management approach (including project managers and team leads) is all about task assignment, problem solving and pushing collaboration with the business (requirements elicitation and clarification). A team new to agile relies on the manager to do this for the team, while in an agile team the manager is all about empowering the team to do these tasks for itself. In between, a transitioning team's manager helps the team adapt to an agile environment.

Reflective practices are all about reflection and learning. A new agile team is limited in its reflective practices; as teams become more agile the practices become more focused, until in experienced teams the practices are embedded in the team's regular activities.

The final dimension is culture. This includes organisational culture, team culture and individual traits that impact the work culture of the team. A new team's culture is likely to be hierarchical - information and decisions flow down to the team from senior management via subject manager experts and/or tech leads. As the team becomes more experienced, it evolves towards a more open and inclusive culture. The most agile teams can communicate and collaborate directly with customers, make their own decisions and control the work they do.

Rashina then describes the relationship between the dimensions - for example, how progress along one dimension affects progress along another -  which she calls hypotheses. There are four hypotheses:

H1: Transition along the development practices dimension is necessary for transition along the team practices and management approach dimensions. The iterative delivery model and technical practices such as TDD and pair programming are the first step along the path to agility; team-driven project management practices and team empowerment follow.

H2: Team practices and management practices reflect and adapt to each other. The success of team practices supports the success of management processes; the self-organising roles play an important part here. Often teams start with managers playing the roles (for example) of mentor, coordinator and translator; as the team becomes more experienced, the team members themselves take on these roles.

H3: Transition along the team practices and management practice dimensions is necessary for transition along the reflective practices dimension. Reflective practices are really only effective once the team is already a long way along the development practices, team practices and the management approach dimensions.

H4: Culture (organisational, team and individual) is an overarching dimension that affects the other four. For example, a highly agile organisational culture allows the team to improve its agility along the other four dimensions, while a poor agile organisational culture will stifle a team's agility.

Rashina suggests that teams can use this research to track their progress along the five dimensions and their overall agility, and hence motivate continuous improvement. Researchers can use these findings to explain variations across dimensions and between different teams.

I believe this research is very important because it tells teams what to focus on as they strive for agility. Whether the team leaps into agile development boots-and-all by (say) adopting Scrum or by transitioning to agile development gradually over time, it tells them they should focus on the core development practices, such as iterative delivery, stand-ups, test-driven development and prioritised user stories, before trying to become experts in team decision-making, self-organisation and team empowerment. After that the team is able to focus on improving its process - not simply by adopting agile processes that they haven't yet tried, but by critically examining their own process and looking for areas of improvement - such as where they can reduce the feedback cycle, where they can reduce time to address issues or implement new high-value features. The theory of constraints is likely to be very useful here. Note this does not mean that these dimensions should be implemented in sequence, one after the other! Rather it means that the effectiveness of the dependent dimensions is reduced if the team hasn't yet evolved in the earlier dimensions.

In parallel, the culture of the team, its members and the organisation needs to evolve to be more open and sharing to maximise the effectiveness of the team's agile practices.

Rashina (and James) have written an academic paper that is very important to the practical agile world.