25 June 2012

Why qualitative research?

Why is my research qualitative, using data gathered from interviews, rather than quantitative, using data gathered from (say) measuring teams or software applications?

Here is the long answer:

When it comes to up-front planning in agile software development, there is general agreement that the 'big up-front design' (BUFD) doesn't work (that's the whole point of agile!), and that the 'no up-front design' is not the answer either. A better solution lies somewhere in between - the 'just enough up-front design'. Just enough up-front design can be implemented in a number of ways, such as the architecture spike, the walking skeleton, George Fairbanks' excellent risk-based approach, and Scott Ambler's agile modelling.

However many of these methods do not help in determining which architectural requirements should be designed up-front, how architectural design should be performed, and how to validate architectural features. There are very few studies into what agile practitioners actually do and what actually works.

I have summarised those unknowns into the one question “How much architecture?”. And the difficulty is there is no single correct answer. (In fact, any particular system may not even have a single correct architecture [Booch], [Fairbanks]). “How much" depends on context – which includes factors such as size, criticality, business model, stable architecture, team distribution, governance, rate of change and age of system [Abramhamsson, Babar, Kruchten]. It goes further than this however: the development team themselves are part of that context; any two  architects are likely to produce different architectures for the same problem with the same boundaries, for “software architecture is a result of technical, business and social influences” – and this includes “the background and experience of the architects”.

In other words, architecture depends not only on the technical and business constraints of a system, but also on the experience of the architect and of the development team, on their judgement and abilities, and on what they believe to be the correct architecture.

And of course, agile development adds another complicating factor: the requirements are not known before starting, and therefore the minimum amount of architectural effort cannot be rationally determined in advance.

Thus it doesn't appear likely that there is some simple formula that a development team can apply to determine when architecture should be designed up-front, and when it should be left to emerge during development. So, this research is taking a different approach – exploring how agile practitioners determine how much architecture is planned up-front – the methods they use, what factors they consider, the choices they make. That sort of research is qualitative, with data obtained by talking to people in interviews. And as I wrote earlier, the outcome of this research will be “an explanation, a discourse, that explains how agile development teams deal with up-front design. It will be a story that teams can use to put their own situations into context, to reassure them that they are thinking about all the right things when planning – or to give them a few ideas of things they should be thinking about.”

And the short answer?

I think this sums it up fairly well:
“...Software engineering is full of why and how questions for which numbers and statistics are of little help. For example, if you've managed a software team, you've probably asked yourself a number of questions. Why won't my developers write unit tests? Why do users keep filling out this form incorrectly? Why are some of my developers 10 times as productive as the others? These questions can't be answered with numbers, but they can with the careful application of qualitative methods.”  [Andre Ko, in “Understanding Software Engineering Through Qualitative Methods” from  Making Software: What Really Works, and Why We Believe It.] 
And the even shorter answer?

Doing a PhD is a very lonely task, so going out and talking to agilists helps me to keep a grip on reality!

(Footnote: some of this material is based on a paper I presented at Agile India earlier this year. Also I'm sorry for the links to articles that aren't free!)

No comments:

Post a Comment