On project management methods

Yesterday, I learned about the upcoming Analogy Fest (which I find very interesting), and I also attended a brief overview of a project management method that includes something called the Seven Keys to Success. This method was claimed to be compatible with Extreme Programming (XP) and other agile methods for software development, but that wasn’t quite the impression I got during the overview. So I started to think a little about analogies for software development.

The Seven Keys to Success is apparently about increasing visibility of project status to stakeholders (customers, managers, et cetera). I associated this with Feature Driven Development, where project status is shown using colors and progress bars – either at the scale of single projects, with icons representing subprojects or subsystems, or at the scale of several projects, with icons for each project.

As for the method itself, it had four or five phases (I’m translating from Swedish, so these might not be the original terms): feasibility study, planning, implementation, and completion (if there was five, I can’t remember the fifth). The implementation phase wasn’t defined by the method, and could, according to the man who gave the overview, be executed using XP or virtually any other method.

The outcome of the planning phase is an overall plan for the project, with estimates for the identified parts of the system to be built. If this plan is coarse, I could see XP fitting into this picture – but otherwise, I feel there would be lots of tension between the two methods.

Software development is very often compared to construction: architects draw blueprints, planners make plans, and then you simply execute the project according to these plans; you build the system; things are rational and predictable; you know what to do, how long things take, et cetera. Failures happen because you deviate from the plans or the method, so you better not.

Let’s try another analogy: writing books. Developing software is like writing a book. How long does a book take to write? Say you define the problem as “a 400 page story about a man who has a mid-life crisis, who leaves his wife and kids for a twenty-something, then regrets it and comes back”. How long does that take? A year? Two years?

In Stephen King’s On Writing, he says that he doesn’t simply “come up” with his stories – rather, he discovers them. To him, he stories are already there, waiting to be found. He might start with an image he has got from reading a fragment in a news article, then he gets to work and over the course of time the story unfolds, and very often, he says, the end result is far from what he anticipated at the beginning of the project.

Now, writing fiction is obviously much more fuzzy than software, but construction – or at least the idealized image of construction, on which the analogy for software development is based – is more rational than software. At the start of a software project, you have a much better understanding of what problem the program must solve, than what is the case for the average writer about to begin a new book. But it’s not like building yet another house or bridge, where so much is about simply repeating what is known to work from centuries of experience in building. Software development is seen as identifying the problem, and then assembling the parts – whether they are crafted or bought.

Technical writing is possibly a better analogy for software development, in the sense that the domain is more well-defined than for fiction – but there’s still the element of exploring a space of possible solutions. How should the book be organized? What topics does it have to cover for the reader to understand it all? What examples must it use to explain difficult concepts?

I was once determined to write a book on test-driven development. I was told that publishers want a table of contents, along with one or two sample chapters before they will take on the book – in other words, a rough idea of the book’s scope and structure, and some text to give the idea of what the full book will be like. The table of contents is by no means cast in concrete when the book is accepted, and you are permitted to rewrite the sample chapters. Of course, you can’t scrap it all and write a book on pet grooming, if the publisher is expecting another book about Java.

Project management methods that begin with feasibility studies, and meticulous planning seems based on the idea that the moment you start writing code, then there’s no turning back. If you haven’t gotten the details right, the project will go on and on and cost millions, until it will bring the entire company down. Why is that? Software is soft! (Or it can be, if you pay continuous attention to its structure.) This method had lots of points during the feasibility study and the planning phases where the team has the option to abandon the project, due to high risk, or whatever. Why is it worse to start writing code, and then abandon the project if it doesn’t seem to be a good idea to carry on?

If you start with feasibility studies and planning, you are exploring the abstract. You are tossing ideas around, trying them, to see if they will work. If it is an unprecedented system that is being planned, then there’s little experience to draw from, and chances are that you are missing things. On the other hand, if a similar system has been built in the past, you can rely on experience in evaluating the ideas. But still, the matters you are dealing with are abstract. No matter how many diagrams you draw, and how much you talk, there’s the risk of leaving things out. It’s likely that you have to involve future users, that aren’t as used to discussing abstract things. They know how things work now, and what doesn’t work that good, but to express this to other people – it’s difficult. Also, there’s many very subtle things that future users don’t even reflect upon in their daily work – they just do things automatically, because they have always done so.

Recently, I have read about emergence, network science, and complexity theory. This has to do with systems of many agents interacting with each other, and although their interactions are fairly simple, they give rise to stunningly complex behavior at the scale of the system – behavior that is virtually impossible to anticipate from studying the nature of the interactions. The implication this has for software, is that as you define (on paper or verbally) the interactions for the components of the future system, you can’t anticipate its actual behavior. For this, you must develop it and try it. (See my post Software systems are complex systems.) This suggests that you should start coding as soon as possible, and do the feasibility study over the course of the entire project, instead of up front. Just because you have developed code doesn’t mean that you can’t cancel the project.

I’ll have to stop now.