Complexity science and software development

A year ago today, I was reading Steven Johnson’s book Emergence, and six months ago, I was reading Mitchell Waldrop’s Complexity. Both are (excellent) books about complexity theory, and I read them to see in which way I could apply this theory to software development.

One thing I thought about a lot was whether it is possible for a team of programmers to work analogously to ants in an anthill—in the sense that they follow “local rules” without the coordination of a master planner, and still produce a well-designed system.

One of the posts, from a year ago, deals with Oliver Selfridge’s “Pandemonium” metaphor. I wrote that he had experimented with pattern recognition in the 1950s, and while rereading Paradise Lost, he had the idea of modeling the system as individual demons, “shrieking” their votes to their superior demon, with a super-demon at the top, collecting votes.

I tried to apply this metaphor to programming teams, where programmers come and go over time, and who “shriek” through code. But this wasn’t a very useful analogy.

In a second post the same day, I continued this line of thought:

In Emergence, Steven Johnson interviews ant researcher Deborah Gordon [who wrote Ants At Work] and … she says something very interesting: “I was interested in systems where individuals who are unable to assess the global situation still work together in a coordinated way … And they manage to do it using only local information.”

This I associated to something Ron Jeffries had written on the Emergent Design mailing list the day before:

I find that refactoring messy code to be clear—good naming, removal of duplication, increasing cohesion, reducing coupling—seems almost magically to produce a good design.

I felt, and still feel, that there must be a way to get by with very little coordination—especially compared to the heavy planning work that is done in the beginning of many projects. This would definitely require programmers to act as autonomous agents, making local decisions to avoid the bottlenecks that come with centralized planning and control.

But that’s not all.

There must also be something that aligns these autonomous agents: a framework, a common vision, a common goal, a shared vocabulary, a shared culture.

At that time, I had consumed a lot of texts and talks by Alan Watts, who in his talk Seeing Through the Net talked about “the spotlight and the floodlight”, warning against becoming preoccupied with the bits, overusing the spotlight, and forgetting about the whole, forgetting to use the floodlight.

This made me slightly confused, as there are numerous examples of systems where agents completely ignore the whole, and where the whole still works perfectly well. But these systems have been shaped by millennia of trial and error, and the agents involved aren’t capable of assessing the state of the systems of which they are part.

In the post from six months ago, I quoted a passage from Complexity:

“Until I went to Santa Fe,” [economist and complexity theorist Brian Arthur] says, “I thought that an economic problem had to be well defined before you could even talk about it. And if it wasn’t well defined, what the hell could you do with it? You certainly couldn’t apply logic.”

[John] Holland’s answer [to the question of how you can learn in a poorly defined problem context] was essentially that you learn in that environment because you have to: “Evolution doesn’t care whether problems are well defined or not.” Adaptive agents are just responding to a reward, he pointed out. They don’t have to make assumptions about where the reward is coming from.

Software projects are very often great examples of poorly defined problem contexts. There are always subtle things in the processes that the software is built to model, things that often are overlooked, and which when discovered can have big impact on the planned design. Because of this, it’s important to keep your options open and defer decisions until as late as possible.

Umm, I lost track of what I was going to say, but there’s something interesting in all this. Got to go.