Exploration of possibilities
Several writers—for instance Stephen King, Patricia Highsmith, and Minette Walters—likens their writing to exploration: they get an idea, which they then explore in the process of writing the book, neither knowing where it will lead nor how it will end, but still following it.
In his book On Writing, Stephen King calls it “archaeology”: you find a bone (often by chance), and then you start uncovering the skeleton, or whatever, and the blunter the tools the more you damage your find; the stories are already there, you just dig them out. He also says his novels often turn out very different from what he expected, both in terms of length and in terms of story.
I like to view this as guided evolution: you start with a seed, plant it and then control its growth. At any given time in its evolution, there is a limited set of possibilities for the future evolution. By evolving the plant in one direction, you rule out some of them, and sometimes open new possibilities.
If you pay attention to these possibilities, and respect them, you get coherence without pre-planning. Why is this scarier to software people than speculating at a time when you don’t know much at all about the system you’re building, and then committing to those speculations (in terms of scope and estimated time to implement)?