I’m reading Richard Feynman’s “What Do You Care What Other People Think?”, in which he talks about, among other things, his work on the Rogers Commission investigating the Challenger accident. I wonder if Edward Tufte has read it; here’s what Feynman writes:

Then we learned about “bullets” – little black circles in front of phrases that were supposed to summarize things. There was one after another of these little goddamn bullets in our briefing books and on the slides.

In The Cognitive Style of PowerPoint, Tufte analyzes, not a NASA slide, but a Boing one, summarizing (with bullets) the results of a risk assessment concerning a possible damage on the left wing caused at liftoff (see Tufte’s website). Feynman did something similar:

The only place [the problem of leaking joints was discussed] was in the “flight readiness reviews” – between flights there was no discussion of the seals problem!

We looked at the summary of the report. Everything was behind little bullets, as usual. The top line says:

• The lack of a good secondary seal in the field joint is most critical and ways to reduce joint rotation should be incorporated as soon as possible to reduce criticality.

And then, near the bottom, it says:

• Analysis of existing data indicates that it is safe to continue flying existing design as long as all joints are leak checked with a 200 psig stabilization …

I was struck by the contradiction: “If it’s “most critical,” how could it be “safe to continue flying”? What’s the logic of this?” [–––]

We went back through the report and found the analysis. It was some kind of computer model with various assumptions that were not necessarily right. You know the danger of computers, it’s called GIGO: garbage in, garbage out! The analysis concluded that a little unpredictable leakage here and there could be tolerated, even though it wasn’t part of the original design.

If all the seals had leaked, it would have been obvious even to NASA that the problem was serious. But only a few of the seals leaked on some of the flights. So NASA had developed a peculiar kind of attitude: if one of the seals leaks a little and the flight is successful, the problem isn’t so serious. Try playing Russian roulette that way: you pull the trigger and the gun doesn’t go off, so it must be safe to pull the trigger again …

In the case of the Columbia report, there was also some kind of computer model which wasn’t relevant in determining risk – not because of false assumptions, but because of the test data being so far off: the debris in the model was 640 times smaller than that which actually hit the shuttle.