My mother taught environmental design and as part of my genetic or environmental heritage, design has remained a secret passion. 99% Invisible is a podcast about design, and over the past two weeks has discussed automation, beginning with the airbus and ending with the driverless car. While many recent articles have focused on the employment consequences of automation, they looked at the downstream effects of automation’s design.
The paradox of automation is that the more efficient the automated system, the more crucial the human contribution of the operators. Humans are less involved, but their involvement becomes more critical. For a much clearer understanding of this paradox take a few moments to read “Should Airplanes Be Flying Themselves” in Vanity Fair, which was the basis of the 99% Invisible’s podcast.
Because of cockpit automation, the kind of evidence based, thoroughly vetted, fully implemented crew resource management, which has led to aviation’s excellent safety record and is the model/analogy most used to discuss healthcare safety:
"The Captain of the Air France flight had logged 346 hours of flying over the past six months. But within those six months, there were only about four hours in which he was actually in control of the airplane - just takeoffs and landings. The rest of the time, auto-pilot was flying the plane.”
But this thought confirmed something I had been sensing for some time especially from my involvement with order entry and order set development.
“ ...In this case it's quite clear that these pilots had had experience stripped away from them for years"
Developing a protocol to treat diabetes is hard work. You gather experts who distill clinical knowledge and heuristics into explicit order sets and protocols. Once in place and running smoothly, care improves and we turn to other issues that require our attention. But after awhile, treating diabetes consists of using an order set, we no longer have experience in managing diabetes. We become like the Air France pilot, we are stripped of experience.
I think the paradox of automation arises because thinking and doing are intertwined. Automating the doing enhances consistency and safety, but automating our thinking removes experience and the wisdom that created the orders and protocols in the first place.
I advocate standardize care but I think we need to design a process where we have the opportunity to fly without the autopilot, to practice medicine in a much more literal sense. I think we should give more thought to how we maintain, review and edit these systems.
Arthur C. Clarke stated it best, "Any sufficiently advanced technology is indistinguishable from magic." We need to be sure that in our desire for safety and consistency we understand our tools or over time we will come to view them as magical, immutable protocols that work but we are not sure why. We will join the pilots in asking this common question in the cockpit, ‘What’s it doing now?’