Probably right, or wrong

In the post below, I ask you to make a diagnosis of a medical condition. Most people get it wrong, probably because the actual diagnosis is far removed from the setting presented. People apply their inductive forces to a new problem, based on probabilistic inferences from other situations with which they are more familiar.

I attended a seminar on Friday at which MIT's Joshua Tenenbaum presented a theoretical basis for this learning process. If you subscribe to Science Magazine, you can read his recent article on the topic: "How to Grow a Mind: Statistics, Structure, and Abstraction."

It turns out that people are reasonably good at inference, from a very young age, as Joshua notes:

Generalization from sparse data is central in learning many aspects of language, such as syntactic constructions or morphological rules. It presents most starkly in causal learning: every statistics class teaches that correlation does not imply causation, yet children routinely infer causal links from just a handful of events, far too small a sample to compute even a reliable correlation!

In a more theoretical section, the author describes a probabilistic, or Baysian, model to explain this learning process:

How does abstract knowledge guide inference from incomplete data? Abstract knowledge is encoded in a probabilistic generative model, a kind of mental model that describes the causal processes in the world giving rise to the learner's observations as well as unobserved or latent variables that support effective prediction and action if the learner can infer their hidden state. . . . A generative model . . . describes not only the specific situation at hand, but also a broader class of situations over which learning should generalize, and it captures in parsimonious form the essential world structure that causes learners' observations and makes generalizations possible.

Except when it doesn't work! As several of you demonstrated below, that same probabilistic model can lead to cognitive errors.

I summarized Pat Croskerry's explanation below:

Croskerry's exposition compares intuitive versus rational (or analytic) decision-making. Intuitive decision-making is used more often. It is fast, compelling, requires minimal cognitive effort, addictive, and mainly serves us well. It can also be catastrophic in that it leads to diagnostic anchoring that is not based on true underlying factors.

Why the dichotomy? How can a learning process that works so well in some cases led us awry in others? I asked Joshua, and he suggested that it might have to do with the complexity of the issue. For those functions that were important in an evolutionary sense as humans evolved -- e.g., recognizing existential threats, sensing the difference between poisonous and healthy plants -- a quick probabilistic inference was all that mattered.

Now, though, in a complex society, perhaps we get trapped by our inferences. The sense of tribalism that led us to flee from -- or fight -- people who looked different and who might have been seeking to steal our territory or food becomes evident now as unsupported and destructive racial or ethnic prejudice.

Likewise, the diagnostic approach to illness or injury that might have sufficed with simple health threats 10,000 years ago no longer produces the right result in a more complex clinical setting. Think about it. If you were a shaman or healer in a tribe, most conditions or illnesses healed themselves. You recognized the common ailments, and you knew you didn't need to do much, and whatever herbs or amulets or incense you used did no harm. If you couldn't cure the disease, you blamed the evil spirits.

In contrast, as a doctor today, you are expected to apply an encyclopedic knowledge to a variety of complex medical conditions -- cancer, cardiovascular disease, liver and kidney failure -- that were relatively unknown back then. (You were more likely to die from something more simple at a much younger age!) Many cases you see today have a variety of symptoms and multivariate causes and different possible diagnoses. It is no surprise that your mind tries to apply -- in parsimonious form -- a solution. The likelihood of diagnostic anchoring is actually quite high, unless you take care. As I note below:

Croskerry thinks we need to spend more time teaching clinicians to be more aware of the importance of decision-making as a discipline. He feels we should train people about the various forms of cognitive bias, and also affective bias. Given the extent to which intuitive decision-making will continue to be used, let's recognize that and improve our ability to carry out that approach by improving feedback, imposing circuit breakers, acknowledging the role of emotions, and the like.

Teach the doctors, please!

If you read the Boston newspapers, you would think that the most important thing going on in health care is a proposal to move from one kind of insurance payment scheme to another. Reporters seem willing to accept relatively unsupported and undocumented assertions that global payments are working. You have to be persistent to find these sentences in this story:

But other doctors and health care executives cautioned against drawing definitive conclusions from the insurer’s early results. They have not been independently reviewed and may not be easily reproduced statewide.

If it is working, why do reporters not demand more transparency to demonstrate it? Why this instead?

Blue Cross did not release specific performance results for doctors groups.

Why no mention in these stories of alternative approaches being taken by other insurers?

Why hold off, too, on this really important statement until after the page turn and deep at the end of this story?

Both supporters and critics of global payment agree that any mandate should be flexible, and phased in slowly, so patients and providers can adjust. Thomas A. Croswell, chief operating officer of Tufts Health Plan, suggested a five-year transition.

Much is often made of Atul Gawande's superb writing about the use of check lists and other quality and safety process improvements. If you read carefully, though, you will see that he seldom mentions progress in the medical schools with which his and other Boston hospitals are affiliated. While we wait five years or more for the new pricing regime, why don't the insurers, the state government, and other stakeholders put pressure on the region's four medical schools to introduce and emphasize the science of process improvement in their curriculum?

Local readers might be interested to know that the process improvement world is alive and well in other regions, irrespective of insurance payment regimes. Two of the heroes in this arena, Brent James and Bob Wachter, recently had a conversation about how to teach quality and safety improvement.

Dr. James gave some history of his efforts at Intermountain Health. Dr. Wachter asks:

[Y]ou and others have written about the culture of medicine being so individualistic. It sounds like we came into this with a culture that you would expect would create tremendous variation from doctor to doctor.

Dr. James replies: 

Looking back, that's absolutely true. Of course it came to be called the craft of medicine, a cottage industry, where it's based on purely personal expertise, personal perfection, if you will. Speaking as somebody out of a surgical background—that concept is so central to what it means to be good, I mean for your patients, the best you can be. You don't want to lose that personal dedication. But you start to extend it a step further. Where it ended up for us was a form of Lean.

And later, he relates:

We did other things that were really important. The first is that we built firmly on the foundation of medicine. By that point, we'd understood that there's a whole bunch of jargon with improvement, but you didn't have to use any of it; you could describe the whole thing in the language of medicine. So rather than asking the natives to learn quality improvement jargon, we spoke the language of the native. The second thing was that in order to graduate you had to complete a successful improvement project. Our aim was to get hands-on experience that was real. And boy did that ever turn out well.

Here is an article about a system clinical safety and effectiveness (CS&E) course taught at the University of Texas. An excerpt:

Unfortunately, most front-line caregivers complete their professional training with almost no exposure to even rudimentary QI concepts or methods....

The University of Texas MD Anderson Cancer Center began such a course in 2005 ... and its success led us to implement courses in four of the six health campuses in the University of Texas system.... 

The purpose of the CS&E course is to provide physicians, other key clinicians, and administrators the skills and knowledge required to lead breakthrough change initiatives. After initial success at UT MD Anderson Cancer Center, all presidents of the UT System health care institutions approved a proposal in 2007 to develop and implement CS&E programs on their own campuses. A UT CS&E Steering Committee with representatives from each campus was established to provide oversight for the course expansion, and in 2008 the University's Regents provided funding from the UT System's malpractice liability reserve fund.

We are very quick to find a rivalry between Texas and Boston in other fields. Let's start one here, where lives hang in the balance.

Our patients are sicker

What more do we need to know? The British Medical Journal published a study showing that Peter Pronovost's program to reduce central line infections in Michigan saved lives.

A new study finds that a safety checklist program developed by a Johns Hopkins doctor has reduced patient deaths in Michigan hospitals by 10 percent, in addition to nearly eliminating bloodstream infections in health care facilities that embraced the prevention effort.

The research, published in the British Medical Journal, is the first to show a drop in patient mortality in hospitals using the Hopkins program. Previous studies have found major reductions in bloodstream infections from using the checklist when inserting catheters or central lines to give patients medication, fluids or nourishment.

Well, duh. But I guess it is important to have scientific verification. But I can almost hear the comments from some places: "That wouldn't work here. Our patients are sicker."

So, how long will it take for this approach to be used across the country? This study is based on work from nine years ago. If this is like other innovations in medical care, it will take a decade and a half more to spread.

Here's my proposal to jump-start it. Publish the monthly rate of central line infections for all hospitals on a public website. CMS, IHI, the Dartmouth Atlas group or some other organization could do this in a nano-second, creating a voluntary website, giving each hospital a password through which it could enter its own data. There is no need to audit the figures. We can trust people to be honest.

And, at the bottom of the website, the host could list the hospitals that have chosen NOT to publish.

Then, you would see the power of transparency.

Lessons from Cairo

I think most of us would be hard-pressed not to be inspired by the moral power of the crowds in people in Egypt as they throw off a form of government that they find repressive. But, it is about the US reaction to all this to which I turn today. It is the subject of New York Times columns by David Brooks and Nicholas D. Kristof.

Both authors noted the ham-handed manner in which the United States deals with authoritarian regimes and with popular movements for freedom.

Brooks notes,
The . . . thing we’ve learned is that the United States usually gets everything wrong.

Policy makers always underestimate the power of the bottom-up quest for dignity, so they are slow to understand what is happening.

Then their instinct is to comfort the fellow members of the club of those in power.

Then, desperately recalibrating in an effort to keep up with events, they inevitably make a series of subtle distinctions no one else heeds.

Kristof says,
Yet one thing nags at me. These pro-democracy protesters say overwhelmingly that America is on the side of President Mubarak and not with them. They feel that way partly because American policy statements seem so nervous, so carefully calculated.

The upshot is that this pro-democracy movement, full of courage and idealism and speaking the language of 1776, wasn’t inspired by us. No, the Egyptians said they feel inspired by Tunisia — and a bit stymied by America.

Everywhere I go, Egyptians insist to me that Americans shouldn’t perceive their movement as a threat. And I find it sad that Egyptians are lecturing Americans on the virtues of democracy.

Brooks provided a broader context for all of this:

I wonder if sometime around 50 years ago a great mental tide began to sweep across the world. Before the tide, people saw themselves in certain fixed places in the social order. They accepted opinions from trusted authorities.

As the tide swept through, they began to see themselves differently. They felt they should express their own views, and these views deserved respect. They mentally bumped themselves up to first class and had a different set of expectations of how they should be treated. Treatment that had once seemed normal now felt like an insult. They began to march for responsive government and democracy.

You will excuse me if I draw the connection to health care. I hope you don't think it inapposite.

I do not think that it has been a fifty year trend in health care, but a more recent one. Patients and families have decided that they should be equal partners in the process of diagnosis and treatment. They believe that they have a right to the information that can represent life or death, health or suffering. As Brooks would put it, "treatment that once seemed normal now feels like a insult." Opinions from trusted authorities no longer carry the weight they used to. Questions are being asked. Answers are being demanded.

A few weeks ago, author Charles Kenney asked the question, Isn't there a compelling -- perhaps even overriding -- moral component to transparency?

I responded,

The answer, of course, is yes. Doctors and others pledge to do no harm. How can you be sure you are living by that oath if you are unwilling to acknowledge how well you are actually doing the job? As scientists, how can you test to see if you are making improvements in evidence-based care if you cannot validate the "prior" against which you are testing a new hypothesis? At the most personal, ethical level, how can you be sure you are doing the best for people who have entrusted their lives to you if you are not willing to be open on these matters?

But transparency threatens the status quo. In the medical world, status quo confers power, influence, prestige, and money on those who have had a reputational advantage. A close friend and colleague put it this way:

Transparency in self-interested institutions who are making fortunes by deluding themselves and the public that they and only they know what the community wants and needs is a very dangerous concept.

The agents of change in this battle will be the same people who are turning things over in Egypt. Normal people who have experienced pain and suffering, or even just disrespect, in the health care system are starting to find their voice. Like the US in the international arena, the powers that be in the government and their agents, having been captured by the powerful forces of the medical and hospital profession, are slow to react and are protective of the status quo.

Policy makers always underestimate the power of the bottom-up quest for dignity, so they are slow to understand what is happening.

Then their instinct is to comfort the fellow members of the club of those in power.

Then, desperately recalibrating in an effort to keep up with events, they inevitably make a series of subtle distinctions no one else heeds.

Defining a defect

From Charles Kenney's book Transforming Health Care, about Virginia Mason Medical Center's journey:

Implementing the program was not a simple matter. Defining a defect in a medical setting presented a challenge.... [D]octors pushed back. The argued that many instances of harm -- ventilator-acquired pneumonia, for example -- should not be considered an error because these things happened in medicine. Complications, they argued, were inevitable.

This is a typical assertion, based on a belief that there is a statistically irreducible amount of harm that must occur in medical settings. There may be such a statistically irreducible amount, but most hospitals are not close to the potential minimum. As Gary Kaplan and his team showed, and as shown at BIDMC, setting an audacious target of zero defects and organizing work to reach that target can enable the people in a organization to reach or get mighty close to that target.

Joseph Gavin strived for such a goal in space flight. Others are doing so in medicine.

Real transparency is a concomitant of success in such a transformation. You cannot improve what you do not acknowledge to be flaws. That is why I pound away below as to its importance and as to why misuse of transparency is unethical.

In her humorous way, Ethel Merman tried to show us the way when she decried the view that "these things happen." But this is deadly serious. Those who stand in the way are causing death and injury as clearly and directly as those who wield weapons.

If you cannot see the video, click here.

Transparency is not marketing

When is transparency not transparency? Answer: When it is marketing.

A recent ad campaign by a well known hospital system suggests that you are better off going to one of its hospitals if you have a stroke because they have a speedy rate of administration of an anti-clotting agent. It is true that rapid administration of this drug is very important.

But the data offered by this hospital system are old, based on the period 2006-2008. According to the Boston Globe, "State officials said that when data for 2009 and 2010 are released next year, they expect the gap between hospitals will have narrowed because of improved care."

Look, no one will argue that you don't get excellent care at this hospital system. Quite the contrary. But to suggest that you will get better care, based on old data, just isn't right. It might even raise unnecessary concern among patients or their families. Imagine, for example, that a loved one is having a stroke and you ask the ambulance to go to a hospital that is farther away because you think that the patient will get faster treatment. The extra time spent in the ambulance might add danger itself.

Also, selective use of clinical outcomes for marketing purposes is a slippery slope. Let's review the issue, for example, of "door-to-balloon" time. The Joint Commission has set a standard for opening blocked arteries with catheterization (percutaneous coronary intervention) within 90 minutes of presentation at an emergency room in a hospital. The hope is to achieve this goal at least 90% of the time.

But one member of this same hospital system only accomplished this standard about 60% of the time for part of 2009. I don't recall a marketing campaign back then that referred to this result.

You cannot be selective about transparency. You have to post the good and the bad. See the VA story below. If you use it for marketing purposes when the numbers are good, you rightfully open yourself up to attack for selective use of statistics.

Let's just accept that transparency is about holding ourselves accountable to a high standard of care and learning from one another, rather than attempting to use it as a marketing tool.

VA stands for "very accountable"

A mutual friend recently introduced me to Scott Gould, Deputy Secretary of the Department of Veterans Affairs, who informed me that VA has recently posted the performance data for all 153 medical centers at Veteran's Health Administration on the web. Here's the introduction from the website:

Welcome to the VA Hospital Compare web site. This site is for Veterans, family members and their caregivers to compare the performance of their VA hospitals to other VA hospitals. Using this tool, Veterans, family members, and caregivers can compare the hospital care provided to patients.

Imagine that. They are actually inviting people to make comparisons of clinical quality in their hospitals. I am guessing that this kind of transparency gives people in the individual hospitals an extra incentive to do well. As I have often said about transparency, its main value is in holding ourselves accountable to the standard of care we say we believe in.

This is clearly exemplified by the VA. See below for more from the website. I say bravo and congratulations!

The Secretary of Veterans Affairs (VA) and the VA’s Under Secretary for Health are committed to transparency − giving Americans the facts. The Veterans Health Administration (VHA) releases the quality goals and measured performance of VA health care in order to ensure public accountability and to spur constant improvements in health care delivery. The success of this approach is reflected in our receipt of the Annual Leadership Award from the American College of Medical Quality.

Raising the bar for the 21st century healthcare
Much of the data in LinKS and ASPIRE are simply not measured in other health systems – VA is raising the bar. When available, VA uses outside benchmarks but often sets VA standards or goals at a higher level. VA scores hospitals more than 30% different from the goal as underperforming or red and those only 10% different from the goal are shown in green in ASPIRE. But a red site within the VA might be a good performer compared to outside counterparts. The scoring system is designed to move VA forward. ASPIRE is not about finding fault but about helping VA to target opportunities for improving performance.