Monday, August 30, 2010

Physician Quality Report Cards, Part II

I am frankly surprised by the number of comments, emails, and telephone calls I have received about my doctor report card blog post. (http://ow.ly/2wcvd) Some were charged with emotion and even anger. The number and tone of the responses indicate that Dr. Danielle Ofri (http://danielleofri.com/?p=1169) and I hit a nerve that resonates differently with different audiences, and we should all try to better understand the several sides to this important, complex, and relevant issue

In this post, I would like to explore how we can equip physicians with the humility, courage, and existential strength needed to want to receive the kind of timely, blunt feedback on performance that is necessary for continuous improvement of clinical care. There are models in professional football, innovative teacher training programs, and public school teacher report card initiatives that might inform us. I would also like to revisit my interest in replacing the current pessimistic model of error and failure with a more optimistic model. (http://ow.ly/2wsPq) While we are all too familiar with the shame and embarrassment associated with being told we are wrong, an optimistic model of error and failure guides us to being more receptive to feedback because the focus is not on us, but rather on the “other” that needs to be taught or cared for. In an optimistic model of error, Kathryn Schulz believes we can respond to feedback about failure with bafflement, fascination, amusement, excitement, curiosity and delight. (http://ow.ly/2wsTn).

Why is it so hard for us to admit error and receive blunt feedback? Why is it so important for all of us to always be right? Chris Argyris wrote about why it is difficult for the successful to learn; success really does not teach as much as failure, and when the usually successful fail or need to improve they become defensive. (http://ow.ly/2wsYy) Argyris believes there is a universal tendency for humans to respond to feedback by trying to achieve four goals: stay in control, win and not lose, feel positive, and behave rationally (http://ow.ly/2wt0P). Harville Hendrix, a marriage therapy expert, believes being wrong is so threatening and unwelcome because of concentric consciousness, having our internal cohesive sense of self disrupted resulting in chaos, and experiencing shame and guilt which is painful. We become rigid in our beliefs and defensive because we do not like feeling insecure. (http://ow.ly/2wt5Y)

Primary care providers are dispirited, angry, and upset by the status quo. Their level of frustration is reflected by some of the over the top comments to my blog such as wanting to “projectile vomit over” those with whom they disagree and calling administrators “idiots.” (http://ow.ly/2wcvd) We should all take this low morale problem seriously. However, I hardly see how such an attitude does anything but drive physicians and patients and administrators farther apart. It also reinforces an unfortunate impression that physicians will only learn from other practicing physicians; they will not listen to or learn from patients or other professionals who just might have something to contribute to improving care.

There is wisdom and perspective to be had from non-physicians (sometimes even those wearing suits) who have tackled similar problems in other fields of endeavor. I have long been depressed by how parochial and slow we in medicine have been to adopt knowledge from other fields. Medicine has lagged behind in adopting information technology; hospitals have been much slower to use social media than other industries; health systems have been reluctant to learn from the lean movement; medical schools have been tardy in the use of computer simulation to teach clinical skills.

My daughter Reva spent three years in the media relations department of the Philadelphia Eagles, and one of the take home messages was the level of individual feedback involved in pro football. After every game, the position coaches grade every player on every play based on the film record. It can be a difficult exercise for one’s ego, but because the feedback is timely, it also does promote learning and improvement. Those who cannot master the plays or techniques are cut from the team.

My son Colin tells me that the MATCH Teacher Residency training program in Boston utilizes similar extensive timely feedback to develop recent college graduates into teachers capable of effectively teaching in high poverty charter schools. These students get graded on a 1 to 10 scale after every class, review videotapes with coaches, and even receive via an earpiece real-time coaching while conducting classes.

Public education is faced with a teacher evaluation system that most agree is broken. Traditionally teachers have been evaluated on the basis of brief, pre-announced visits by principals who offer a confidential and highly subjective assessment of their teaching. More than 90% of teachers receive a passing grade according to a four state study by the New Teacher Project. (http://ow.ly/2wcGk)

An evaluation tool that is increasingly gaining popularity with educational reformers is called value added analysis. Value added analysis uses students past performance on tests to project his or her future test results. The difference between the prediction and the student’s actual performance after a year is the value that the student’s teacher added or subtracted. (http://ow.ly/2wcGk) This method has been used for years by educational researchers who claim it is good at identifying star performers and the weakest teachers; it is not particularly good at differentiating between the middle group of teachers.

Value added analysis, developed by economists in the 1970s, has recently been used by some school districts to determine which teachers should be rewarded and which instructors need help. District of Columbia Chancellor Michelle Rhee recently fired 26 teachers based in part on their poor performance on a value added analysis. (http://ow.ly/2wcGk)

The Los Angeles Times hired a Rand Corporation researcher to run value added analysis on more than 6,000 third through fifth grade teachers in LA, using data that had been largely ignored by the school district. In the first article published this month, superior and inferior teachers are identified by name and interviewed. (http://ow.ly/2wcGk)

After a single year with teachers who ranked in the top 10% in effectiveness, students scored an average of 17 percentile points higher in English and 25 points higher in math than students whose teachers ranked in the bottom 10%. Students often backslid significantly in the classrooms of ineffective teachers, and thousands of students in the study had two or more ineffective teachers in a row. (http://ow.ly/2wcIJ)

The newspaper plans to run a series of articles revealing individual teachers’ effectiveness. The union president of United Teachers Los Angeles has called for a boycott of the newspaper. (http://ow.ly/2wcIJ)

Teacher report cards are not perfect. Some teachers who rank highly one year fall to the bottom of the scores the next year, according to a Florida study in the journal Education Finance and Policy. (http://ow.ly/2sNdj) Small class size makes it important to use several years’ worth of data; the LA Times study used multiple years’ worth of data for each teacher. Arne Duncan, the Secretary of Education in the Obama Administration, advocates using report cards as part of the evaluation of teacher performance and considers it an improvement over the current subjective process. Daniel Willingham, a cognitive psychologist at the University of Virginia states, “Damn near anything is going to be an improvement on the status quo.” (http://ow.ly/2sNdj) The question is are report cards better than what we use now; the question is not are report cards perfect.

Several of those who commented on my original blog thought that report cards are better issued at the large hospital or physician network level, rather than to individual physicians. Dr. Anne Marie Cunningham of Wales provides the most complete argument for this tactic in a long and thoughtful blog post, but she also admits in a comment that there are “no published studies of the effect of publicly reporting performance data on quality improvement activity among physicians or physician groups.” (http://ow.ly/2wcyG) So we really do not know if either the system or individual approach will be useful. Dr. Cunningham also posts a video that contends that doctors have autonomy and that team based care is just a myth. I find it interesting that nobody contested the Colorado Kaiser mammogram study where individual report cards identified and removed substandard radiologists who missed suspicious lesions.

Gilles Friedman and e-Patient Dave, both advocates of participatory medicine, make the point that patients need a direct conduit into the medical record without being filtered by the physician’s interpretation. (http://ow.ly/2wcvd) The experience of social media sites like PatientsLikeMe and CureTogether to involve patients in meaningful clinical research is something that I enthusiastically support. However not all practicing clinicians have come around to Don Berwick’s recommendation that the patient control and own his own medical record and that physicians behave as guests and not hosts in the care of their patients. (http://healthaffairs.org/blog/author/berwick/)

The comment by bev MD (http://ow.ly/2wcvd) provides us, I think, with a way forward:

The thing that frustrates me in this whole debate is docs' idea that this entire concept is invalid and should not be applied to them. Then, when it is (and get real guys; this is going to happen, period), they quibble with the metrics and behave as victims….

As Paul Levy repeatedly has commented on his blog, Running a Hospital, we need to get busy and do this ourselves - or it will be imposed externally by those who do not know how to do it. This is happening right now - wake up and get control, or be controlled!

Practicing clinicians, patients, administrators, measurement experts, and others need to work together to create standards that can be measured and that can eventually improve the health status and overall well being of a population. We need to figure out accountability that works, whether at the system or individual level. We need to provide individual clinicians who score low with the tools, time, and institutional support that can help them improve the care they deliver. We need to continually improve the report cards so that they are meaningful and do help improve care.

The fact is we spend so much money on education and on medical care in the United States that we need to develop ways to ensure that we are getting our money’s worth.

US Education Secretary Arne Duncan was talking about teacher report cards when he said: “The truth is always hard to swallow, but it can only make us better, stronger and smarter. That’s what accountability is all about – facing the truth and taking responsibility.” (http://ow.ly/2wcQN) But, he could just as well have been talking about doctor report cards.

I hope all of us in medicine can react as gracefully as Karen Caruso who was identified in the LA Times analysis as in the bottom 10% of elementary school teachers in boosting students’ test scores.

“For better or worse testing and teacher effectiveness are going to be linked….If my student test scores show I’m an ineffective teacher, I’d like to know what contributes to it. What do I need to do to bring my average up?” (http://ow.ly/2wcGk)

1 comment:

  1. Kent;

    Not sure why you haven't gotten any comments on this post, but I thought it was excellent. Hopefully THCB will pick it up and we can start again with the comments. (:

    bev M.D.

    ReplyDelete