Are Mystery Shoppers Such a Bad Idea for Health Care Quality Improvement?

The decision by the Obama administration to employ “mystery shoppers” to pose as patients to see how difficult it is to get an appointment with a physician has sparked criticism from physicians. However, access to primary care physicians is a very real public policy issue that needs to be understood if we are to successfully care for the more than 30 million Americans who receive coverage under the Affordable Care Act. Is the use of “mystery shoppers” a bad idea?

Dr. Raymond Scalettar certainly thinks it is a bad idea. “I don’t like the idea of the government snooping. It’s a pernicious practice – Big Brother tactics, which should be opposed.” (http://www.nytimes.com/2011/06/27/health/policy/27docs.html?ref=health)

Dr. George Petruncio says, “This is not the way to build trust in government. Why should I trust someone who does not correctly identify himself.” (http://www.nytimes.com/2011/06/27/health/policy/27docs.html?ref=health)

Westby Fisher, MD writes in his blog: “When information gathering trumps patient care - particularly fictitious care - we've got a problem. Is this a new quality standard we can expect from our new government health care initiative?

Just like scam-artists that phish for unsuspecting people's financial information online, governmental appointment phishing should not be tolerated in any way, shape, or form. It is fraud - plain and simple.” (http://drwes.blogspot.com/2011/06/appointment-phishing.html?utm_source=feedburner&utm_medium=twitter&utm_campaign=Feed%3A+DrWes+%28Dr.+Wes%29&utm_content=Twitter)

Several physicians on twitter retweeted Dr. Fisher’s blog post and indicated they agreed with his analysis.

A mystery shopper is a quality improvement tool that has been used in the retail and hotel industry for decades. In 2004 the health care industry accounted for 2% of mystery shopper revenue, but the use in health care is increasing.

(http://www.ncpa.org/sub/dpd/index.php?Article_ID=10273)

In health care, mystery shoppers have been used to provide feedback on every step in the patient experience: making an appointment, environment of the waiting room, encounter with nurses and office staff, and visit with the physician. There has been an increased interest in mystery shoppers by health care providers since CMS started posting patient satisfaction information on hospitals as part of the Website Hospital Compare. (http://www.boston.com/business/healthcare/articles/2007/06/13/on_sly_workers_rate_hospital_service/)

The use of mystery shoppers has been reported to lead to better patient flow and improved wait times, extension of office hours, improved telephone etiquette, better physician communication with patients, and more time with patients answering questions about surgery.

(http://www.boston.com/business/healthcare/articles/2007/06/13/on_sly_workers_rate_hospital_service/)

(http://www.pjonline.com/pdf/articles/pj_20040515_mysteryshopping.pdf.2004;272:615-617)

(http://www.ama-assn.org/amednews/2006/09/18/bise0918.htm)

The AMA Council on Ethical and Judicial Affairs studied the mystery shopper question and concluded that a sound program would include:

1) The places that unannounced visits will affect should know about the program.

2) The information should be used for improvement and not punitive actions.

3) Mystery shoppers should not be the sole source of data for evaluating clinical performance

4) The program should not adversely affect access to medical care by legitimate patients.

(http://www.ama-assn.org/ama1/pub/upload/mm/38/a08cejoreports.pdf)

I disagree with my colleagues that a properly planned and implemented mystery shopper program is a bad idea for trying to improve health care. For far too long, we in medicine have been too arrogant to learn lessons from other industries that improve quality. I think we need all the help we can get to take better care of patients.

Not Everything in Health & Wellness Can Be Reduced to a Single Number

"To measure is to know." Lord Kelvin

"If you can not measure it, you can not improve it." Lord Kelvin

Vs.

“Asking science to explain life and vital matters is equivalent to asking a grammarian to explain poetry.” Nassim Nicholas Taleb

“Technology is at its best when it is invisible.” Nassim Nicholas Taleb

How can technology help us live healthier lives? Why did Google Health fail? Why are Klout and Twitter Grader publicly issuing a number to me by name about how influential I am? What do Lord Kelvin and Nassim Nicholas Taleb have to teach us?

I was taught in medical school and pathology residency that health was defined as absence of disease; this definition pleased me because not much important could happen to the patient until I peered into my microscope and rendered a diagnosis. I looked up to Virchow and Rokitansky who were the most important and influential physicians in the most advanced medical centers in the 19^th century.

In the mid-20^th century the World Health Organization (WHO) famously stated: “Health is a state of complete physical, mental, and social well-being, and not merely the absence of disease or injury.”

“The dialogue between Asclepios, the god of medicine, and Hygieia, the goddess of health – the external intervention and the well-lived life – goes back to the beginning. Only in the twentieth century did the triumph of ‘scientific’ modes of inquiry in medicine (as in most walks of life) result in the eclipse of Hygieia. Knowledge has increasingly become defined in terms of that (and only that) which emerges from the application of reductionist methods of investigation.” (http://www.amazon.com/Why-Some-People-Healthy-Others/dp/0202304906/ref=sr_1_1?s=books&ie=UTF8&qid=1309114252&sr=1-1)

The WHO rejection of the “absence of disease” definition of health has been warmly embraced by some and largely ignored by many more. After all if the WHO definition means that our wellness is affected by all human activities, what should the Department of Health and Human Services focus on and budget for? The WHO definition of health has “been honoured in repetition, but rarely in application.”

(http://www.amazon.com/Why-Some-People-Healthy-Others/dp/0202304906/ref=sr_1_1?s=books&ie=UTF8&qid=1309114252&sr=1-1)

At one time I thought Google Health would become the Personal Health Record (PHR) that would allow individual patients to keep track of their medical and daily activity data and apply the WHO definition of health to their own life; I blogged about PHRs because I hoped they would solve the enormous problem of hospital based IT systems not communicating with each other.

(http://icsihealthcareblog.wordpress.com/2009/07/17/kent-bottles-thoughts-on-phrs/)

I still remember the excitement when Google CEO Eric Schmidt described Google Health for the first time at HIMSS 2008, and I played around with it. I found it easy to use and understand, but I never really used it for my own health and wellness purposes. The Wall Street Journal had a similar experience: “We signed up for Google Health…and a bunch of other personal health records for a story, but never quite felt compelled to actually use them.” (http://blogs.wsj.com/health/2011/06/24/google-health-r-i-p/)

Google Health failed because

· Patients are not that interested in entering their data into a PHR

· Google underestimated the complexity of health care

· Most consumer health data is not in a structured, machine computable format

· As an untethered PHR, professionals distrusted the accuracy of the data

· Google did not work with or engage physicians in the effort.

· It did not coordinate with technology developers.

The best analyses of the sad ending of Google Health can be found here (http://www.readwriteweb.com/archives/google_health_why_its_ending_what_it_means.php), here (http://chilmarkresearch.com/2011/06/24/rip-google-health/), and here (http://www.nytimes.com/2011/06/25/technology/25health.html?ref=health)

Because I am no longer protecting the narrow interests of pathology, the WHO definition of health continues to ring true. Since all human activity affects health and wellness and since we have such poor memories, how does one better understand one’s own body and mood? The personal informatics (http://personalinformatics.org/) and quantified self movements (http://quantifiedself.com/) are true descendants of Lord Kelvin; they believe that scientific, objective, quantified measurements can lead to behavior change that can promote wellness and health. Dr. Joseph Kvedar changed his attitude about yard work when the data from his Bodymedia armband convinced him that he burned more calories doing gardening than bicycling (http://www.technologyreview.com/blog/themeasuredlife/26918/?ref=rss) Shaun Rance used the anonymous website drinkingdiary.com to reduce his drinking after his father received a diagnosis of end-stage liver disease. Jon Cousins built Moodscope, a self-tracking system to manage his bipolar affective disorder; he even shares his personal mood data with a few friends. His self-tracking metrics are supplemented by human sympathy made possible by his online connections and community. (http://www.nytimes.com/2010/05/02/magazine/02self-measurement-t.html?scp=1&sq=nytimes+sunday+magazine+quantified+self+wired&st=cse&pagewanted=print)

While much that is written about self-tracking celebrates the potential of measurement to help the individual with sleep, exercise, sex, food, weight loss, mood, alertness, productivity, and spiritual well being, there is a dark side to the movement. Alexandra Carmichael, one of the founders of CureTogether where patients conduct research on diseases, wrote about why she stopped monitoring 40 measurements about herself: “Each day my self-worth was tied to the data. One pound heavier this morning? You’re fat. Skipped a day of running? You’re lazy. It felt being back in school. Less than 100 percent on an exam? You’re dumb.” (http://quantifiedself.com/2010/04/why-i-stopped-tracking/)

Google Health failed largely because consumers did not see the value of a PHR and because it was time-consuming to input data. Gary Wolf thinks four things have changed that make self-tracking easier and more acceptable:

· Electronic sensors got smaller and better

· People carry smartphones that are powerful computers

· Social media made it normal to share everything

· The development of the cloud

Other researchers believe that technology may support health and wellness not by having individuals self-track, but by monitoring our social networks. This approach has been called the social contagion theory of disease, and much can be learned without the individual doing anything except carrying around his smartphone. (http://www.amazon.com/Connected-Surprising-Networks-Friends-Everything/dp/0316036137/ref=sr_1_1?s=books&ie=UTF8&qid=1309114482&sr=1-1)

By analyzing the famous Framingham Heart Study, Nicholas A. Christakis and James H. Fowler were able to map 5,124 subjects for a connection web of 53,228 ties between families, friends, and co-workers. Obesity appears to spread among friends like a virus. “When a Framingham resident became obese, his or her friends were 57% more likely to become obese too.” “A Framingham resident was roughly 20% more likely to become obese if the friend of a friend became obese.” “You may not know him personally, but your friend’s husband’s co-worker can make you fat. And your sister’s boyfriend can make you thin.”

This social contagion process seems to also work for drinking, smoking, loneliness, and happiness. Christakis and Fowler believe that “these behaviors spread partly through the subconscious signals that we pick up from those around us, which serve as cues to what is considered normal behavior.” (http://www.nytimes.com/2009/09/13/magazine/13contagion-t.html?_r=1).

Alex Pentland, director of MIT’s Human Dynamics Laboratory, uses cell phone data to identify the influencers in any social network that are most likely to change other people’s behaviors. Dr. Pentland says, “Just by watching where you spend time, I can say a lot about the music you like, the car you drive, your financial risk, your risk for diabetes…We are trying to understand the molecules of behavior in this really complete way.” (http://online.wsj.com/article/SB10001424052748704547604576263261679848814.html)

Although I have never been studied by MIT, I have been assigned a number that supposedly indicates how influential I am. According to Twitter Grader I am ranked 88,837^th out of 9,826,593 with a grade of 100 out of 100. Klout states that I am a “pundit” with a score of 69 out of 100. PeerIndex gives me a score of 18. What do these numbers really mean? I have no idea.

“Now you are being assigned a number in a very public way, whether you want it or not,” said Mark W. Schaefer of Rutgers University. More than 2,500 companies are now using klout data, and special offers are just starting to come into my email box. The CEO of Klout of course thinks this is a great idea: “For the first time, we’re all on an even playing field. For the first time, it’s not just how much money you have or what you look like. It’s what you say and how you say it.” Others are not so sure and worry about creating social media caste systems and that such rating systems lack sentiment analysis (negative comments can be as affective as positive comments in raising one’s score). (http://www.nytimes.com/2011/06/26/sunday-review/26rosenbloom.html?ref=opinion)

So what can we conclude from this dizzying tour of health, wellness, measurement, technology, and life. I think we need both Lord Kelvin and Nassim Nicholas Taleb to guide us. I think it is a mistake to choose the WHO definition over the absence of disease definition of health, as though they are polar opposites. There is a continuum of meanings for the word health. “At one end of that continuum is well-being in the broadest sense, the all-encompassing definition of the WHO, almost a Platonic ideal of the Good. At the other end is the simple absence of negative biological circumstances – disease, pain, disability, or death.” (http://www.amazon.com/Why-Some-People-Healthy-Others/dp/0202304906/ref=sr_1_1?s=books&ie=UTF8&qid=1309114252&sr=1-1) I think Google will in the future realize they shut down Google Health too soon. As smartphones and sensors make it easier and easier to automatically input data, people will want a place to store all of their observations of daily living and medical data. Taleb is right; technology is at its best when it is invisible. However, some of the most important things in life are not reducible to a number or a PowerPoint slide with bullet points. Twitter Grader says I am 100 out of 100; Klout 69 out of 100; and PeerIndex 18 out of 100. Not all of these rankings can be accurately measuring my influence to my twitter community and readers of my blog. And let’s not forget what we are learning from social contagion theory; it is not just about the individual. I also think that there may be something to the Keas approach which, as I understand it tries to use game theory to make it more fun to use technology to help us change unhealthy behaviors. Tom Chatfield’s Fun Inc: Why Gaming Will Dominate the Twenty-First Century and Jane McGonigal’s Reality Is Broken: Why Games Make Us Better and How They Can Change the World certainly gave me many ideas about how to create flow for human beings in the service of individual and population health.

Why I Am an Information Flaneur

While reading the New York Times Sunday Book Review (in a hard copy format that got my fingers dirty with ink) I came across this quotation in a Evgeny Morozov book review of The Filter Bubble by Eli Pariser:

[Citizens] should not be content as mere passive recipients of tweets, pokes and bytes; they should aspire to become what some Internet scholars call “information flâneurs,” treading the unbeaten paths in cyberspace and defying the narrow categories stealthily assigned to them by Web services. http://www.nytimes.com/2011/06/12/books/review/book-review-the-filter-bubble-by-eli-pariser.html?ref=books

Here is a video of Eli Pariser discussing the filter bubble

http://www.ted.com/talks/eli_pariser_beware_online_filter_bubbles.html

I had never heard of a flâneur before and quickly went to Wikipedia where I learned that Charles Baudelaire expanded the definition of this word, which means stroller to define “a person who walks the city in order to experience it.” Now I admit that Baudelaire is one of those French authors I realize I am supposed to know, but I don’t remember ever reading. The concept of a flâneur has influenced social science theory of urban life, architecture, and photography. The Wikipedia article also mentioned that Nassim Nicholas Taleb endorses the concept in his essay on why he walks in The Black Swan, a book I have read twice. When I finally found my copy of The Black Swan, I discovered that I had indeed underlined the passage about walking and flâneurs. My memory is not what it used to be; it is increasingly harder and harder to remember what I read and where I left that book I remember as being important.

Later the same day I noticed a tweet about Ethan Zuckerman writing about serendipity. I had never heard of him, but his lengthy blog version of a keynote speech was all about cities and mentioned the flâneur concept. Talk about serendipity.

http://www.ictdev.org/pulse/20110512/ideas/chi-keynote-desperately-seeking-serendipity

I was even more surprised when I read the first comment on his blog from Marian Dork who gave a link to his article The Information Flaneur: A Fresh Look at Information Seeking http://mariandoerk.de/ This article has validated my approach to information gathering of wandering around and finding things I did not know I was looking for. This approach to Internet behavior came about because I am always looking for a specific book in my house that is hard or impossible to find. However, even when I fail to locate the original book I am looking for, I often find another more pertinent book I did not realize I owned.

I have now changed my twitter profile to proclaim that I am an Information Flâneur. Here are some things I have learned in the past few weeks by poking around twitter and the World Wide Web. With the lens of hindsight, I now realize I have been concerned with how disruptive technologies will affect the future of health care.

Here is my blog on a recent New York City session on living with man-made machines (I was excited that the WSJ.com picked up my post):

onespot.wsj.com/.../how-robots-will-teach-us-who-we-are-as

And here is an thought provoking video with a view of the future of medicine by Daniel Kraft of Stanford University:

on.ted.com/Kraft11

And here is a blogger who is thinking about the relationship between playing video games and becoming a surgeon:

http://ow.ly/5gK5f

A physician at PartnersHealth in Boston thinks we need to move from being couch potatoes to being advocates of the quantified self-movement:

http://chealthblog.connected-health.org/2011/06/13/from-couch-potato-to-quantified-self-this-journey-must-be-defined-and-encouraged/

And last but not least, a medical student at Upstate Medical Center in Syracuse, New York predicts what medicine will look like in 20 years when he will still be practicing and I will be 79 if I am so lucky:

http://adjacentpossiblemed.blogspot.com/2011/04/healthcare-in-20-years-in-500-words.html

I will continue wandering around the information in my world, and I will continue to try to make meaning of my place in the world by writing blogs. Dork writes, “The flâneur is an urban wanderer, who leisurely walks through streets and squares interpreting and re-imagining the city. Following the flâneur’s attitude toward the city, the information flâneur sees beauty and meaning in growing information spaces. By envisioning the information flâneur as a curious, creative, and critical persona, we promote a shift from negative concepts such as needs and problems towards positive information experiences.” I just thought I was trying to figure what is going to happen in healthcare in these confusing time

AI Experts On Robots and Living with Thinking Machines

On Saturday June 4, 2011 I took a Megabus from Philadelphia (one-way fare $8.00) to Hunter College in New York City to attend Man-Made Minds: Living With Thinking Machines, a World Science Festival program.

Rodney Brooks, who until recently was Panasonic Professor of Robotics at MIT and who now is Chief Technology Officer at Heartland Robotics, was the first expert to speak, and he emphasized a very practical approach that was not too concerned with any negative consequences of living with thinking machines. Brooks comes across as the typical, can do engineer who wants to revitalize American manufacturing by providing workers with robotic tools. For Brooks everything from ant behavior to robot behavior to human behavior comes down to simple rules and computation. I thought he did a nice job of explaining the concept of the “uncanny valley.” Although he did not mention Masahiro Mori who coined the term in 1970, Brooks discussed how the “uncanny valley” idea holds that when avatars look and act almost, but not perfectly, like actual humans, humans respond negatively to the robot. The " uncanny valley" is a dip in the graph of the positivity of human reaction as a function of a robot's lifelikeness. (http://en.wikipedia.org/wiki/Uncanny_valley) Brooks also showed videos of Kismet and Domo that highlighted how important eye contact is for sociable humanoid robots. Here is an article where Aaron Edsinger, Domo’s developer, talks about how eyes are critical for human to robot interaction (http://www.livescience.com/1419-robot-eyes-humans-human-eyes.html). Here is a simple overview video of Brooks on robots (http://www.youtube.com/watch?v=C9p8B7-5MTI), and Steve Talbott has a nice article on the Brooks approach to robotics here (http://www.southerncrossreview.org/43/talbott-hal.htm).

David Ferrucci described the four-year Watson/Jeopardy! Project that involved about 20 people. He explained in a clear and concise way how Watson takes a question, analyzes the question, generates hypotheses, collects and evaluates evidence, weighs and combines the final answer to reflect a level of confidence, and then provides the answer. Here is a 20 minute video that goes into more detail than Ferrucci did at Hunter College in New York yesterday (http://www.youtube.com/watch?v=3G2H3DZ8rNc) I also can recommend Stephen Baker’s book Final Jeopardy: Man vs. Machine and the Quest to Know Everything (http://www.amazon.com/Final-Jeopardy-Machine-Quest-Everything/dp/0547483163/ref=sr_1_5?s=books&ie=UTF8&qid=1307315691&sr=1-5).

Eric Horvitz of Microsoft was perhaps the most relevant to my interest in robots and the future of medicine. His demonstration of the avatar assistant on a computer screen outside his office really drove home how powerful robots can be. Because this computer program has been learning about Eric for over 10 years, she can predict that he will not be attending a meeting that is on his schedule for 4:00 PM, and she offers that time to a colleague (who she recognizes by name) who needs to talk to Eric. This New York Times article shows a picture of the computer assistant and describes how capable she is in helping Eric run his life (http://www.nytimes.com/2010/06/25/science/25voice.html?pagewanted=1) Horvitz’s homepage has a great list of papers for the general reader that I have started to sample (http://research.microsoft.com/en-us/um/people/horvitz/layreader.htm). Horvitz showed an impressive tape of a medical kiosk where an avatar speaks directly and convincingly to both the mother and the child who has diarrhea. After asking several questions, the avatar decides the child is not sick enough for immediate attention and makes an appointment for the child with his doctor later in the week. The moderator Faith Salie got a big laugh when she noted that the avatar was much more compassionate in relating to the child and his mother than the human triage nurses she has encountered in New York City Hospitals.

Hod Lipson who directs the Creative Machines Lab at Cornell University’s School of Mechanical and Aerospace Engineering and Computing and Information Science provided the most futuristic and revolutionary perspective on living with thinking machines. Lipson’s development of evolutionary robotics where machines self-replicate and self-assemble provides a new lens to look at what is consciousness, what is self-awareness and what it is to be human. What fascinated me was his simulation of evolution to allow machines to learn without any human developers or programming. The machines were able to learn because they probed their external world and only those that were successful get to breed new generations of machines. I must admit that I was blown away by this approach having the robots “discover” or “learn” basic laws of physics like Force = Mass X Acceleration. It is also hard to wrap my mind around these robots discovering basic laws of the universe that we humans cannot yet understand ourselves. The NY Times has a nice article about this amazing idea (http://www.nytimes.com/2009/04/07/science/07robot.html?scp=1&sq=%22april%202,%202009%22%20hod%20lipson&st=cse). For a video where he demonstrates some of this approach please go to http://www.youtube.com/watch?v=lMkHYE9-R0A.

The moderator Salie seemed to be shocked that all the panelists appeared to endorse the prediction that eventually robots will be built that will be hard to tell apart from human beings. Only Horvitz seemed concerned about unintended consequences which concerns me. Horvitz at least brought up two obvious candidates for unintended consequences: Criminal Artificial Intelligence and liability issues around autonomous systems that cause unexpected damage to humans or property.

The evening ended with each expert providing a take home message. Rodney Brooks, ever the practical, optimistic engineer, emphasized robots increasing our manufacturing productivity. David Ferrucci observed that computers and AI will allow us to learn and make new discoveries from the large amounts of data now swamping us humans. Eric Horvitz, the most thoughtful and philosophical of the bunch, said robots will teach us who we are as humans. Hod Lipson described AI as a new lens to look at what consciousness is, what self-awareness is, and what it means to be human.