On July 27, 2016, the Centers for Medicare and Medicaid Services (CMS) publicly released its five-star hospital quality rating system,¹ to much pushback from the healthcare industry.² CMS developed its rating system, built from metrics utilized in its Hospital Compare database, in an effort to support consumer decision-making on hospital choice.³ However, critics of the ratings system, such as the American Hospital Association (AHA), suggest that the system may be misleading for consumers, as it may not accurately reflect the quality of care provided by hospitals,⁴ and may be negatively biased in rating hospitals that serve underprivileged patients⁵ or relatively more complex cases.⁶ The difference between the reputations of hospitals and CMS’s rating may have implications for both hospitals and consumers, such as negative press attention for hospitals⁷ and inaccurate information for consumers.⁸ This Health Capital Topics article provides a brief overview of the development of CMS’s five-star rating system, and explores potential implications for hospitals and consumers affected by the perceived shortcomings of the new system.

CMS’s five-star hospital rating system is an aggregation of 64 of the over 100 metrics for quality of care within hospitals currently reported on Hospital Compare, a publicly accessible database of various healthcare metrics.⁹  Hospital Compare initially reported ten (10) quality measures (e.g., heart attack, heart failure, pneumonia, surgical care),¹⁰ but has since expanded to include data from the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey, which measures consumers’ views on hospital care.¹¹ Further, the database added measures related to a hospital’s 30-day mortality rate for heart attack and heart failure patients, as well as mortality rates for pneumonia,¹² and 30-day readmission measures for heart attack, heart failure, and pneumonia.¹³ CMS added voluntary data from hospitals participating in the American College of Surgeons National Surgical Quality Improvement Program,¹⁴ and the Hospital Value-Based Purchasing Program.¹⁵ CMS selected variables for the five-star ratings system by excluding those measures that were: discontinued; duplicates of other measures; ambiguous; or, not reported by a sufficiently large number of hospitals.¹⁶

CMS calculates the star ratings through a five-step process:

1. Selection and standardization of measures for inclusion in the rating system;
2. Assignments of measures to seven groups:

   1. Mortality;
   2. Safety of Care;
   3. Readmission;
   4. Patient Experience;
   5. Effectiveness of Care;
   6. Timeliness of Care; and,
   7. Efficient Use of Medical Imaging;
3. Calculation of group scores;
4. Calculation of hospital summary scores as a weighted average of group scores; and,
5. Application of a clustering algorithm to categorize summary scores into the rating system.¹⁷

Mortality, safety of care, readmission, and patient experience were each weighted at 22 percent, while effectiveness of care, timeliness of care, and efficient use of medical imaging were each weighted at four
(4) percent.¹⁸

The development of the five-star rating system built from Hospital Compare measurements has faced significant roadblocks since the announcement of the system in October 2014.¹⁹ Beginning on April 16, 2015, CMS posted quarterly hospital quality star ratings based on patients’ experience of care (via HCAHPS)²⁰ on the Hospital Compare website,²¹ but refrained from releasing an aggregate, five-star rating system at the request of Congress and the healthcare industry due to confusion surrounding the methodology of the rankings.²² In July 2015, CMS conducted a “hospital dry run” of the five-star rating system to provide stakeholders the opportunity to review the methodology behind its particular rating through a hospital-specific report and user guide,²³ in addition to an academic review of the methodology, communication with hospital representatives, all while delaying its release to the public.²⁴

On July 27, 2016, CMS issued its first public release of the overall hospital quality star rating on Hospital Compare,²⁵ generating significant criticism from the hospital industry.²⁶ CMS ranked 3,617 hospitals overall, with 102 hospitals receiving the highest, five-star rating; 927 hospitals receiving four-star ratings; 1,752 hospitals receiving the average, three-star rating; 707 hospitals receiving two-star ratings; and, 129 hospitals receiving the lowest, one-star rating.²⁷ 1,042 hospitals were not rated (including all hospitals in Maryland due to lack of collected information by CMS).²⁸

Almost immediately, CMS’s five-star rating system faced pushback from the hospital industry, primarily because many of the hospitals “considered as the nation’s best by private rating sources…or viewed as the most elite within the medical profession” scored poorly according to CMS, and many lesser-known hospitals received higher ratings than expected.²⁹ As a comparison, U.S. News & World Report recognizes the top 20 hospitals nationally by featuring an “Honor Roll,” a points-based system that measures the quality of care provided by a hospital.³⁰ The points system uses 12 data-driven specialty rankings, four reputation-only rankings, and a measure of procedures and conditions, which are aggregated across each hospital to build the “Honor Roll” rankings.³¹ The top five hospitals in the “Honor Roll,” along with recorded points and CMS star rating³² in parentheses, are as follows:³³

1. Mayo Clinic, Rochester, Minnesota (418 points, 5 stars);
2. Cleveland Clinic, Cleveland, Ohio (378 points, 4 stars);
3. Massachusetts General Hospital, Boston, Massachusetts (371 points, 4 stars);
4. Johns Hopkins Hospital, Baltimore, Maryland (349 points, rating not issued due its location in Maryland); and,
5. UCLA Medical Center, Los Angeles, California (331 points, 2 stars).

As the above list shows, only one top-five hospital (according to U.S. News & World Report) received a five-star rating from CMS (i.e., Mayo Clinic), while UCLA Medical Center, the fifth-ranked hospital according to the “Honor Roll,” received the second lowest rating of two stars from CMS. This trend holds for top hospitals not included in this list, including a significant amount of major academic medical centers that scored poorly.³⁴ Approximately 60 percent of teaching hospitals received below-average scores, and formed one-third of the facilities that received a one-star rating.³⁵

Kate Goodrich, M.D., director of the CMS Center for Clinical Standards and Quality, suggested that “specialized and cutting-edge care…are not reflected in [the] quality ratings.”³⁶ Elizabeth Mort, M.D., senior vice-president for quality and safety at Massachusetts General Hospital, suggested that perhaps the inclusion of 30-day readmission rates in the methodology, which may negative bias high performing hospitals in survival (i.e., mortality),³⁷ casting doubt on the methodology that seems “significantly flawed.”³⁸ Further, CMS’s methodology does not adjust for socio-economic status³⁹ or patient complexity, both of which were alleged to negatively influence the ratings of safety net hospitals and teaching hospitals.⁴⁰ Although it is unclear how such ratings will influence patient volumes at hospitals, the five-star rating system does not directly affect Medicare reimbursement through payment bonuses or penalties because such reimbursement is predicated on some of the underlying data in which the five-star rating system aggregates, such as the 30-day readmission rates.⁴¹

This enhanced transparency through CMS’s star ratings system may potentially affect consumers’ decision-making. CMS developed its rating system as an effort to improve consumers’ information for choosing hospitals;⁴² however, industry members have argued that the ratings system may overgeneralize the quality of care provided by hospitals. Clay Dunangan, M.D., Senior Vice President and Chief Clinical Officer of BJC HealthCare in St. Louis, stated:

“The star rating system is an oversimplification of performance data and [is not] very helpful for consumers trying to identify the best provider for their specific healthcare needs…[the data] is misleading and does not serve the public with all the facts.”⁴³

CMS plans to update the rankings quarterly, and has stated it will consider public feedback in an effort to improve the system.⁴⁴ Currently, it is uncertain what steps CMS will take, if any, to improve the perceived inaccuracy of Hospital Compare’s new star rating system, including whether modifications will occur after a period of input by consumers and hospitals.