Measuring Patients' Experiences with Individual Primary Care Physicians: Results of a Statewide Demonstration Project

×

In the Literature

In the U.S. health care market, there is growing interest in surveying patients to assess their experiences not only with health plans but with individual physicians. Some thorny issues hover around the feasibility and merit of such measurements: What sample size is needed to ensure the data gathered is reliable? Can the experiences of a doctor's patients from one health plan be generalized to those of patients in other plans? How much of a patient's experience is actually attributable to the influence of individual doctors as opposed to other elements, including the practice, the larger medical group, or the health plan?

In "Measuring Patients' Experiences with Individual Primary Care Physicians" (Journal of General Internal Medicine, January 2006), a team of researchers led by Dana Gelb Safran, Sc.D., of Tufts University, analyzed the most extensive effort to date to assess patients' experiences with their doctors through a survey: the Massachusetts Ambulatory Care Experiences Survey (ACES) project, a statewide pilot conducted in 2002–2003. According to Safran, ACES not only demonstrates the feasibility of obtaining highly reliable measures of patients' experiences with individual physicians and practices, but also yields findings that emphasize the importance of looking beyond health plans to improve overall quality of health care delivery.

About the ACES Project
The ACES project involved a collaboration among five commercial health plans and one Medicaid plan. The survey was designed to measure two broad dimensions of patient experiences: quality of the physician–patient interactions and organizational/administrative features of the process of obtaining health care. Only those practices with at least two adult generalist physicians were eligible; of those, only practices where two-thirds of the physicians had at least 75 patients in each of two or more commercial plans were included. In all, 9,625 patients, representing 215 generalist physicians at 67 practice sites, responded.

Feasibility and Value Demonstrated
Safran's analysis goes a long way toward quelling many of the primary concerns associated with measuring physicians' performance. Perhaps most importantly, the ACES measures achieved high levels of data reliability with a sample size of 45 patients per physician. In other words, with relatively modest sample sizes, a view of the individual physician emerged that is consistent across patients. The required sample sizes are easily attainable because any active patient in a physician's panel is a qualified participant. This contrasts with previous physician-level measures of clinical quality that were limited by the difficulty of achieving adequate sample sizes because measures only applied to a subset of a physician's practice—diabetic patients or patients with asthma, for example.

The authors also introduced a methodology to ensure minimal risk of misclassification. First, they limited the report format to only three performance categories. Second, they created buffers, or "zones of uncertainty," around the lines demarcating performance thresholds (i.e., cutpoints). The buffering is necessary, the authors point out, because even with exceedingly high measurement reliability, a score that is adjacent to a performance cut-point has a 50–50 chance of landing on the wrong side of the line. The authors advise that the design of any quality reporting protocol should limit the number of performance categories and devise systems to handle fairly cases closest to performance cutpoints.

As to whether performance issues can be fairly ascribed to physicians themselves (as opposed to the systems in which they work and the patients they care for), the project found that individual physicians and practice sites accounted for the vast majority of the variability in patients' care experiences. Network organizations and health plans exhibited little apparent influence; these findings bear out previous studies of practice site, network, and health plan influence on patients' experiences.

Data collection costs of the study indicate that obtaining comparable information for adult primary care physicians statewide would cost about 50 cents per adult resident, with extrapolated per capita costs across all U.S. adults appearing similar.

Policy Implications
Average performance scores across the physician population spanned more than 20 points out of 100. Meaningful improvement could likely be accomplished simply by working to narrow this differential. The well-documented benefits of high-quality clinician–patient interactions—including patients' adherence to medical advice, improved clinical status, loyalty to a physician's practice, and reduced malpractice litigation—suggest the value of doing so.

With considerable national attention focused on providing patient-centered care, Safran's analysis demonstrates the feasibility of obtaining highly reliable data on patients' experiences with their individual physicians and practices. The finding that doctors and practice sites account for the majority of variance in patients' reports indicates the appropriateness of focusing on these factors to improve the quality of our health care system.

Facts and Figures

  • Medicaid enrollees reported shorter primary care relationships than commercially insured individuals but otherwise reported similar primary care experiences.
  • The estimated cost per U.S. adult of measuring experiences with individual physicians was 50 cents.
  • The cost per completed ACES survey was $10 for commercial plans and $24 for the Medicaid plan.

Publication Details

Publication Date: January 1, 2006
Authors: Dana Gelb Safran, Melinda Karp, M.B.A., Kathryn Coltin, M.P.H. et al.
Citation:

D. G. Safran, M. Karp, K. Coltin et al., Measuring Patients' Experiences with Individual Primary Care Physicians: Results of a Statewide Demonstration Project, Journal of General Internal Medicine, January 2006 21(1):13-21

Forgot Password

An email has been sent to {{email}} with your reset password.

Account was not found. Please go to the login page and enter a new password.

Account was not found. Please try again with a different email address.

There was an error when attempting to send to {{email}}. Please contact an administrator.