Overview

A century ago, Boston surgeon Ernest A. Codman recognized that hospitals need to track their surgical outcomes in order to make improvements. The American College of Surgeons has begun offering hospitals an opportunity to participate in a nationwide program that, through the measurement and improvement of risk-adjusted surgical outcomes, could finally realize Codman's vision.

Issue: Post-operative complications impose a substantial mortality and morbidity burden on patients and excess costs on society. One approach to improving surgical outcomes is to promote widespread adherence to process-of-care measures, such as the timely administration of prophylactic antibiotics. Another approach is to encourage referrals to high-volume surgical centers, which may be more likely to deliver high-quality care. A third approach, first proposed by Ernest Codman more than a century ago, is to measure hospitals' surgical outcomes and identify any deficiencies. This approach has been used in a handful of state and regional cardiac surgery programs, such as the New York State Cardiac Surgery Reports and the Northern New England Cardiovascular Disease Study Group, but has not been applied more broadly. The Veterans Administration (VA) has a successful program for measuring and improving the outcomes of noncardiac surgery that has generated interest among non-VA hospitals.

Objective and Intervention: The American College of Surgeons created a national program to measure and improve risk-adjusted surgical outcomes using validated methods first developed and proven in VA hospitals. This program is open to all interested private-sector hospitals that pay a participation fee and meet other participation criteria (see Process below).

Organization: The American College of Surgeons (ACS) is a scientific and educational association of 65,000 surgeons. It was founded in 1913 to improve the care of surgical patients by setting high standards for surgical education and practice.

Implementation Milestones: Responding to public concern about the quality of surgical care in veterans' hospitals, Congress in 1986 mandated that the VA report risk-adjusted surgical outcomes compared with a national average. Because no such methods or data existed, the VA had to develop them. This effort culminated in 1994 with the creation of a National Surgical Quality Improvement Program (NSQIP). Its role is to measure and report comparative risk-adjusted surgical outcomes at the institutional level and to promote improvement through self-assessment, site visits, and dissemination of best practices. In 1999, the VA undertook a feasibility study at three academic medical centers and concluded that NSQIP methods were transferable to non-VA hospitals [1]. The ACS then collaborated with the VA to implement NSQIP at 14 academic hospitals through a pilot project funded by the Agency for Healthcare Research and Quality; four community hospitals later joined this private sector initiative. Based on the pilot's success, ACS launched NSQIP nationally in October 2004 with an initial focus on general and vascular surgery. To date, 37 private sector hospitals have committed to participate in the ACS NSQIP.

Key Measures: NSQIP collects data on 40 preoperative clinical risk factors (e.g., diabetes and heart disease), 20 categories of 30-day postoperative morbidity (e.g., venous thrombosis, wound infections, and pneumonia), and 30-day postoperative mortality on patients having major operations under general, spinal, or epidural anesthesia. Benchmark reports allow institutions to compare their surgical volume, patient risk profiles, outcomes, and length-of-stay to the average for all institutions and those in a peer group. An observed-to-expected (O/E) ratio is calculated for each institution. This is a risk-adjusted ratio of the observed number of deaths and complications divided by the number expected. Risk adjustment is based on a validated regression model that is recalibrated annually. An O/E ratio greater than one indicates that the hospital is experiencing more deaths and complications than expected based on its patients' risk factors; a ratio of less than one indicates better-than-expected outcomes. All results are de-identified so that hospitals can only view their own performance data.

Process: Each institution designates a surgeon to oversee its participation in the program and must hire a full-time, surgical clinical nurse reviewer to prospectively collect clinical data. This information is transmitted via a secure Web-based portal to the University of Colorado Health Outcomes Program, which performs data analysis and risk-adjustment. A third-party contractor, QCMetrix, conducts ongoing training for nurse reviewers and provides informatics, data management, and consulting services, including annual inter-rater reliability site visits. The ACS provides overall program management and oversight. An NSQIP advisory committee sets program standards and policies, interprets results for feedback to participants, and reviews requests to use the NSQIP database for research purposes. Best practices from hospitals with superior performance are identified and shared among participants. Hospitals with worse than expected outcomes are encouraged to conduct structured internal assessments and/or to undergo a site visit by external experts to identify opportunities and actions for improvement.

Results: The VA's validation study, conducted between 1991 and 1993, found that its methods for risk-adjusting outcomes accurately predict quality of care structures and processes. During the first nine years of data collection, the VA NSQIP documented a 27 percent reduction in 30-day postoperative mortality and a 45 percent reduction in 30-day postoperative morbidity for noncardiac surgery performed at 128 VA hospitals [2]. A preliminary analysis of the NSQIP private sector initiative indicates that the 14 participating hospitals achieved a significant decrease in surgical complications during the three-year study period. Individual hospitals used NSQIP data to identify problem areas and make improvements—reducing specific surgical morbidities such as postoperative wound infections and urinary tract infections.

Lessons Learned: Clinically meaningful data, standardized outcome definitions, and a validated risk-adjustment mechanism make NSQIP the gold-standard for quality measurement in surgery today. "We now have data to talk about the kinds of things that happen after surgery, and it's only through good data that you can improve," says Darrell Campbell, M.D., chief of clinical affairs at the University of Michigan Health System and chair of the NSQIP advisory committee. Professional society sponsorship brings credibility to help assure surgeons and hospitals that NSQIP has proper oversight, is used in a way that's fair, and is here to stay, he adds. Participation in NSQIP changes the nature of a hospital's surgical mortality and morbidity conference, in which surgeons discuss cases and identify bad outcomes. "This is where the rubber meets the road in a hospital," says Campbell. "In the past, we didn't have a good way of knowing" whether a bad outcome was "just a fluke" or part of a bigger problem. "With this system, we have very good data to show whether we are comparable to other hospitals of our type. If we see that we're an outlier in some way, then that's an area of focus."

The cost of participation in NSQIP might be viewed as a barrier to adoption, especially for smaller hospitals. Institutions must pay a $35,000 participation fee and fund the salary of a full-time nurse reviewer. However, given that a major surgical complication generates $11,600 in extra costs on average, a reduction in complications might offset a hospital's participation costs [3]. Also, the actual cost might be less for institutions with advanced clinical information systems capable of generating some of the data. Recognizing that the costs of surgical complications—and savings from their prevention—are shared by payers, Blue Cross Blue Shield of Michigan has agreed to defray a portion of the NSQIP participation fee for 15 Michigan hospitals. This "pay-for-participation" arrangement is based on expectations that the hospitals will collaborate to improve outcomes as a group, Campbell explains. The hospitals have agreed to share the grouped results with Blue Cross, but they are not required to disclose individual institutional performance.

N ext Steps: The ACS NSQIP team is planning to expand its focus from general and vascular surgery to subspecialty modules, starting with bariatric surgery. Other potential focus areas and disciplines include trauma and pediatric surgery. The NSQIP team also expects its repertoire of outcome measures to expand over time and will likely include long-term survival, functional outcomes, quality-of-life, patient satisfaction, and cost-effectiveness measures [2]. Campbell sees NSQIP as complementary to the Surgical Care Improvement Project (SCIP), which aims to reduce surgical complications by promoting adherence to process-of-care measures. The NSQIP program is seeking grant support to determine the correlation between surgical outcomes (as measured by NSQIP) and process measures. "If we don't see any concrete improvement in outcomes, then let's pick other process measures," says Campbell. Although public pressure for transparency eventually might persuade hospitals to publicly report their surgical outcomes, NSQIP will continue to focus on system-level factors that influence outcomes. Small sample sizes inhibit valid risk-adjusted comparisons of surgeon-specific performance, which is best evaluated in the traditional manner at the peer level, Campbell explains. The VA NSQIP program will continue in parallel, and the VA plans to compare its results with the ACS NSQIP private sector data as it expands nationally. Thus, these efforts fulfill the congressional mandate that the VA compare its performance to national benchmarks.

 Implications: The success of a voluntary, professionally led quality-improvement approach such as NSQIP is vital for sustaining confidence in professional self-regulation. "This is the age of accountability and we, as surgeons, need to be accountable for our performance," says Campbell. "If surgeons don't develop the outcome metrics that make sense and seem fair to them, then the government will come in and non-medical people will devise a system that will never get buy-in. [NSQIP] is something that was generated by surgeons for surgeons and patients. Providers and the payers will benefit but, ultimately, patients benefit if we decrease the incidence of complications." Will NSQIP catch on? "There's no question that it will be used more widely because it's the best validated tool for collecting outcomes from surgery," says David William Rattner, M.D., chief of gastrointestinal and general surgery at Massachusetts General Hospital, an NSQIP participant. "As people demand outcomes, this will be the standard that they refer to." Perhaps most important, surgeons are demanding this type of measurement. As a concrete example of their support, Campbell recalls that a surgeon stood up at a recent meeting and said that NSQIP is "'the most important advance in surgery in the last decade.' And I believe that's true. We've really needed a uniform system for quality measurement nationally."

For Further Information: Visit the ACS NSQIP Web site or contact Marchelle Werner, program coordinator for the ACS NSQIP.

References
[1] A. S. Fink et al. (2002) The National Surgical Quality Improvement Program in Non-Veterans Administration Hospitals: Initial Demonstration of Feasibility. Annals of Surgery 236, 344–53.

[2] S. F. Khuri et al. (2002) The Comparative Assessment and Improvement of Quality of Surgical Care in the Department of Veterans Affairs. Archives of Surgery 137, 20–7.

[3] J. B. Dimick et al. (2004) Hospital Costs Associated with Surgical Complications: A Report from the Private-Sector National Surgical Quality Improvement Program. Journal of the American College of Surgeons 199, 531–7.

This study was based on publicly available information and self-reported data provided by the case study institution(s). The aim of Fund-sponsored case studies of this type is to identify institutions that have achieved results indicating high performance in a particular area, have undertaken innovations designed to reach higher performance, or exemplify attributes that can foster high performance. The studies are intended to enable other institutions to draw lessons from the studied organizations' experiences in ways that may aid their own efforts to become high performers. The Commonwealth Fund is not an accreditor of health care organizations or systems, and the inclusion of an institution in the Fund's case studies series is not an endorsement by the Fund for receipt of health care from the institution.