ORIGINAL SCIENTIFIC ARTICLE Postoperative Myocardial Infarction in Administrative Data vs Clinical Registry: A Multi-Institutional Study David A Etzioni, MD, MSHS, FACS, Cynthia Lessow, RN, Liliana G Bordeianou, MD, FACS, Hiroko Kunitake, MD, FACS, Sarah E Deery, MD, MPH, Evie Carchman, MD, FACS, Christina M Papageorge, MD, George Fuhrman, MD, FACS, Rachel L Seiler, MPH, James Ogilvie, MD, FACS, Elizabeth B Habermann, PhD, MPH, Yu-Hui H Chang, PhD, Samuel R Money, MD, FACS Previous studies have documented significant differences between administrative data and registry data in the determination of postoperative MI. The goal of this study was to characterize discordance between administrative and registry data in the determination of postoperative myocardial infarction (MI). STUDY DESIGN: This study was performed using data from the American College of Surgeons NSQIP merged with administrative data from 8 different hospitals, between 2013 and 2015. From each of these sources, the occurrence of a postoperative MI, as ascertained by administrative data and NSQIP data, were compared. In each situation in which the 2 sources disagreed (discordance), a 2-clinician chart review was performed to generate a “gold standard” determination as to the occurrence of postoperative MI. RESULTS: A total of 43,289 operations met our inclusion criteria for analysis. Within this cohort a total of 230 cases of MI were identified by administrative data and/or NSQIP data (administrative rate 0.41%, NSQIP rate 0.42%). A total of 89 discordant ascertainments were identified, of which 42 were adminþ/NSQIP- and 47 were admin-/NSQIPþ. Accuracy (99.9% for both) and concordance (kappa ¼ 0.89 [95% CI 0.86 to 0.92] for administrative data, kappa ¼ 0.87 [95% CI 0.84 to 0.91] for NSQIP data) of the 2 systems were similar when compared against our gold standard (chart review). The majority of errors were related to false negatives, with sensitivity rates of 81% in both data sources. CONCLUSIONS: In this multi-institutional study, administrative data and NSQIP demonstrated a similar ability to determine the occurrence of postoperative MI. These findings do not demonstrate an advantage of registry data over administrative data in the determination of postoperative MI. (J Am Coll Surg 2017; -:1e8. 2017 by the American College of Surgeons. Published by Elsevier Inc. All rights reserved.) BACKGROUND: Postoperative myocardial infarction (MI) occurs in 1.4% to 5.0% of patients undergoing noncardiac surgery, and is associated with a 30-day mortality rate of 11.6%.1,2 As a potentially preventable adverse outcome of surgery, MI is a valid target of research and quality improvement aimed at minimizing patient-level risk. Given the importance of MI as a focus of analysis, the mechanisms used to ascertain the occurrence of postoperative MI deserve attention. The occurrence of a postoperative MI can be identified on the basis of data collected for administrative/ billing purposes (eg Medicare data, discharge data) or registry data that are abstracted prospectively/separately for the purposes of quality measurement. Each of these 2 mechanisms relies on trained coders to translate an analog medical record into a digital representation (discharge abstract). Disclosure Information: Nothing to disclose. Support for this study: This publication was made possible by funding from the Mayo Clinic Robert D and Patricia E Kern Center for the Science of Health Care Delivery. Received August 22, 2017; Accepted September 19, 2017. From the Departments of Surgery, Mayo Clinic Arizona, Phoenix, AZ (Etzioni, Lessow, Money); Massachusetts General Hospital, Boston, MA (Bordeianou, Kunitake, Deery); University of Wisconsin, Madison, WI (Carchman, Papageorge); Ochsner, New Orleans, LA (Fuhrman); University of Queensland Medical Center, Queensland, Australia (Seiler); and Spectrum Health Medical Center, Grand Rapids, MI (Ogilvie); and the Mayo Clinic Robert D and Patricia E Kern Center for the Science of Health Care Delivery, Surgical Outcomes Program, Rochester, MN (Etzioni, Lessow, Habermann, Chang). Correspondence address: David A Etzioni, MD, MSHS, FACS, Department of Surgery, 5777 E Mayo Blvd, Phoenix, AZ 85054. email: etzioni. [email protected] ª 2017 by the American College of Surgeons. Published by Elsevier Inc. All rights reserved. 1 https://doi.org/10.1016/j.jamcollsurg.2017.09.016 ISSN 1072-7515/17 2 Etzioni et al Data and Postoperative Myocardial Infarction The relative ability of these 2 approaches to get the diagnosis of a postoperative MI “right” is unknown. What is known is that when compared, these 2 systems differ considerably. In 1 study comparing Medicare data with data from the NSQIP, 84% of postoperative MIs detected in Medicare data were not considered MIs in the NSQIP dataset.3 This level of discordance is troubling, especially given the fact that data from both of these sources are used for public reporting.4,5 Medicare data are additionally used as the basis for value-based purchasing, and have the potential to redirect a significant portion of domestic health care expenditure and patient volume.6 With this study, we analyzed the discordance between administrative and NSQIP data in the determination of postoperative MI. Our approach was not hypothesis driven, but instead, sought to characterize and apportion the potential reasons why the ascertainment of MI differs between registry data and administrative data. METHODS Cohort The unit of observation for this analysis was a surgical hospitalization, occurring in 1 of 8 distinct hospitals, between 2013 and October 2015. In developing this cohort, data from the American College of Surgeons (ACS) NSQIP were used as a starting point. Within the ACS NSQIP, participating hospitals contribute data regarding a random sample of their patients to the ACS, including preoperative/intraoperative factors, and 30-day postoperative complications.7 At each site, ACS NSQIP data were merged with 2 specific elements of hospital discharge data: diagnosis codes (using ICD-9 scheme) and present on admission (POA) classifications relevant to each code. Definition of complication Within administrative data, the occurrence of a postoperative MI was based on the ICD-9 codes 410.X0 (myocardial infarction, episode of care unspecified) and 410.X1 (myocardial infarction, initial episode of care). This set of codes was selected because it matches the codes analyzed in another recent comparison of administrative vs NSQIP-ascertained complications.3 Within NSQIP data, the occurrence of a postoperative MI was determined based on the variable, “intraoperative or postoperative myocardial infarction.” The occurrence of an MI was based on clinical criteria as defined in the ACS NSQIP Operations Manual. Contents of this manual are considered proprietary, but an abridged version is available online.8 The essence of this definition is: An acute myocardial infarction that occurred intraoperatively or within 30 days after surgery, as manifested by 1 of the following: J Am Coll Surg Documentation of ECG changes indicative of acute MI (1 or more of the following): ST elevation > 1 mm in 2 or more contiguous leads, new left bundle branch, and/or new q-wave in 2 of more contiguous leads; new elevation in troponin greater than 3 times upper level of the reference range in the setting of suspected myocardial ischemia; or physician diagnosis of myocardial infarction. The definition adds that a cardiology consultation that determines an absence of ischemia/MI is sufficient to “overrule” other aspects of the patient’s determination. Data from NSQIP include not only occurrence of the MI, but also the first date on which an MI was diagnosed. An MI diagnosed postoperatively and before discharge was therefore considered a postoperative MI. The primary analysis of this study focuses on agreement between administrative data and NSQIP data. Administrative data, by definition, is oblivious to post-discharge complications. Therefore, any MI diagnosed post-discharge in the NSQIP dataset was considered a non-MI (for NSQIP data). Chart review Every operation with discordance between administrative data and NSQIP data regarding the occurrence of a postoperative MI was reviewed within each participating hospital using an abstraction form. Abstracted data included presence of a preoperative MI, highest troponin value, date of highest troponin value, whether a clinical diagnosis of MI was made, the date of such diagnosis, whether a cardiology consultation was obtained, the date of the consultation (if one occurred), and the final cardiology diagnosis (if cardiology consultation occurred). Each of these reviews was performed independently by 2 clinicians experienced in perioperative care, at least 1 of whom had to be an attending surgeon. Surgical case reviewers (SCRs) from NSQIP could not be reviewers. Reviewers were blinded to the type of discordance and the date of the occurrence within the NSQIP data. After the dual review was completed, the 2 reviews were reconciled. In this reconciliation, local reviewers were asked to consider each data element that differed between the 2 reviewers. In cases in which consensus was unable to be reached, the data element was considered “uncertain/unclear.” Exclusion criteria Operations performed on an outpatient basis (admit date ¼ discharge date) were excluded, as were those in which the patient was discharged on October 1, 2015 or later. This date-based exclusion was necessary because after this date, the ICD-10 scheme replaced the ICD-9 scheme in most hospital discharge datasets. Within administrative data, the POA indicator was capable of identifying patients who had an MI that was considered present on admission. When such an indicator was present, Vol. -, No. -, - 2017 Etzioni et al Data and Postoperative Myocardial Infarction 3 Figure 1. Cohort flowchart. Admin, administrative; MI, myocardial infarction; POA, present on admission; postop, postoperative; preop, preoperative. the hospitalization was excluded because it was (by definition) no longer possible to determine the occurrence of a postoperative MI within the administrative dataset. Within NSQIP data, postoperative occurrences more than 30 days after the index operation are not abstracted. Therefore, any MI that occurred during the index hospitalization and more than 30 days postoperatively were excluded from analysis. Statistical analysis The primary goal of this study was to analyze concordance between different mechanisms of ascertaining a specific complication. Description of concordance was performed using Cohen’s kappa, which can range between -1 (very poor agreement) and 1 (perfect agreement). Accuracy, sensitivity, and specificity were measured according to standard epidemiologic estimations.9 Comparisons of these tests characteristics were performed using either McNemar’s test or using a weighted generalized score statistic.10,11 Statistical analyses were conducted with SAS version 9.4 (SAS Institute Inc). Institutional review This research study was reviewed by the IRB of each of the participating hospitals. RESULTS Cohort After applying our inclusion and exclusion criteria (Fig. 1), the total cohort for analysis included 43,289 patients, with individual hospitals contributing between 3,203 and 8,163 observations to the analysis. Cohort characteristics are included in Table 1. The average age (SD) of analyzed patients was 59.5 15.8 years, and 53.6% were women. Unadjusted myocardial infarction rates, concordance Unadjusted rates of postoperative in-hospital MI in the 2 datasets were similar, with 0.42% for administrative data and 0.41% for NSQIP. The Cohen’s kappa value comparing the evaluations from administrative vs NSQIP determination was 0.71 (95% CI 0.66 to 0.77). A total of 230 cases of MI were identified by administrative data and/or NSQIP data. Of these 230 cases, 128 cases (56%) were considered an MI by both systems. Chart review Our methods identified 102 cases of discordance, and each of these underwent a dual-clinician chart review. 4 Etzioni et al Table 1. Patient Population Data and Postoperative Myocardial Infarction Characteristic Total n Age, y, mean SD Sex, n (%) Male Female Missing Surgical subspecialty, n (%) Cardiac General surgery Gynecology Neurosurgery Orthopaedics Otolaryngology Plastics Thoracic Urology Vascular Length of stay, d Mean SD Median Unadjusted MI diagnosis rate, % Administrative data NSQIP data Year of operation, n (%) 2013 2014 2015 Data 43,276 59.5 15.8 19,919 (46.0) 23,067 (53.3) 290 (0.7) 81 15,593 2,456 3,157 9,448 2,271 1,888 1,034 4,223 3,125 (0.2) (36.2) (5.7) (7.3) (21.8) (5.3) (4.4) (2.4) (9.8) (7.2) 4.28 8.4 3 0.42 0.41 15,398 (35.6) 15,924 (36.8) 11,954 (27.6) MI, myocardial Infarction. Of these 102 cases, 13 were excluded from further analysis due to evidence that a recent MI was part of the patient’s preoperative risk profile. Of the remaining 89 discordant cases, 42 were “type A discordance,” for which administrative data identified postoperative MI that NSQIP did not. Another 47 were “type B discordance,” for which NSQIP data identified a postoperative MI that administrative data did not. Each of these 89 cases underwent a dual-clinician review. Of these 89 discordant cases, the vast majority (79 of 89) were considered false negatives based on this review and a minority (10 of 89) were considered false positives. Based on the chart review, each discordant record could be categorized into 1 of 9 mutually exclusive groups (Table 2). In the vast majority of false negatives, a cardiology consultation determined either MI or ischemia in the presence of elevated troponins (group 1a). Of the false positives, 8 cases were considered by the NSQIP reviewers to be MI despite a cardiology evaluation that determined neither MI nor ischemia to be present (group J Am Coll Surg 3a). Two false positives were considered an MI by administrative mechanisms despite normal troponins and the absence of a clinical diagnosis in the record (group 3b). False positives/negatives and accuracy In order to compute the accuracy, sensitivity, and specificity for administrative vs NSQIP data, 3 important assumptions needed to be made. First, that every case in which both mechanisms (administrative data and NSQIP) ascertained an MI was truly an MI. Second, in every case in which both mechanisms determined that no MI occurred, no MI had occurred. Third, in situations of discordance that the 2-clinician chart review performed in this study was correct (“gold standard”). Based on these assumptions, the operational characteristics of administrative vs NSQIP data could be calculated, and these are reported in Figure 2. A comparison of administrative and NSQIP against the chart review (gold standard) demonstrates similar kappa values (p ¼ 0.45), accuracy (p ¼ 0.45), sensitivity (p ¼ 0.91), and specificity (p ¼ 0.06). Figure 3 displays unadjusted hospital-specific rates of MI (administrative, NSQIP) and concordance (kappa values) between administrative and NSQIP data. Sample sizes by site are not shown in order to preserve site identity. Sensitivity analyses The approach used in this study to identify complications using ICD-9 codes is only 1 of several potential strategies. In order to test the robustness of our approach, 2 sensitivity analyses were performed. First, the administrative codes used as the basis for ascertaining MI data were modified. The strategy outlined here, based on ICD-9 codes 410.X0 and 410.X1, is not the only viable approach. Code 410.X1 denotes a “first episode of care” for an MI. The 410.X0 code, however, denotes “episode of care unspecified” and may therefore not be appropriate in determining postoperative MI. The performance of administrative data using an approach that is restricted to ICD-9 codes 410.X1 was tested as alternative strategy #1. Second, the position of the MI diagnosis code within the list of diagnosis codes relevant to the hospitalization may be related to the accuracy of MI coding. Performance of administrative data in determining MI based only on the presence of an MI code in the first 3 listed diagnoses was tested as alternative strategy #2. The operational characteristics of these 2 alternative strategies relative to the baseline strategy are shown in Table 3. Neither of the alternative strategies performed superiorly to the basic strategy. Vol. -, No. Table 2. -, - 2017 Etzioni et al Data and Postoperative Myocardial Infarction 5 Categorization of Discordant Cases Chart review finding Type A discordance (Adminþ/NSQIP-), (n ¼ 42) n Type B discordance (Admin-/NSQIPþ), (n ¼ 47) n 29 3 3 3 29 3 1 5 0 1 2 0 0 0 0 2y 8* 0 False negative - group 1: troponin elevation greater than 3x upper limit of normal, and: 1a: Cardiology evaluation ¼ ischemia/MI 1b: Cardiology evaluation ¼ unclear/uncertain 1c: No cardiology evaluation, clinical diagnosis of MI 1d: No cardiology evaluation, no clinical diagnosis of MI 1e: No cardiology evaluation, clinical diagnosis of MI ¼ unclear/ uncertain False negative - group 2: no troponin elevation greater than 3x upper limit of normal and: 2a: Clinical diagnosis of MI/ischemia in chart (either cardiology or other physician) 2b: Clinical diagnosis of MI/ischemia unclear/uncertain False positive - group 3: MI excluded 3a: Troponin elevation, cardiology evaluation ¼ no MI, no ischemia 3b: No troponin elevation, no clinical diagnosis of MI in chart *NSQIP false positives; all other entries in this column are administrative false negatives. y Administrative claim false positives; all other entries in this column are NSQIP false negatives. Admin, administrative; MI, myocardial infarction. DISCUSSION A postoperative MI is a dramatic event, and one that has clear impact on the short-term and long-term outcomes for a surgical patient. Intuitively, the occurrence of an MI is dichotomous; either an MI occurs or does not. With this study, we assessed the performance of 2 competing mechanismsdadministrative data and registry datadin determining the occurrence of a postoperative MI. In this comparison, these 2 very different mechanisms had nearly identical operational characteristics. At some level, this finding should be reassuring. Both administrative data and NSQIP data are publicly reported and therefore they should give similar results. Unadjusted rates of postoperative in-hospital MI were 0.42% in administrative data and 0.41% in NSQIP data. Kappa values between these datasets and a gold standard (as defined in this study) were high, at 0.89 for administrative data and 0.87 for NSQIP. The sensitivity for detecting MI, however, was lower than what might be considered acceptable, at approximately 81% for both systems. Previous head-to-head comparisons of NSQIP data and administrative data have found dramatic differences in terms of their ascertainment of postoperative MI. In a comparison of 117,752 Medicare patients, Lawson and colleagues3 found that 84% of postoperative MIs within Medicare data were not considered MIs in NSQIP data. A similar, smaller study in the UK examined concordance between a National Health Service Database and NSQIP, and found that 59% of MIs recorded in an administrative dataset (1,323 operations) were not MIs in NSQIP. Our study found that 30% of administratively coded MIs were not considered MIs in the NSQIP dataset. It would be a mistake, however, to consider these proportions as estimates of a false positive rate within administrative datasets. Instead, these estimates are evidence of discordance between 2 different approaches to ascertaining the occurrence of postoperative complications. In this study, we undertook an intensive, 2-clinician review of each situation in which there was discordance between the 2 datasets in order to better understand the differences between these 2 datasets. We found that both approaches have similar sensitivity, specificity, and accuracy for determining that an MI occurred. In both of these datasets, the vast majority of errors (79 of 89, 89%) were the result of false negativesdsituations in which a coder/abstractor failed to ascertain an MI that was clearly present. Figure 2. Operational characteristics of administrative vs NSQIP determination of myocardial infarction relative to gold standard. Admin, administrative; MI, myocardial infarction. 6 Etzioni et al J Am Coll Surg Data and Postoperative Myocardial Infarction Figure 3. Concordance (administrative vs NSQIP data) by study site. Admin, administrative; MI, myocardial infarction. The approach taken in this study is more powerful than earlier investigations for 3 reasons. First, this study represents a review of the experience of 8 different hospitals, each with its own individual administrative and NSQIP coders. Therefore our findings are more representative of the national experience than a single-institution study would be. Second, each chart review conducted in this study was focused on a single complication, and therefore, the scrutiny applied to characterizing each occurrence of discordance was magnified. Third, an additional level of scrutiny was applied through our use of a dual-clinician review. This dual review, combined with the reviewer focus on a single complication, imparts a greater level of diagnostic precision than is present in either administrative or NSQIP data. Table 3. This study has several limitations. First and foremost, this study was not designed to examine the clinical and interventional algorithms that are used to diagnose and treat postoperative MI. Strategies to screen postoperative patients with a sensitive troponin assay, with the goal of stratifying and reducing MI risk, are an emerging area of research, but are not yet widely used in clinical practice.12 In this study, the clinical thresholds used to diagnose MI surely varied between sites, as well as between physicians within sites. The severity of MI was not analyzed, nor was the use of revascularization. These aspects of clinical care we considered separate from the goal of the studydto characterize the accuracy of existing methods in the determination of postoperative MI. Evaluation of ICD-9 Code Selection Criteria Strategy Basic strategy 1: Default definition of MI (ICD-9 codes 410.X1 and 410.X0) Alternate strategy 1: Restrict to ICD-9 codes 410.X1 Alternate strategy 2: MI code in first 3 diagnosis code positions only MI, myocardial infarction. Operational characteristics relative to gold standard Kappa (95% CI) Accuracy, % Sensitivity, % Specificity, % 0.89 (0.86e0.93) 99.9 81.2 99.9 0.57 (0.50e0.64) 99.7 40.1 99.7 0.67 (0.61e0.73) 99.8 50.7 99.8 Vol. -, No. -, - 2017 Etzioni et al Other limitations are also worth mentioning. In order to analyze rates of accuracy, an assumption was made that in all cases of agreement between administrative and NSQIP data, both datasets were “correct.” This assumption was necessary in order to focus on the relatively small number of cases in which there was discordance, but it is possible that some of these cases in which concordance was present were mis-classified by both data sources (“disconcordance”). The selection of which codes from within the administrative dataset were used to denote the occurrence of an MI is also a potential limitation. Several different approaches have been described,13 and the approach used in this study, when compared against 2 alternative strategies, yielded the best performance. Our study may also be criticized for its reliance on the ICD-9 coding scheme, which was phased out in 2015. There is no good reason to believe, however, that coding behavior under the ICD-10 scheme will be significantly different. Despite these limitations, this study stands to inform the ongoing movement toward increasing measurement, analysis, and reporting of surgical outcomes. Within this movement, administrative- and registry-based mechanisms have competing/complementary roles in informing clinical leaders regarding surgical quality of care. Regardless of which approach is used, some aspects of measurement can be strengthened. First, the status of any major event such as an MI should be clearly designated as present on admission (POA), present at time of surgery (PATOS), or postoperative. Both administrative data and NSQIP data have deficiencies in this area. Administrative data do not have the ability to designate whether a condition occurred between admission and operation (eg a patient who had an MI after admission but before surgery), so there is the potential for preoperative occurrences to be inaccurately considered as a complication. Data from NSQIP do not capture whether or not an MI was present at the time of surgery, and therefore may be underestimating risk in patients with a preoperative MI. In this analysis, a significant number of discordant cases had evidence of a preoperative MI (13 of 102, 13%). Second, the nature of clinical documentation should change to meet evolving information needs. Well-established criteria exist to diagnose MI.14 Clinical leadership needs to become more accustomed to applying these criteria in an explicit/categorical fashion so that translation of the medical record into a digital format is straightforward and less prone to errors in translation. With these steps, the ability of interested parties to better ascertain the occurrence of postoperative MI would be improved. These changes would be of significant benefit to patients, health care policy, and outcomes researchers. Data and Postoperative Myocardial Infarction 7 CONCLUSIONS This multi-institutional analysis of postoperative MI in 43,289 patients challenges accepted beliefs regarding the relative strengths and weaknesses of administrative and registry data. Registry data, such as NSQIP, are generally considered superior because of standardized training of clinical abstractors and the use of stringent clinical definitions to govern abstraction. These data come at a high price, however, in terms of personnel and program costs. Administrative data, which are generated primarily for billing purposes, are widely considered less accurate because of the absence of universally accepted criteria and variability in the clinical acumen of administrative coders/abstractors. The decision as to which of these 2 data sources to use as the basis for quality measurement is, theoretically, rooted in a tradeoff between cost and accuracy of data. As the quality movement progresses, a quantitative characterization of this tradeoff is critical. Author Contributions Study conception and design: Etzioni, Lessow, Habermann, Chang, Money Acquisition of data: Etzioni, Lessow, Bordeianou, Kunitake, Deery Carchman, Papageorge, Fuhrman, Seiler, Ogilvie Analysis and interpretation of data: Etzioni, Lessow, Bordeianou, Kunitake, Deery Carchman, Papageorge, Fuhrman, Seiler, Ogilvie Drafting of manuscript: Etzioni, Habermann, Chang, Money Critical revision: Etzioni, Lessow, Bordeianou, Kunitake, Deery Carchman, Papageorge, Fuhrman, Seiler, Ogilvie, Habermann, Chang, Money REFERENCES 1. Devereaux PJ, Xavier D, Pogue J, et al. Characteristics and short-term prognosis of perioperative myocardial infarction in patients undergoing noncardiac surgery: a cohort study. Ann Int Med 2011;154:523e528. 2. Devereaux PJ, Goldman L, Cook DJ, et al. Perioperative cardiac events in patients undergoing noncardiac surgery: a review of the magnitude of the problem, the pathophysiology of the events and methods to estimate and communicate risk. CMAJ 2005;173:627e634. 3. Lawson EH, Louie R, Zingmond DS, et al. A comparison of clinical registry versus administrative claims data for reporting of 30-day surgical complications. Ann Surg 2012;256: 973e981. 4. NSQIP Hospital Compare Data. Available at: https:// data.medicare.gov/Hospital-Compare/NSQIP-HOSPITALCOMPARE-DATA/c635-s3cy. Accessed March 8, 2017. 5. Medicare.gov Hospital Compare. Available at: https://www. medicare.gov/hospitalcompare/search.html. Accessed March 8, 2017. 8 Etzioni et al Data and Postoperative Myocardial Infarction 6. Hospital Value-Based Purchasing. Available at: https://www.cms. gov/Outreach-and-Education/Medicare-Learning-NetworkMLN/MLNProducts/downloads/Hospital_VBPurchasing_Fact_ Sheet_ICN907664.pdf. Accessed April 20, 2017. 7. Ingraham AM, Richards KE, Hall BL, Ko CY. Quality improvement in surgery: the American College of Surgeons National Surgical Quality Improvement Program approach. Adv Surg 2010;44:251e267. 8. ACS National Surgical Quality Improvement Program. Available at: http://site.acsnsqip.org/wp-content/uploads/2014/11/ ACS_NSQIP_PUF_User_Guide_2013.pdf. Accessed April 4, 2017. 9. Wikipedia. Confusion matrix. Available at: https://en. wikipedia.org/wiki/Confusion_matrix. Accessed April 18, 2017. J Am Coll Surg 10. Kosinski AS. A weighted generalized score statistic for comparison of predictive values of diagnostic tests. Stat Med 2013;32: 964e977. 11. McNemar Q. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika 1947;12:153e157. 12. Devereaux PJ, Biccard BM, Sigamani A, et al. Association of postoperative high-sensitivity troponin levels with myocardial injury and 30-day mortality among patients undergoing noncardiac surgery. JAMA 2017;317:1642e1651. 13. Mentz RJ, Newby LK, Neely B, et al. Assessment of administrative data to identify acute myocardial infarction in electronic health records. J Am Coll Cardiol 2016;67:2441e2442. 14. Thygesen K, Alpert JS, Jaffe AS, et al. Third universal definition of myocardial infarction. Circulation 2012;126:2020e2035.