вход по аккаунту



код для вставкиСкачать
A More Accurate Measure of the False-Negative Rate
of Papanicolaou Smear Screening is Obtained by
Determining the False-Negative Rate of the
Rescreening Process
Andrew A. Renshaw, M.D.
Sheryl A. DiNisco, C.T. (ASCP)
Lori J. Minter, C.T. (ASCP)
Edmund S. Cibas, M.D.
BACKGROUND. The false-negative rate (FNR), or fraction, of Papanicolaou (Pap)
Division of Cytology, Department of Pathology,
Brigham & Women’s Hospital, Boston, Massachusetts.
smear screening has been proposed as a useful quality assessment measure. The
FNR should account for the FNR of the rescreening process itself. The authors
measured the FNR of the rescreening process by rescreening a set of abnormal
METHODS. A randomly selected group of negative (150) and abnormal (91) smears
were rescreened in a blinded fashion. A diagnosis of atypical squamous cells of
undetermined significance (ASCUS) or worse was used as a positive (abnormal)
result. All discrepancies were confirmed by consensus review. The true FNR of
screening Pap smears was calculated as:
true FNR Å calculated FNR/(1-FNR of rescreening)
RESULTS. When rescreened, 17 originally negative cases were interpreted as ASCUS
and 5 as unsatisfactory. Twenty-three originally abnormal cases (22 ASCUS and
1 low grade squamous intraepithelial lesion) were interpreted as negative. After
consensus review, only 1 of the originally negative cases was believed to be ASCUS
and 1 unsatisfactory; 18 of the 23 originally abnormal cases were believed to be
rescreening errors and 5 of the 23 originally abnormal cases were believed to be
false-positives. The FNR of Pap smear screening as traditionally calculated was
6.1%, which was slightly less than the laboratory’s usual FNR. The FNR of review
screening was 20.9%. The true FNR of Pap smear screening was 7.8% and the falsepositive rate was 0.6%.
CONCLUSIONS. The FNR of rescreening is not insubstantial. It can and should be
measured by rescreening abnormal smears, and when taken into account yields
a more accurate measure of the FNR of Pap smear screening. Cancer (Cancer
Cytopathol) 1997;81:272–6. r 1997 American Cancer Society.
KEYWORDS: cytology, cytopathology, Papanicolaou smear, gynecology, false-negative rate, accuracy, statistics.
Address for reprints: Andrew A. Renshaw, M.D.,
Department of Pathology, Brigham & Women’s
Hospital, 75 Francis St., Boston, MA 02115.
Received July 14, 1997; accepted July 26, 1997.
etermining the false-negative rate (FNR) of Papanicolaou (Pap)
smear screening is an important part of quality assessment in any
cytology laboratory.1 Reducing the FNR of Pap smear screening
should be part of the quality improvement program of every cytology
laboratory. Needless to say, the accuracy of the FNR measurement,
and hence the value of any efforts to reduce it, are directly dependent
on the quality of the tests that are performed to measure FNR. If the
FNR is not measured accurately, it is difficult to reliably measure any
quality improvements. Measuring the FNR of screening is no different
q 1997 American Cancer Society
/ 730a$$380y
10-09-97 09:36:57
W: Can Cyto
FNR of Rescreening/Renshaw et al.
than any test performed in science, and one way to
measure the accuracy of a test is to determine the
results of the test using control cases.
Although there is only one definition of the FNR,
namely the proportion of women with disease who are
misdiagnosed as negative, there are many FNRs for Pap
smears, depending on the type of FNR measured (sampling or screening), the definition of and method used to
identify errors, and the population of women studied.2,3
For example, one can define the population of women
with disease as all women with biopsy proven lesions;
identifying women with prior negative smears reveals
both sampling and screening errors. Alternatively, one
can use a second screening of Pap smears to identify
missed cases and thus obtain an estimate of the number
of women with lesions.4 This method measures only
screening and interpretive errors, not sampling errors.
The results of this method depend on the accuracy of the
rescreening process. A more thorough rescreening will
identify more errors and a higher FNR than a less thorough rescreening. To ensure that the results of this
method are reliable, it is appropriate to include controls
to measure how thorough a rescreening is being performed. By including abnormal cases in the rescreening
process, the thoroughness of rescreening (in effect, the
FNR of rescreening) can be measured directly. Including
abnormal cases in the rescreening sample is similar to
another method of estimating the FNR, namely ‘‘seeding’’
the daily workload with known abnormal cases.5–7
Including abnormal cases in the rescreening pool
not only serves as an excellent control but also can be
used to determine a more accurate measure of the
true FNR. To illustrate this, let us define an error as a
case that was screened and diagnosed as normal but
when reexamined has abnormal cells present (a
screening error). These errors also may be termed
false-negative cases. True-positive cases would include those smears that were termed abnormal in
which abnormal cells are present on the smear. To
calculate the FNR of screening without including the
FNR of rescreening (the so-called ‘‘calculated FNR’’)
one would use an equation such as
calculated FNR Å false-negative cases/
(false-negative cases / true-positive cases)
However, this calculated FNR depends on the error
rate in rescreening, or in other words, the calculated
FNR is directly dependent on the percentage of errors
that are detected. How close this calculated FNR is to
the actual or true FNR can be expressed as
calculated FNR/true FNR
Å percentage of errors detected in rescreening
/ 730a$$380y
10-09-97 09:36:57
If 100% of errors are detected, the calculated FNR
equals the true FNR. If no errors are detected, than
the true FNR cannot be determined. Because the percentage of errors detected in rescreening is equal to
the true-positive rate of rescreening, and the sum of
the true-positive rate and the FNR must equal 1, then
substituting in the equation mentioned earlier it follows that:
Equation 1. True FNR
Å calculated FNR/(1-FNR of rescreening)
Although the mathematics of this analysis are relatively straightforward,2 to date the FNR of rescreening
has not been measured. Determining the FNR of rescreening as outlined earlier does require some effort.
Whether the FNR of rescreening is worth measuring
depends on its magnitude. In particular, if the FNR of
rescreening is very low and approaches zero, then
there is less justification for the added inconvenience.
To assess whether the FNR of rescreening can be measured as outlined earlier and to determine its magnitude, the authors reviewed a random sample of normal
and abnormal smears.
For the purposes of this study, an abnormal Pap smear
is one with a diagnosis of atypical squamous cells of
undetermined significance (ASCUS) or worse, unless
otherwise stated. When a stricter definition of an abnormal smear is used, so that only cases with a diagnosis of low grade squamous intraepithelial lesion (LSIL)
or above are considered to have an abnormal Pap
smear, this is specifically stated. In this situation all
ASCUS cases are considered negative. Based on the
calculated FNR of the laboratory, the authors estimated that at least 100 normal cases would have to
be rescreened to detect at least 1 false-negative case.
Using this as a guideline, a sample of 150 negative and
91 abnormal (75 ASCUS, 14 LSIL, and 2 high grade SIL)
Pap smears from the Brigham & Women’s Hospital in
Boston had all markings removed from the slides and
were rescreened by a cytotechnologist in a blinded
fashion without knowledge of the patient’s clinical history. Within each group (negative and abnormal) all
cases were randomly selected from cases reviewed in
the study laboratory over a 3-month period (January
1 – March 31, 1997). No specific time limits were placed
on the rescreening process, although in general the
same amount of time was spent on each case as would
have been spent during a routine screening. Although
the cytotechnologist who performed all of the rescreening did know that there were more abnormal
cases than usual included in the study, the exact pro-
W: Can Cyto
CANCER (CANCER CYTOPATHOLOGY) October 25, 1997 / Volume 81 / Number 5
Calculation of Error Rates
Distribution of cases in laboratory
A. Total negative cases
B. Total abnormal cases (ASCUS and above)
C. Percentage of abnormal cases (B/(A / B)
False-negative rate of rescreening
D. No. of abnormal cases rescreened
E. No. of screening false-positives detected
F. No. of true abnormal cases rescreened (D 0 E)
G. No. of rescreening false-negative cases
H. FNR of rescreening (G/F)
Total screening false-positives
I. No. of screening false-positive cases detected
J. Total no. of false-positive cases in laboratory (I 1 B/D)
Total screening false-negatives
K. No. of negative cases rescreened
L. No. of negative cases believed to be unsatisfactory
M. Total no. of negative cases rescreened (K 0 L)
N. No. of screening false-negative cases detected
O. Total no. of false-negative cases in laboratory (N 1 A/M)
Screening error rates
P. Screening false-positive rate (J/(J / A 0 O)
Q. Calculated screening false-negative rate (O/(O / B 0 J)
R. True screening false-negative rate (Q/(1 0 H)
ASCUS: atypical squamous cells of undetermined significance; FNR: false-negative rate.
portion of cases was unknown. All discrepant cases
were reviewed in a consensus conference, and only
cases in which there was unanimous agreement that
the original diagnosis was incorrect were considered
errors. Clinical history was available for the consensus
review. The authors include two cytopathologists with
different ASCUS/SIL ratios (2.6 and 4.5). For the last
several years and specifically during the time period
from which the smears were taken, the incidence of
abnormal smears for the laboratory was constant at
9 – 10%. The total number of cases reviewed during the
first 3 months of 1997 (Table 1) was used to calculate
the error rates.
The mathematics for the analysis have been described in detail elsewhere.2,4 For this report, the total
number of false-negative cases for the laboratory was
determined by multiplying the number of false-negatives detected by the ratio of the total number of negative cases in the laboratory over the number of negative smears reviewed. Thus, the FNR of screening determined in this study is based on the negative cases
reviewed for this study. This number then is divided
by the total number of abnormal smears in the laboratory to determine the FNR, also referred to as the falsenegative fraction. All the cases that originally were diagnosed as abnormal yet were believed to be negative
on rescreening were assigned to one of two categories
based on the consensus review. Those cases that on
/ 730a$$380y
10-09-97 09:36:57
consensus review were believed to be abnormal were
considered rescreening errors, and those that were believed to be truly negative were considered false-positives. The FNR of rescreening was determined by dividing the number of false-negative cases by the total
number of abnormal cases rescreened, which was
equal to the original number of abnormal cases minus
the false-positive cases. The ‘‘true’’ FNR of screening
Pap smears was determined using Equation 1.
When rescreened, 17 originally negative cases were
interpreted as ASCUS. After consensus review, only
one of the originally negative cases was believed to be
a screening error; the remaining cases were believed
by at least one of the authors to be reasonably diagnosed as negative. The one false-negative smear contained five to ten cells with mildly enlarged, hyperchromatic, and irregular nuclei that were believed to be
best diagnosed as ASCUS.
Five cases originally diagnosed as negative were
believed to be unsatisfactory on rescreening. After review, only one was believed to be unsatisfactory.
Twenty-three originally abnormal cases (22 ASCUS and 1 LSIL) were interpreted as negative when
rescreened. After consensus review, 18 of the 23 originally abnormal cases were interpreted as rescreening
errors and 5 of the 23 originally abnormal cases were
believed to be false-positives. The 18 rescreening cases
were not uniform; one had many cells diagnostic of
LSIL, the remaining 17 cases had either rare or many
cells diagnostic of ASCUS. The 5 false-positive cases
all had a small number of cells (õ20 and usually õ10)
with enlarged slightly hyperchromatic nuclei that were
believed by the reviewers to be best classified as benign (reactive) cellular changes.
The calculation of error rates are detailed in Table
1. The FNR of Pap smear screening as calculated traditionally was 6.1%. The FNR of review was 20.9%. The
true FNR of Pap smear screening was 7.8% and the
false-positive rate was 0.6%.
When a stricter definition of abnormal (only cases
with a diagnosis of LSIL or above) was used, the FNR
of review was 6.25%, the FNR of screening was 0%,
and the false-positive rate was 0%.
The authors have shown that the FNR of rescreening
can be measured, and that it is substantial enough
that it should be measured. The traditionally calculated FNR of screening underestimated the true FNR
by ú20%.
The method employed technically was easy, but
required some additional work, and may not have
W: Can Cyto
FNR of Rescreening/Renshaw et al.
been completely free of bias. Abnormal cases were
collected at random by one of the authors, but the
resulting sample of abnormal cases had a higher ASCUS/SIL ratio (4.7) than the laboratory’s usual ASCUS/
SIL ratio (approximately 3.0). It is possible that if more
SIL and fewer ASCUS cases had been used in the sample, the FNR of rescreening may have been smaller.
Nevertheless, although only 14 LSIL cases were included in this study, one was still missed on rescreening, and this case had many cells that were diagnostic
of LSIL. Thus, including only SIL cases would not reduce the FNR of rescreening to zero. Conversely, the
incidence of abnormal cases in this population was
significantly higher than in the authors’ usual laboratory case load. It has been shown previously that the
vigilance of an observer is increased with an increased
incidence of abnormal findings8; this increased vigilance may have resulted in a lower FNR of rescreening
than would have been detected if the incidence of
abnormal cases was lower.
When cases that were diagnosed originally as abnormal were believed to be negative on rescreening,
another thorough search of the slide was necessary to
determine whether significant cells were missed. In
most cases additional groups of cells were found, suggesting that these were true screening errors. However,
because the original dots were removed, it cannot be
determined with certainty whether these represent
screening or interpretation errors. In the future, photocopying or otherwise recording the location of the
dots before removing them may help to distinguish
these types of errors. Finally, the number of negative
smears reviewed was small, and the resulting calculated FNR may not be very precise. In fact, the traditionally calculated FNR (6.1%) is lower than the usual
FNR of the laboratory (approximately 15%), although
both numbers fall well within the range of FNRs that
have been published previously.4,9 – 11) A more precise
measure of the calculated FNR could be obtained by
rescreening more negative smears, but this was not
the major focus of this study. Nevertheless, hopefully
by including the FNR of rescreening in the determination of the FNR of screening, some of the variation in
reported FNRs,9 ranging from 5%4 up to 68%,12 may
be reduced.
Performing this type of a study requires some
courage on the part of the cytotechnologist who performs the rescreening. Unlike routine rescreening in
which there is no measure of how well the task is being
performed, the quality of the rescreening process is
being defined. In addition, because of the way discrepancies are resolved, most of the cytotechnologist’s interpretations will be believed to be in error. The
method the authors described can be performed only
/ 730a$$380y
10-09-97 09:36:57
with the attitude that mistakes are to be learned from
rather than hidden.13 To the authors’ knowledge, no
previous study has measured the FNR of rescreening
in the way shown in the current study, making comparisons difficult. However, in one previous study that
used different terminology and in which 2 laboratories
exchanged slides, the FNRs for dysplasia (40 – 68%)
were substantially larger than the rate determined in
the current study.12
An added benefit of this process is the determination of the false-positive rate. Much of the controversy
surrounding ASCUS is focused on its less than excellent reproducibility.14 – 16 The term ‘‘false-positive’’ in
the context of this article is a direct measure of the
disagreement between the cytopathologists in this laboratory on the threshold between a diagnosis of ASCUS and a negative diagnosis. It is reassuring that the
false-positive rate appears low; as has been pointed
out before,2 a high false-positive rate would suggest
that there is significant intralaboratory variation in the
use of an ASCUS diagnosis. Because the diagnosis of
ASCUS is not very reproducible, it also is useful to
determine the false-positive rate, and the corresponding FNR, at thresholds other than ASCUS. When this
is done using a stricter definition of an abnormal
smear (in this case LSIL rather than ASCUS), the FNR
of review decreased to 6.25%, and the screening FNR
and screening false-positive rate decreased to zero. It
is not surprising that with this stricter, more well defined, and more reproducible threshold the error rates
decreased. However, it is probable, based on the results of previous studies,9,17 that with review of many
more additional cases both the FNR and the falsepositive rate of screening would be greater than zero.
Nevertheless, what is an acceptable false-positive
rate, regardless of the threshold, is unclear; no comparable data exist. Because the false-positive rate is based
on the total number of negative cases, a very large
denominator, the calculated false-positive rate always
will appear low on this basis. In addition, the authors
required unanimous agreement that the original abnormal diagnosis was incorrect to include the case as
a false-positive. In fact, there was significant disagreement on many of these cases (and the false-negative
cases as well), and if only one observer’s opinion had
been relied on the false-positive rate could have been
greater. The beauty of the method described in the
current study is that it is one of checks and balances;
if another observer has different criteria and decides
that the false-positive rate is higher than the authors
have calculated, this will immediately be reflected in
a lower FNR of rescreening. In other words, the observer’s opinion has been placed squarely between two
competing quality assessment measures; if the ob-
W: Can Cyto
CANCER (CANCER CYTOPATHOLOGY) October 25, 1997 / Volume 81 / Number 5
server shifts his or her criteria in one direction, that
shift immediately is apparent in both measurements.
In this way differences in criteria can be measured and
distinguished from differences in the FNR of screening
or rescreening.
Finally, it is important to consider the FNR of rescreening in determining the effectiveness of several
automated screening technologies. Those technologies that select cases for rescreening by cytotechnologists will have to include the FNR of rescreening by
the cytotechnologists in determining their overall FNR.
The authors have shown that by including abnormal smears in the rescreening process it is possible to
measure the FNR of the rescreening mechanism itself.
Because the FNR of rescreening is substantial, it
should be measured. FNRs that are determined without an estimate of the FNR of rescreening are unreliable.
Krieger PA. Strategies for reducing Papanicolaou smear
screening errors. Principles derived from data and experience with quality control. Arch Pathol Lab Med 1997;
Renshaw AA. Analysis of error in calculating the false-negative rate in the interpretation of cervicovaginal smears: the
need to review abnormal cases. Cancer (Cancer Cytopathol)
Davey DD. Quality and liability issues with the Papanicolaou
smear. The problem of definition of errors and false-negative
smears. Arch Pathol Lab Med 1997;121:267–9.
Krieger P, Naryshkin S. Random rescreening of cytologic
smears: a practical and effective component of quality assurance programs in both large and small cytology laboratories.
Acta Cytol 1994;38:291–8.
Hindman WM. An effective quality control program for the
cytology laboratory. Acta Cytol 1976;20:233–8.
/ 730a$$380y
10-09-97 09:36:57
Hindman WM. A proposal for quality control in gynecologic
cytology. Acta Cytol 1987;31:384–5.
Hindman WM. An approach to the problem of false negatives
in gynecologic cytologic screening. Acta Cytol 1989;33:814–8.
Fowkes FGR. Diagnostic vigilance. Lancet 1986;xx:493–4.
Frable WJ. Does a zero standard of error exist for the Papanicolaou smear? Arch Pathol Lab Med 1997;121:301–10.
Department of Health and Human Services. Improving the
quality of clinician Pap smear technique and management,
client Pap smear education, and the evaluation of Pap smear
laboratory testing: a resource guide for Title X family planning projects. Washington, DC: U.S. Department of Health
and Human Services, 1989.
Naryshkin S. The false-negative fraction for Papanicolaou
smears. How often are ‘‘abnormal’’ smears not detected by
a ‘‘standard’’ screening cytologist? Arch Pathol Lab Med
Yobs AR, Plott AE, Hicklin MD, Coleman SA, Johnston WW,
Ashton PR, et al. Retrospective evaluation of gynecologic
cytodiagnosis II. Interlaboratory reproducibility as shown in
rescreening large consecutive samples of reported cases.
Acta Cytol 1987;31:900–10.
Derman H. Quality and liability issues with the Papanicolaou smear. Lessons from the science of error prevention.
Arch Pathol Lab Med 1997;121:287–91.
Davey DD, Naryshkin S, Nielsen ML, Kline TS. Atypical squamous cells of undertermined significance: interlaboratory
comparison and quality assurance monitors. Diagn Cytopathol 1994;11:390–6.
Sidaway MK, Tabbara SO. Reactive change and atypical
squamous cells of undertermined significance in Papanicolaou smears: a cytohistologic correlation. Diagn Cytopathol
Cocchi V, Carretti D, Fanti S, Baldazzi P, Casotti MT, Piazza
R, et al. Intralaboratory quality assurance in cervical/vaginal
cytology: evaluation of intercytologist diagnostic reproducibility. Diagn Cytopathol 1997;16:87–92.
Joste NE, Crum CP, Cibas ES. Cytologic/histologic correlation for quality control in cervicovaginal cytology. Am J Clin
Pathol 1995;103:32–4.
W: Can Cyto
Без категории
Размер файла
80 Кб
Пожаловаться на содержимое документа