Papnet-assisted cytological diagnosis intensifies the already marked variability among cytological laboratories

Objective: The main objective was to assess the sensitivity, specificity and reliability of PAPNET-assisted diagnosis in comparison with conventional screening.

Setting: Seven Italian and one English University or Research Institutes, and a random sample of an other 20 Italian Laboratories of the Italian National Health Service (INHS) provided the cervical smears.

Methods: During the training phase every center examined in rotation four sets of slides for a total of 300 representative slides. Afterwards, 900 "positive" slides were added to the 3,100 slides which were collected consecutively without any selection or exclusion. The eight main centers were divided into four couples and each couple of centers examined 775 slides with the PAPNET system, "blindly" to the original diagnosis. An expert cytopathologist (M.A.) of the National Institute of Health (NIH) reassessed 40% of the slides with an original negative diagnosis to evaluate the false negative rate. Two expert NIH cytopathologists (M.A., G.M.) re-examined all slides where a disagreement had been observed between the original and one or both of the study diagnoses. The main analyses concerned the following three main categories: WNL and unsatisfactory for evaluation; ASCUS, AGUS and LSIL; HSIL and carcinoma. A special algorithm was devised to define the reference diagnosis for sensitivity and specificity assessment.

Results: Laboratories, even belonging to the same couple, classified as "no review" a very different proportion of slides ranging from 35% to 74%. The index of kappa agreement between the members of couples examining the same sets of slides was low or very low, ranging from 0.30 to 0.03. The sensitivity of the review classification was particularly low in some laboratories. Surprisingly, only a small correlation was observed between the sensitivity of the review classification and the proportion of slides classified as "review". The "tentative" diagnosis on PAPNET tiles of the "review" slides was almost as reliable as the microscopic diagnosis. In the overall performance, there were many significant differences among the eight laboratories. The best laboratory had a sensitivity of 95% and a specificity of 96%. At least three laboratories displayed unacceptably low sensitivity and one a very low specificity.

Conclusion: Altogether these results seem to confirm that there are wide differences among cytological laboratories per se, and that these differences are intensified by the use of an instrument like PAPNET. The huge variation in performance may be explained by differences in basic skills and by different training, but it is difficult to understand exactly what could have been done to reduce it.

Pap-net; Screening; Cervical Cytology; Inter-observer variation; Diagnosis validation

[1] IARC Working Group on Cervical Cancer Screening and UICC Project Group on the Evaluation of Screening Programmes for Cancer of the uterine cervix: (eds.) Hakama M.. Miller A. B., Dayne, IARC Scientific Publications n. 76, 1986.

[2] Franco E., Monsonego J. (eds).: "New Developments in Cervical Cancer Screening and Prevention". Oxford, Blackwell Science., 1997.

[3] Hakama M. In: "New Developments in Cervical Cancer Screening and Prevention". Blackwell Science, Oxford, 1997, 190.

[4] Koss L. G.:'The Papanicolaou test for cervical cancer detectlon A triumph and a tragedy". JAMA, 1989, 269, 737.

[5] Slagel D. D., Zaleski S., Cohen M. B.: "Efficacy of automated cervicai cytology screening". Diag. Cytopathol., 1994, 13, 26.

[6] van Der Graaf Y., Voijs G. P.: "False negative rate in cervical cytology". J. Clin. Pathol., 1989, 40, 438.

[7] Mody D. R., Davey D. D., Branca M.: "Quality assurance and risk reduction guidelines". Acta Cytol., 2000, 44, 496.

[8] Wied G. L., Bartels P. H., Bibbo M. et al. Editorial: "Computer assisted quality assurance". Acta Cytol., 1996, 40, 1.

[9] Richart R. M.: "Evaluation of true false negative rate in cytology" Am. J. Obst. Gynaec., 1964, 89, 723.

[10] Rilke F., Mango L., Alasio L.: "Sensitivity of the PAPNET cytological screening system to a routine mixture of conventional smears". Mod. Pathol.,1993, 6, 31.

[11] Rombach J. J., Cranendonk R., Velthius F. J.: "Monitoring laboratory performance by statistical analysis of concordance". Acta Cytol., 1987, 31, 887.

[12] Ronco G., Montanari G., Aimone Y., Parisio F. et al.: "Estimatmg the sensitivity of cervical cytology: errors of interpretation and test limitations". Cytopathology, 1996, 7, 151.

[13] Jordan S. W.: "Great expectations: cytology provision of CLIA-88 and the role of professional societies". Cytopatho/ogy Annual, 1992, 235.

[14] Kaminsky F. C., Burke R. J., Haberle K. R., Mullins M. S.: "Rescreening policies in cervical cytology and their effect on detecting the truly positive patients". Acta Cytol., 1995, 39, 239.

[15] Ashfaq R., Salinger F., Solares B. et al.: "Evaluation of the PAPNET System for prescreening triage of cervico-vaginal smears". Acta Cytol., 1997, 41 (4), 1058.

[16] Attwood M. E., Woodman C. B. J., Luesley D., Jordan J. A.: ··rrevious cytology in patients with invasive carcinoma of the cervix". Acta Cytol.. 1985, 29, 108.

[17] Farsworth A., Chambers F. A., Goldschmidt C. S.: "Evaluation of the PAPNET System in a general pathology service". M. J. A., 1996, 165, 429.

[18] Kharazi M., Keyhani-Rofagha S., O'Toole R.: "Automated screening of cervical cytology using PAPNET: comparative cytological findings with biopsy". Am. J. Clin. Pathol., 1993, 99, 337.

[19] Kish J. K., Sheik D.: "Evaluation of the performance of the PAPNET automated cytological screening systern on the conventionally prepared cervical Papanicolaou smears at Hinsdale Hospital". Hisdale, Illinois., Anal. Quant. Cytol. Histol., 1992, 14, 267.

[20] Klinkhamer P. J. J. M., Voijs G. P., de Haan A. F. J.: "lntraobserver and interob、erver variability in the quality assessment of the cervical smears". Acta Cytol., 1989, 33, 215.

[21] Kok M. R.. Boon M. S., Boon M. E.: "Effects of applying neural networks in cervical screening: lower overtreatment rates and less over-diagnosis for patients with mild/moderate dysplastic smears" J. Cell. Pathol., 1996, 1,109.

[22] Koss L. G., Eunice Lin M. D., Schreiber K., Elgert P., Mango L.: "Evaluation of the PAPNET cytologic screening system for quality control of cervical smears". Am. J. Cl. Path., 1994, 101, 220.

[23] Landis J. R., Koch G. G.: "The measurement of observer agreement for categorical data". Biometrics, 1977, 33, 159.

[24] Fleiss J. L.. Cohen J.: "Statistical Methods for Rates and Proportions". New York: John Wiley and sons,1981.

[25] Mitchell H., Medley G.:''Detection of laboratory false negative smears by the PAPNET cytologic screening system". Acta Cytol., 1998, 42, 265.

[26] PRISMATIC Project Management Team. Assessment of automated primary screening on PAPNET of cervical smears in the PR!SMATIC trial., Lancet., 1999, 353, 1381.