For a long time, doctors have been diagnosing disorders of the body’s hematopoietic system using a light microscope. The analysis of individual blood cells is largely performed manually. Now, artificial intelligence can lend them a digital hand.
Diagnosing malignant diseases of the hematopoietic system requires a range of different laboratory techniques. In addition to modern immunological and molecular genetic methods, the process of morphologically diagnosing white blood cells using a light microscope, as introduced by the pathologist Rudolf Virchow over 170 years ago , continues to play an essential role in this effort.
A cornerstone of hematological diagnostics: Cytomorphology
In contrast to many other molecular methods which have been computerized for a long time now, morphological diagnostics still rely on assessments performed by human cytologists. In this process, leukocytes from peripheral blood or bone marrow are applied to a slide, stained and subsequently manually screened and classified at high optical magnification. Current standards stipulate that at least 200 cells should be evaluated per patient , which can be very time-consuming and limit the number of available examinations. Furthermore, the results are subject to the personal interpretation and professional experience of the respective examiner. This diminishes the reproducibility of morphological diagnostics .
Assistance from novel software: Using deep learning for image analysis
In this type of situation, it makes sense to resort to advances in automated image analysis. Over the last few years, scientists have succeeded in using artificial neural networks and methods known as “deep learning” to develop automated image analysis systems that can classify image content with the same accuracy as a human examiner . These methods are also successfully used on data generated by medical imaging, e.g. in radiology, pathology and dermatology.
One of the decisive advantages of neural networks is that the software automatically identifies relevant structural elements in images, based on the training data it receives as input. Steps such as explicitly measuring the size of the nucleus or staining the cytoplasm, for example, are not required.
Data quality and quantity must fulfill high standards
Successfully employing neural networks in image analysis hinges on the availability of large amounts of image data, which, like in routine diagnostics, must be annotated by cytologists. For this purpose, our working group created a data set containing 18,000 images of individual leukocytes taken from 100 patients diagnosed with acute myeloid leukemia (AML) and from 100 control patients. The specimens were digitized at 100x magnification with the help of a slide scanner. Like in routine diagnostics, the image data were then annotated by two medical technicians with extensive experience in cytological analysis on the basis of a standard 15-category scheme used in hematology. This allowed us to compare the consistency of the annotations from two independent sources .
One major prerequisite for applying neural networks is the availability of high-quality image data and annotations. For medical image data, however, meeting the necessary requirements in terms of the quality and quantity of image data can be particularly difficult. In order to provide a starting point for further advances in the field, the authors therefore published the image data set on a platform called “The Cancer Imaging Archive,” which is run by the U.S. National Cancer Institute .
Cell identification with human accuracy
This image data set was then used to train a number of different neural networks to classify individual white blood cells in the standard diagnostic scheme comprising 15 morphological categories. Furthermore, the networks can be used to find answers to two questions that are of central importance to diagnostics and therapy: i) whether a particular cell is typical of a blast cell, i.e. whether it possesses an AML-like shape, and ii) whether the cell would be present in the specimen in normal cases. The neural network answered both of these questions with the accuracy of a human annotator. Also, with regard to the direct classification of individual cells, the neural network outperformed other approaches. The success of deep learning can therefore be applied to the morphological examination of leukocytes.
Peering into the “black box”
Fig. 1 Sample image from the single-cell data set next to a saliency map, which illustrates how relevant individual pixels are for the network’s decision-making process when classifying these cells. The lighter a pixel, the more relevant it is for cell classification.
One aspect of neural networks that could potentially be problematic when it comes to applying these networks in the medical field is that, on the one hand, they allow for a very precise classification of image data, but on the other, they offer no explanation as to why each cell is put into a particular category. This property is sometimes called the “black box” of neural networks. For scientists, however, it would be desirable if the networks went beyond merely classifying the image content and also revealed how the classification decision was made. Over the last years, this question has developed into an active field of research in its own right . With regard to the classification of leukocytes, we calculated what are known as saliency maps  that allowed us to illustrate which pixels of an image the network considered particularly relevant when making a classification decision (Fig. 1). As a result, it emerged that the network had learned to analyze the same areas of images that would also be of importance for human cytologists, such as the structure of the cell nucleus and the cytoplasm.
Conclusion: Opportunities and challenges
When classifying leukocytes to diagnose types of leukemia, neural networks performed at a very high level in answering clinically relevant questions, for example related to the presence of blast cells. They also classified cells into standard morphological categories with great precision. Examinations of such networks indicate that they classify cells based on images structures that are also relevant for human examiners.
Despite these successes, however, there remains a potential for errors to be made. For example, this method imposes a high level of requirements on the whole process of pre-analytics, staining and digitization, e.g. to rule out systematic distortions of the training data. Efforts to standardize many of these aspects have been made over the past years .
Even very successful networks can potentially make an erroneous classification, which is why the results provided by the network must always be viewed in the context of other lab results and clinical parameters. If these aspects are taken into account, however, neural networks have the potential to provide valuable assistance in making diagnostic decisions.
The work presented is a joint project of the Helmholtz Zentrum München, German Research Center for Environmental Health, Munich, Germany, and the University Hospital of Ludwig-Maximilians-Universität (LMU) München, Munich, led by Dr. Carsten Marr and Dr. Christian Matek from the Institute of Computational Biology at Helmholtz Zentrum München as well as Prof. Dr. med Karsten Spiekermann from the Department of Medicine III, University Hospital, LMU Munich, Campus Großhadern.
Category: Bioinformatics | Diagnosis
 Virchow R. (1845) Weisses Blut, Froriep’s Notizen, 36, 151-156
 Wei, A. H., Löwenberg, B., Bloomfield, C. D. (2017) Diagnosis and management of AML in adults: 2017 ELN recommendations from an international expert panel, Blood, 129, 424-447, DOI: 10.1182/blood-2016-08-733196
 Font, P. et al. (2013) Inter-observer variance with the diagnosis of myelodysplastic syndromes (MDS) following the 2008 WHO classification, Ann. Hematol., 92, 19-24, DOI: 10.1007/s00277-012-1565-4
 Goodfellow, I., Bengio, Y., Courville, A. (2016) Deep Learning, Cambridge, MA, MIT Press
 Matek C., Schwarz, S., Spiekermann, K., Marr C. (2019) Human-level recognition of blast cells in acute myeloid leukemia with convolutional neural networks, Nat. Mach. Intell. 1, 538-544, DOI: 10.1038/s42256-019-0101-9
 Matek, C., Schwarz, S., Spiekermann, K., Marr, C. A single-cell morphological dataset of leukocytes from AML patients and non-malignant controls (AML-Cytomorphology_LMU). The Cancer Imaging Archive, on the internet at: https://doi.org/10.7937/tcia.2019.36f5o9ld (as of Sep 20, 2020)
 Samek, W., Montavon, G., Vedaldi, A., Hansen, L. K., Müller, K. -R. (Eds.) (2019) Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, Lecture Notes in Artificial Intelligence, 11700, Heidelberg, Springer
 Simonyan, K., Vedaldi, A., Zisserman, A. (2014) In “Deep inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps,” pp. 1–8, International Conference on Learning Representations (ICLR)
 Haroske, G., Zwönitzer, R., Hufnagl, P. et al. (2018) Leitfaden „Digitale Pathologie in der Diagnostik“, Pathologe, 39, 216-221, DOI: 10.1007/s00292-018-0433-y
Header image: iStock.com | Vertigo3d; single-cell data set: © C. Matek
Date of publication: