A Survey on Multi Modal Fusion for Histopathology Image Analysis
Main Article Content
Abstract
For centuries, histopathology has gained a remarkable place in the medical field. The disciplines of histopathology have witnessed outstanding advancements in microscopic technologies. The interpretation of traditional histopathological images by pathologists presents several challenges: high inter-observer variability, stain variability and artifacts, and complexity of tissue microenvironments. The appearance of digital scanners has converted conventional histopathology into computational pathology (CPath) by providing Whole Slide Images (WSIs). CPath tackles the complexity and limitations inherent in conventional histopathological images. The digital histopathology imaging offers a new frontier for the integration of artificial intelligence in WSIs analysis. Deep learning-based approaches have been explored for the automated analysis of digital histopathological images. Notably, these approaches have been performed for tumor classification, and prognosis prediction. However, relying solely on histopathological images may not provide a comprehensive understanding of the disease mechanisms. Several studies have highlighted the potential benefits of combining histopathological images with multimodalities data distinguishing between intra-modalities and inter-modalities to enhance diagnostic accuracy and disease prediction. Multi-modal fusion integrates histopathology images with genomic, clinical, or radiological data to improve diagnostic and prognostic models. Specifically, neural networks represent and combine histopathological images and transcriptomic data into deep survival layer for prognosis prediction. This paper describes the history of histopathology and its complexity which is a challenge in interpreting clinical cases. This review offers a thorough overview of deep learning-driven multi-modal approaches in histopathology, highlighting how they might enhance prognostic modeling and diagnostic precision. Future research should concentrate on optimizing data fusion methods and guaranteeing the clinical applicability of AI models.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
IJCERT Policy:
The published work presented in this paper is licensed under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. This means that the content of this paper can be shared, copied, and redistributed in any medium or format, as long as the original author is properly attributed. Additionally, any derivative works based on this paper must also be licensed under the same terms. This licensing agreement allows for broad dissemination and use of the work while maintaining the author's rights and recognition.
By submitting this paper to IJCERT, the author(s) agree to these licensing terms and confirm that the work is original and does not infringe on any third-party copyright or intellectual property rights.
References
C. Alvarez-Jimenez et al., “Identifying cross-scale associations between radiomic and pathomic signatures of non-small cell lung cancer subtypes: Preliminary results,” Cancers, vol. 12, no. 12, p. 3663, 2020.
V. Ariotta et al., “H&E image analysis pipeline for quantifying morphological features,” J. Pathol. Inform., vol. 14, p. 100339, 2023.
Ashtaiwi, “Optimal histopathological magnification factors for deep learning-based breast cancer prediction,” Appl. Syst. Innov., vol. 5, no. 5, p. 87, 2022.
K. M. Boehm et al., “Multimodal data integration using machine learning improves risk stratification of high-grade serous ovarian cancer,” Nat. Cancer, vol. 3, no. 6, pp. 723–733, 2022.
N. Braman et al., “Deep orthogonal fusion: Multimodal prognostic biomarker discovery integrating radiology, pathology, genomic, and clinical data,” in Med. Image Comput. Comput. Assist. Interv. (MICCAI), 2021, pp. 667–677.
R. Brixtel et al., “Whole slide image quality in digital pathology: Review and perspectives,” IEEE Access, vol. 10, pp. 131005–131035, 2022.
Carmichael et al., “Joint and individual analysis of breast cancer histologic images and genomic covariates,” Ann. Appl. Stat., vol. 15, no. 4, p. 1697, 2021.
D. Caruso et al., “Radiomics in oncology, part 2: Thoracic, genito-urinary, breast, neurological, hematologic and musculoskeletal applications,” Cancers, vol. 13, no. 11, p. 2681, 2021.
L. Chan et al., “Histosegnet: Semantic segmentation of histological tissue type in whole slide images,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 2019, pp. 10662–10671.
L. Chen et al., “Histopathological images and multi-omics integration predict molecular characteristics and survival in lung adenocarcinoma,” Front. Cell Dev. Biol., vol. 9, p. 720110, 2021.
R. J. Chen et al., “Pathomic fusion: An integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis,” IEEE Trans. Med. Imaging, vol. 41, no. 4, pp. 757–770, 2022.
C. Cui et al., “Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: A review,” Prog. Biomed. Eng., vol. 5, no. 2, p. 022001, 2023.
M. Dabass, R. Vig, and S. Vashisth, “Review of histopathological image segmentation via current deep learning approaches,” in Proc. Int. Conf. Comput. Commun. Autom. (ICCCA), 2018, pp. 1–6.
K. Dia et al., “The cross-scale association between pathomics and radiomics features in immunotherapy-treated NSCLC patients: A preliminary study,” Cancers, vol. 16, no. 2, p. 348, 2024.
S. Diao et al., “Deep multi-magnification similarity learning for histopathological image classification,” IEEE J. Biomed. Health Inform., vol. 27, no. 3, pp. 1535–1545, 2023.
Dimitropoulos et al., “Classification of nuclei in follicular lymphoma tissue sections using different stains and Bayesian networks,” in Med. Biol. Eng. Comput. (MEDICON), 2016, pp. 234–238.
C. Dwivedi et al., “Multi stain graph fusion for multimodal integration in pathology,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022, pp. 1835–1845.
H. Fischer et al., “Hematoxylin and eosin staining of tissue and cell sections,” Cold Spring Harb. Protoc., vol. 2008, no. 5, p. prot4986, 2008.
R. J. Gillies, P. E. Kinahan, and H. Hricak, “Radiomics: Images are more than pictures, they are data,” Radiology, vol. 278, no. 2, pp. 563–577, 2016.
Y. Gu and J. Yang, “Densely-connected multi-magnification hashing for histopathological image retrieval,” IEEE J. Biomed. Health Inform., vol. 23, no. 4, pp. 1683–1691, 2018.
Guo et al., “CroMAM: A cross-magnification attention feature fusion model for predicting genetic status and survival of gliomas using histological images,” IEEE J. Biomed. Health Inform., 2024.
J. Hao et al., “PAGE-Net: Interpretable and integrative deep learning for survival analysis using histopathological images and genomic data,” in Pac. Symp. Biocomput., 2020, pp. 355–366.
Hemker, N. Smidjievski, and M. Jamnik, “HEALNet–Hybrid multi-modal fusion for heterogeneous biomedical data,” arXiv, 2023.
Herzog et al., “Integrating uncertainty in deep neural networks for MRI based stroke analysis,” Med. Image Anal., vol. 65, p. 101790, 2020.
D. J. Ho et al., “Deep multi-magnification networks for multi-class breast cancer image segmentation,” Comput. Med. Imaging Graph., vol. 88, p. 101866, 2021.
W.-C. Huang et al., “Automatic HCC detection using convolutional network with multi-magnification input images,” in Proc. IEEE Int. Conf. AI Circuits Syst. (AICAS), 2019, pp. 194–198.
Hussein et al., “Once upon a microscopic slide: The story of histology,” J. Cytol. Histol., vol. 6, 2015.
L. Jiang et al., “A deep learning-based approach for glomeruli instance segmentation from multistained renal biopsy pathologic images,” Am. J. Pathol., vol. 191, no. 8, pp. 1431–1441, 2021.
S. Kalra et al., “Pan-cancer diagnostic consensus through searching archival histopathology images using artificial intelligence,” NPJ Digit. Med., vol. 3, no. 1, p. 31, 2020.
W. Kang et al., “Application of radiomics-based multiomics combinations in the tumor microenvironment and cancer prognosis,” J. Transl. Med., vol. 21, no. 1, p. 598, 2023.
S. L. Kee et al., “Predicting papillary renal cell carcinoma prognosis using integrative analysis of histopathological images and genomic data,” in Int. Work-Conf. Bioinform. Biomed. Eng., 2023, pp. 208–221.
K. Mirabadi et al., “GRASP: GRAph-structured pyramidal whole slide image representation,” arXiv, 2024.
Z. Khodabakhshi et al., “MRI-based radiomics for predicting histology in malignant salivary gland tumors: Methodology and ‘proof of principle’,” Sci. Rep., vol. 14, no. 1, p. 9945, 2024.
Lahiani et al., “Generalising multistain immunohistochemistry tissue segmentation using end-to-end colour deconvolution deep neural networks,” IET Image Process., vol. 13, no. 7, pp. 1066–1073, 2019.
G. Lee et al., “Supervised multi-view canonical correlation analysis (sMVCCA): Integrating histologic and proteomic features for predicting recurrent prostate cancer,” IEEE Trans. Med. Imaging, vol. 34, no. 1, pp. 284–297, 2014.
H. Li et al., “Integrative analysis of histopathological images and genomic data in colon adenocarcinoma,” Front. Oncol., vol. 11, p. 636451, 2021.
S. Li et al., “A novel pathological images and genomic data fusion framework for breast cancer survival prediction,” in Proc. IEEE Eng. Med. Biol. Soc. (EMBC), 2020, pp. 1384–1387.
Z. Li et al., “Flow chamber staining modality for real-time inspection of dynamic phenotypes in multiple histological stains,” PLoS ONE, vol. 18, no. 5, p. e0284444, 2023.
Liu et al., “MGCT: Mutual-guided cross-modality transformer for survival outcome prediction using integrative histopathology-genomic features,” in Proc. IEEE Int. Conf. Bioinform. Biomed. (BIBM), 2023, pp. 1306–1312.
X. Liu et al., “MSMTSeg: Multi-stained multi-tissue segmentation of kidney histology images via generative self-supervised meta-learning framework,” IEEE J. Biomed. Health Inform., 2024.
C. Lu, R. Shiradkar, and Z. Liu, “Integrating pathomics with radiomics and genomics for cancer prognosis: A brief review,” Chin. J. Cancer Res., vol. 33, no. 5, p. 563, 2021.
Z. Lu et al., “Deep-learning–based characterization of tumor-infiltrating lymphocytes in breast cancers from histopathology images and multiomics data,” JCO Clin. Cancer Inform., vol. 4, pp. 480–490, 2020.
Z. Lv et al., “TransSurv: Transformer-based survival analysis model integrating histopathological images and genomic data for colorectal cancer,” IEEE/ACM Trans. Comput. Biol. Bioinform., vol. 20, no. 6, pp. 3411–3420, 2022.
M. T. McCann et al., “Automated histology analysis: Opportunities for signal processing,” IEEE Signal Process. Mag., vol. 32, no. 1, pp. 78–87, 2014.
Michigan Medicine, “Introduction to histology stains,” 2024. [Online]. Available: https://histology.medicine.umich.edu/resources/introduction-histology-stains
K. Mirabadi et al., “GRASP: GRAph-structured pyramidal whole slide image representation,” arXiv, 2024.
Mobadersany et al., “Predicting cancer outcomes from histology and genomics using convolutional networks,” Proc. Natl. Acad. Sci. USA, vol. 115, no. 13, pp. E2970–E2979, 2018.
D. Montalvo-García et al., “Stromal tissue segmentation in multi-stained serial histopathological sections of pancreatic tumors,” in MICCAI Workshop Comput. Pathol. Multimodal Data (COMPAYL), 2024.
M. Moscalu et al., “Histopathological images analysis and predictive modeling implemented in digital pathology—Current affairs and perspectives,” Diagnostics, vol. 13, no. 14, p. 2379, 2023.
S. Mukhopadhyay et al., “Whole slide imaging versus microscopy for primary diagnosis in surgical pathology: A multicenter blinded randomized noninferiority study of 1992 cases (pivotal study),” Am. J. Surg. Pathol., vol. 42, no. 1, pp. 39–52, 2018.
G. Musumeci, “Past, present and future: Overview on histology and histopathology,” J. Histol. Histopathol., vol. 1, no. 5, pp. 10–7243, 2014.
M. Naouar et al., “Robust tumor detection from coarse annotations via multi-magnification ensembles,” arXiv, 2023.
A. Paray et al., “Gram staining: A brief review,” Int. J. Res. Rev., vol. 10, pp. 336–341, 2023.
L. Pei et al., “Deep neural network analysis of pathology images with integrated molecular data for enhanced glioma classification and grading,” Front. Oncol., vol. 11, p. 668694, 2021.
Y. Peng, J. Bian, and J. Xu, “FedMM: Federated multi-modal learning with modality heterogeneity in computational pathology,” arXiv, 2024.
Petersen et al., “Deep probabilistic modeling of glioma growth,” in Med. Image Comput. Comput. Assist. Interv. (MICCAI), 2019, pp. 806–814.
D. G. Poalelungi et al., “Advancing patient care: How artificial intelligence is transforming healthcare,” J. Pers. Med., vol. 13, no. 8, p. 1214, 2023.
Qu et al., “Advancing H&E-to-IHC stain translation in breast cancer: A multi-magnification and attention-based approach,” arXiv, 2024.
G. Raipuria, A. Srivastava, and N. Singhal, “TransEM-Net: Transformer based efficient multi-magnification network for histopathology,” in Proc. IEEE Int. Symp. Biomed. Imaging (ISBI), 2023, pp. 1–4.
Rasoolijaberi et al., “Multi-magnification image search in digital pathology,” IEEE J. Biomed. Health Inform., vol. 26, no. 9, pp. 4611–4622, 2022.
S. Rathore et al., “Radiopathomics: Integration of radiographic and histologic characteristics for prognostication in glioblastoma,” arXiv, 2019.
J. Ren et al., “Differentiation among prostate cancer patients with Gleason score of 7 using histopathology whole-slide image and genomic data,” in Proc. SPIE Med. Imaging, vol. 10579, p. 1057904, 2018.
J. Ren et al., “Recurrence analysis on prostate cancer patients with Gleason score 7 using integrated histopathology whole-slide images and genomic data through deep neural networks,” J. Med. Imaging, vol. 5, no. 4, p. 047501, 2018.
V. Salian et al., “Special stains in histopathology: An oral pathology perspective,” J. Dent. Health Oral Res., vol. 2, no. 3, pp. 1–19, 2021.
Z. Senousy et al., “MCUa: Multi-level context and uncertainty aware dynamic deep ensemble for breast cancer histology image classification,” IEEE Trans. Biomed. Eng., vol. 69, no. 2, pp. 818–829, 2021.
C. G. Serna et al., “MMO-Net (Multi-Magnification Organ Network): A use case for organ identification using multiple magnifications in preclinical pathology studies,” J. Pathol. Inform., vol. 13, p. 100126, 2022.
Shao et al., “Multiparametric MRI and whole slide image-based pretreatment prediction of pathological response to neoadjuvant chemoradiotherapy in rectal cancer: A multicenter radiopathomic study,” Ann. Surg. Oncol., vol. 27, pp. 4296–4306, 2020.
W. Shao et al., “Characterizing the survival-associated interactions between tumor-infiltrating lymphocytes and tumors from pathological images and multi-omics data,” IEEE Trans. Med. Imaging, 2023.
W. Shao et al., “Integrative analysis of pathological images and multi-dimensional genomic data for early-stage cancer prognosis,” IEEE Trans. Med. Imaging, vol. 39, no. 1, pp. 99–110, 2019.
T. Soyab, “Special stains used in histopathological techniques: A brief view,” Indian J. Forensic Med. Toxicol., vol. 14, no. 4, pp. 8632–8636, 2020.
S. R. Stahlschmidt, B. Ulfenborg, and J. Synnergren, “Multimodal deep learning for biomedical data fusion: A review,” Brief. Bioinform., vol. 23, no. 2, p. bbab569, 2022.
K. Tan et al., “A multi-modal fusion framework based on multi-task correlation learning for cancer prognosis prediction,” Artif. Intell. Med., vol. 126, p. 102260, 2022.
K. Thandiackal et al., “Differentiable zooming for multiple instance learning on whole-slide images,” in Eur. Conf. Comput. Vis. (ECCV), 2022, pp. 699–715.
University of Toronto Department of Laboratory Medicine, “Histopathology learning modules,” 2024. [Online]. Available: https://dlm.lmp.utoronto.ca/taxonomy/term/40
B. Tosun and C. Gunduz-Demir, "Graph run-length matrices for histopathological image segmentation," IEEE Trans. Med. Imaging, vol. 30, no. 3, pp. 721-732, Mar. 2011.
J. Van der Laak, G. Litjens, and F. Ciompi, "Deep learning in histopathology: the path to the clinic," Nat. Med., vol. 27, no. 5, pp. 775-784, May 2021.
R. S. Vanguri et al., "Multimodal integration of radiology, pathology and genomics for prediction of response to PD-(L)1 blockade in patients with non-small cell lung cancer," Nat. Cancer, vol. 3, no. 10, pp. 1151-1164, Oct. 2022.
Veta et al., "Breast cancer histopathology image analysis: a review," IEEE Trans. Biomed. Eng., vol. 61, no. 5, pp. 1400-1411, May 2014.
R. Wang et al., "Fast cancer metastasis location based on dual magnification hard example mining network in whole-slide images," Comput. Biol. Med., vol. 158, p. 106880, Jul. 2023.
X. Wang et al., "Lung cancer subtype diagnosis by fusing image-genomics data and hybrid deep networks," IEEE/ACM Trans. Comput. Biol. Bioinform., vol. 20, no. 1, pp. 512-523, Jan./Feb. 2023.
Waqas et al., "Multimodal data integration for oncology in the era of deep neural networks: a review," Front. Artif. Intell., vol. 7, p. 1408843, Mar. 2024.
S. Xu et al., "Integrative analysis of histopathological images and chromatin accessibility data for estrogen receptor-positive breast cancer," BMC Med. Genomics, vol. 13, no. 1, pp. 1-12, Dec. 2020.
M. Zaveri et al., "Recognizing magnification levels in microscopic snapshots," in Proc. 42nd Annu. Int. Conf. IEEE Eng. Med. Biol. Soc. (EMBC), Jul. 2020, pp. 1416-1419.
H. Zeng et al., "Integrative models of histopathological image features and omics data predict survival in head and neck squamous cell carcinoma," Front. Cell Dev. Biol., vol. 8, p. 553099, Oct. 2020.
F. Zhang et al., "A deep-learning-based prognostic nomogram integrating microscopic digital pathology and macroscopic magnetic resonance images in nasopharyngeal carcinoma: a multi-cohort study," Ther. Adv. Med. Oncol., vol. 12, pp. 1-14, 2020.
G. Zhang et al., "Image-to-images translation for multiple virtual histological staining of unlabeled human carotid atherosclerotic tissue," Mol. Imaging Biol., pp. 1-11, 2022.
Y. Zhao et al., "Application of deep learning in histopathology images of breast cancer: a review," Micromachines, vol. 13, no. 12, p. 2197, Dec. 2022.
K. Zou et al., "A review of uncertainty estimation and its application in medical imaging," Meta-Radiology, vol. 1, p. 100003, 2023.
C. Cui et al., "Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review," Prog. Biomed. Eng., vol. 5, no. 2, p. 022001, 2023.