SegTME-UNI2: A Foundation Model-Based Framework for Generalisable Multiclass Cell Segmentation and LLM-Driven Tumour Microenvironment Characterisation in Histopathology

The article introduces SEGTME-UNI2, a framework for multiclass cell segmentation and tumor microenvironment characterization using histopathology images. Central to this framework is the UNI2-UPERHOVER model, which integrates the ViT-Giant foundation model with two UperNet decoders for six-class segmentation and gradient regression. The framework employs a three-stage progressive pseudo-label curriculum to improve model performance without weight transfer, utilizing a large dataset of TCGA-UT patches, and includes a feature extraction pipeline that generates interpretable clinical reports through a fine-tuned NVIDIA BioNeMo GPT model. The release of the pseudo-labelled TCGA-UT dataset and the model checkpoint facilitates advancements in TME profiling and spatial biology research.