UK Edition

Monday, 16 November 2026

Time Trade

Markets, trading & finance — British perspective

Funds

A Strong Foundation

Time Trade session note (2026-11-16): Abstract Biomedical Foundation Models (FMs) are transforming AI-enabled healthcare research and entering clinical validation. However, their susceptibility to… Primary source: original at ETF Trends (etftrends.com).

· ETF Trends

Abstract

Biomedical Foundation Models (FMs) are transforming AI-enabled healthcare research and entering clinical validation. However, their susceptibility to learning non-biological features — including variations in laboratory procedures and scanner hardware — poses risks for clinical deployment. We introduce PathoROB, a public benchmark quantifying FM robustness to non-biological features. Representation-level robustness is assessed using the robustness index, while output-level robustness is evaluated across clinically relevant settings, including patch- and slide-level prediction, case retrieval, and clustering tasks. Our experiments reveal robustness deficits across all 20 evaluated FMs, with substantial differences between them. We find that non-robust FM representations can cause major diagnostic downstream errors preventing safe clinical adoption. Using more robust FMs, vision-language alignment, and post-hoc robustification reduces (but does not yet eliminate) the risk of such errors. This work establishes that robustness evaluation is essential for validating pathology FMs before clinical adoption and provides a blueprint for assessing and improving robustness across biomedical domains.

Similar content being viewed by others

Introduction

Biomedical Foundation Models (FMs) are large-scale AI models pre-trained on increasingly large unlabeled biomedical datasets1,2,3,4. They drastically improved performance and generalization capabilities over standalone supervised models and non-biomedical pre-training across domains5,6,7,8,9,10,11,12. In digital pathology, FM pre-training has been scaled up to millions of Whole Slide Images (WSIs) and billions of model parameters13,14. Some of the resulting models demonstrate remarkable capabilities at a wide range of diagnostic tasks, such as pan-cancer classification or rare cancer detection6,15,16,17. They further advance the prediction of clinically relevant biomarkers from histology that typically require additional molecular or immunohistochemical testing — such as MSI, HER2, and EGFR6,18,19,20,21 — and enable real-world clinical utility of ML-based biomarkers21.

As the development of pathology FMs is progressing rapidly, measuring their capabilities and differences becomes increasingly challenging22. To this end, many recent efforts have focused on contributing pathology benchmarks to assess the performance potential of foundation models in various clinically relevant settings6,20,23,24,25,26,27,28,29,30. However, one major issue that deserves systematic analysis is the apparent lack of robustness of FMs to technical variability across medical centers (hospitals, laboratories, biobanks, etc.). Such variability is caused by numerous factors, including biopsy acquisition technique, tissue preparation and sectioning, staining protocols, and whole slide scanning, among other factors. These differences neither reflect medical nor biological tissue characteristics. Nevertheless, machine learning models can be negatively influenced by these types of variation31,32. Note that such systematic technical data biases, also known as batch effects33,34,35, are not limited to digital pathology, but pose a fundamental issue across biomedical disciplines, e.g., in radiology36,37 or molecular biology34,38,39,40.

Foundation models might be expected to provide more robust information thanks to their large and diverse pre-training datasets. However, the self-supervised learning methods applied to pre-train pathology FMs are designed to capture any differences in the data37, which includes technical variation. In fact, recent work suggests that pathology FMs encode technical/medical center information in their representations41,42,43,44,45,46. For example, Filiot et al.43 considered different stainings and scanners applied to the same slides and observed substantial variations in the resulting FM representations. Other factors prevalent in real-world diagnostic slides, such as differences in tissue fixation, section thickness, and quality, were not considered in that study43, though.

With this work, we intend to contribute to the above-described challenge by thoroughly investigating FM robustness, its medical consequences, and strategies for improving FM robustness. As a part of this endeavor, we constructed PathoROB, a benchmark for systematically measuring pathology foundation model robustness against non-biological variation across medical centers. It consists of four multi-class multi-medical center datasets from three public sources that facilitate comparisons between biological and non-biological signals present in FM representations of histopathology images. We present three metrics for assessing FM robustness and its implications: the performance drop in downstream tasks, a clustering score reflecting the global organization of the embedding space, and the robustness index: a metric measuring the degree to which foundation embeddings represent biological features rather than technical ones. Furthermore, we describe a framework to make foundation models more robust without retraining them and compare different ways to achieve this. Applying PathoROB to 20 current pathology FMs exposed substantial performance differences related to pre-training scale and objective, but also revealed considerable robustness deficits in the representation spaces of all FMs. A low robustness index was correlated with diminished generalization performance and potentially dangerous failures in clinically relevant FM applications, including supervised downstream models, image clustering, and diagnostic case search. Using post-training robustification methods like image-space stain normalization47 and representation-space batch correction48,49,50 considerably improved robustness and reduced the risk of downstream errors, but could not eliminate them fully. In summary, our work demonstrates the importance of including robustness criteria in FM development, FM alignment, or downstream model training.

Results

Multi-level robustness benchmarking with PathoROB

Foundation model representations in histopathology encode both biological features (e.g., cell shape and size, tissue architecture, presence of lesions) and technical, non-biological effects (e.g., medical center signatures: staining variations, scanner technology, tissue section thickness). In this work, we investigate FM robustness from two complementary perspectives: we assess the impact of non-biological features on FM representation spaces (representation-level robustness) and on FM-derived downstream models across various practically relevant settings (output-level robustness). We also expose connections between these perspectives. In the following, (i) we propose multiple measures for FM robustness that consistently reveal robustness deficits across a large class of existing foundation models. (ii) We further demonstrate that limited robustness can have fatal consequences for downstream tasks such as diagnostics. Finally, (iii) we present a framework for techniques that alleviate these shortcomings and lead to more robust models. As a basis for demonstrating both the shortcomings and their mitigations, we assembled a data resource composed of four histopathology datasets from three clinical sources51,52,53,54,55,56,57, each designed to have both multiple medical centers and multiple biological classes (specifically: normal vs. malignant, tumor types, tissue compartments) (Fig. 1a). Together, these datasets and our proposed robustness metrics form PathoROB, a robustness benchmark from real-world multi-center data for pathology foundation models (Fig. 1b). It covers a total of 99,392 patches from 28 biological classes and 34 medical centers (for dataset statistics, see Sup. Note 1). With PathoROB, we will in the following evaluate 20 popular foundation models covering various architectures, pre-training objectives, pre-training dataset sizes, and model sizes (see Table 1), resulting in insights into their potential and limitations.

Measuring biological representation robustness

We distinguish between biological features and confounding features. Biological features reflect a patient’s true condition, e.g., whether a tissue sample shows a particular subtype of lung adenocarcinoma; the aim and promise of foundation models is to capture such features reflecting actual underlying biology and morphology. We refer to all remaining features as confounding features, as they can bias downstream predictions.

We define biological representation robustness as the ability to capture biological features while ignoring confounding ones. The biological representation robustness index, or robustness index for short, quantifies the degree to which this ideal situation is reached by measuring the relative dominance of biological over confounding features. This metric results from an analysis of the representation space that can be performed for biomedical foundation models across all domains (see Fig. 2a and “Methods”).

Given a dataset, we collect the k nearest neighbors of all samples. From the set of nearest neighbors, we select the subsets of neighboring samples with either the Same biological class and Other confounding class (SO), or the Other biological class and Same confounding class (OS). Given these subsets of the k neighbors of the evaluation samples, the robustness index \({{\mathcal{R}}}\) is defined as the relation between the sizes of these sets:

It ranges from 0 (not robust) to 1 (fully robust). Specifically, \({{\mathcal{R}}}=0\) / \({{\mathcal{R}}}=1\) indicates that technical / biological features completely define the local neighborhoods in the FM representation space. For the motivation behind the metric and a more detailed description, see Methods and Supplementary Note 2.

The robustness index captures the relative strength of biological versus confounding encodings. The prevailing view of robustness, however, is restricted to the latter: insensitivity to noise, distribution shifts, or confounding features; e.g.,43. To capture this common notion of robustness, we define the confounder insensitivity metric that quantifies only how strongly FM representations encode confounding features (see “Methods”). We find, however, that confounder insensitivity correlates only marginally with output-level robustness in downstream models, viz. ρ = − 0.06 / 0.39 with the Average Performance Drop (APD) OOD / ID; see Supplementary Table 9. Furthermore, perhaps counterintuitively, models with better biological prediction performance also encode medical centers more strongly (Fig. 1d), yet often yield more robust downstream performance (see below). In line with this, the biological representation robustness index is highly correlated with output-level robustness (ρ = 0.80 / 0.90 with APD OOD / ID, see Supplementary Table 9), providing evidence for the practical utility of the metric. For a comparison of further representation robustness metrics and their correlation with output-level robustness, see Supplementary Note 2.9.

Representation robustness of pathology foundation models

As discussed above, for a foundation model to achieve biological representation robustness, its representation space should be dominated by biological features and not confounding technical features such as scanner type, H&E staining variations, or section thickness. We note that this is typically not the case (see Fig. 1c, more t-SNE plots in Supplementary Note 6.1): qualitatively, we find in the t-SNE plot that the representation space of the FM Phikon-v2, for example, is organized by medical center, showing that at the highest level, the model represents images primarily based on the medical center origin. In contrast, Virchow2’s representation space is observed as split by biological information (normal/tumor), with a secondary organization by medical center. Interestingly, the medical center of origin could still be reliably predicted from the feature vectors (Fig. 1d) with a mean medical center prediction accuracy between 88%–98% averaged over three datasets — a characteristic that is medically useless and potentially harmful.

On a quantitative level, Fig. 2b summarizes the robustness index results on the TCGA dataset. The models vary widely in biological representation robustness, and essentially, no models were found to have fully robust local neighborhoods, regardless of the neighborhood size k. Averaged over all datasets, the robustness index scores ranged from 0.446–0.861. Intuitively, this indicates that roughly 55.4%–13.9% of the local neighborhoods in embedding space were defined by the medical center instead of biological features. Image/text models (CONCH, CONCHv1.5) and recent large-scale self-supervised models (e.g., Virchow2, Atlas, H-optimus-0) show a higher degree of robustness, while smaller-scale models, which have been primarily trained on TCGA, express lower robustness scores (Ciga, Phikon, Phikon-v2, RudolfV, Kang-DINO, CTransPath).

Figure 2c shows the relation between biological representation robustness and the prediction accuracy of a k-nearest neighbors (KNN) classifier for predicting the biological class. We argue that robustness should be included as an additional objective in medical foundation model evaluation. Only two of the twenty models provide a Pareto-optimal tradeoff between prediction accuracy and robustness index: Virchow2 and Atlas. All other models score lower than these in either accuracy or robustness, or both. Prediction accuracy and robustness provide different, complementary information, with only limited correlation (Spearman rank correlation; two-sided permutation test: ρ = 0.549, p = 0.0128, 95% CI [0.140, 0.798]); this highlights the importance of measuring and optimizing both accuracy and robustness, and different models provide different tradeoffs.

The three image/text models (CONCH, CONCHv1.5, MUSK), while lacking behind top vision-only prediction accuracy, were considerably more robust than many vision-only (SSL) models with comparable or higher prediction accuracy, indicating that additional language supervision (with primarily biological content in the captions) might have helped to suppress the confounding factors, thus increasing biological representation robustness.

When only considering SSL models, we can observe a strong correlation (Spearman rank correlation; two-sided permutation test: ρ = 0.692, p = 0.0047, 95% CI [0.278, 0.889]) between the logarithmic number of slides used for FM pre-training and the robustness index (Fig. 2d). This highlights that training on larger, diverse datasets also leads to improved biological representation robustness. Yet, despite large-scale pre-training, no foundation model was close to achieving a perfect robustness score.

Overall, these results highlight that larger SSL models trained on more diverse datasets or the use of image/text objectives yield increased performance. While representation robustness measures how foundation model embeddings already prioritize relevant biological over technical data features, it needs to be shown how representation robustness impacts the robustness of clinically relevant downstream tasks (output-level robustness). This is done in the following three sections for different tasks relevant for application for routine diagnostics and discovery.

Robustness of supervised ROI prediction models

One of the most important use cases of (upstream) foundation models is the development of task-specific downstream models for region-of-interest (ROI) classification. For this, we train a shallow neural network on top of the FM representations of a (small) labeled dataset to predict a certain biological target. However, if the FM representations carry medical center signatures, the downstream model may learn to rely on these technical features rather than on true biological signals to make its predictions (aka. Clever Hans / shortcut learning31,37,58,59,60,61,62). Furthermore, there are situations in which the downstream training process cannot distinguish which features reflect true biological signals vs. technical artifacts. These issues will lead to suboptimal generalization and prediction errors on unseen data. An in-depth explanation of Clever Hans learning, its connection with spurious correlation, and a constructed example are provided in Supplementary Note 3.

To corroborate this point, we trained downstream models on data with increasing correlation between medical centers and biological targets (varying degrees of Cramér’s V), and measured their accuracy on unseen data from the same medical centers (in-domain generalization, ID) and from previously unseen medical centers (out-of-domain generalization, OOD) (Fig. 3a; see Methods and Supplementary Note 4 for details). Intuitively, such correlations would incentivize the model to exploit medical center information in the FM representations, as they help to solve the training task, even though they are clearly not helpful for generalization. Such spurious correlations are frequently found in histopathology datasets due to a class imbalance across contributing centers, particularly for rare diseases46,63,64 (see Supplementary Note 3 for details). By systematically varying the degree of spurious correlation, we can quantify how performance degrades as confounding increases and assess when the confounder transitions from being merely encoded in the representation to driving predictions, leading the downstream model to rely on medical center information instead of the biological signal. With increasing spurious correlations, we observed that the generalization performance deteriorated across all foundation models and prediction tasks (Fig. 3b, top). For Camelyon, where the visual differences between medical centers are strong, the tumor detection accuracies dropped from > 92% for balanced training data (Cramér’s V = 0) to 53–87% for fully correlated data (Cramér’s V = 1), indicating that downstream models learned to exploit medical center signatures next to biological features regardless of the foundation model used. In TCGA and Tolkach ESCA — datasets with more subtle center differences — the generalization performance was slightly more stable, but the drops observed for most FMs were arguably still unacceptable for clinical applications (TCGA: −1% to −25%, mean −12%; Tolkach ESCA: −0% to −14%, mean −5%). Note that decreases in accuracy could even be observed for moderate levels of spurious correlation in the training data, depending on the foundation model. The out-of-distribution generalization performance also declined as training data correlation increased, although the drop was generally lower than that for in-domain data (Fig. 3b, bottom). A closer look into the FM representations revealed a potential explanation for the observed Clever Hans learning effect: irrelevant medical center information is strongly encoded along the directions of greatest variance in the FM representations, making it readily accessible for downstream models to exploit (see Supplementary Note 6.2 for a detailed feature space analysis and elaboration).

For foundation model comparison, we aggregated the diminishing generalization accuracy into an average performance drop metric that quantifies the robustness of FM-derived supervised downstream models to Clever Hans learning (see “Methods” and Supplementary Note 4 for details). The FMs with the highest biological representation robustness (Virchow2, CONCHv1.5, Atlas) also produced the most robust downstream models, with CONCHv1.5 delivering the most stable performance on in-distribution data (−0.83% relative decrease on average) and Atlas in out-of-distribution settings (≈ 0% performance drop) (Fig. 3c). Downstream models trained on top of representations from H0-mini, Virchow, CONCH, H-optimus-0, and UNI2-h were generally less stable (ID: −1.61% to −2.76%, OOD: −0.19% to −1.34%). Most other FMs suffered from much more severe relative decreases (up to −7.83% on ID and −9.40% on OOD). Overall, the average performance drop was strongly correlated with the robustness index (Fig. 3d, Spearman rank correlation; two-sided permutation test: ρ = 0.904, p = 4.72 ⋅ 10−8, 95% CI [0.769, 0.962] for the in-domain and ρ = 0.798, p = 2.44 ⋅ 10−5, 95% CI [0.550, 0.917] for the out-of-domain generalization). This indicates that more robust foundation model representations lead to more robust downstream models under heterogeneous clinical case distributions (aka. imbalanced multi-site training data).

In addition to training shallow neural networks on top of frozen FM representations, we also finetuned the foundation model weights to align the FM representations to the respective downstream prediction tasks. We observed that both full and parameter-efficient finetuning exacerbated the issue of Clever Hans learning and led to even stronger performance drops (see Supplementary Notes 7.1-7.2).

To further elucidate the medical implications of limited FM robustness and Clever Hans learning, we inspected the predictions of the shallow downstream models trained on fully correlated data (Cramér’s V = 1) for the Camelyon tumor detection task. Figure 4a showcases the exploitation of medical center signatures as a Clever Hans feature. For balanced training data (Cramér’s V = 0), i.e., homogeneous clinical case distributions across contributing medical centers, the downstream model derived from the Virchow2 representations predicted all displayed samples correctly, as expected. However, for correlated training data (Cramér’s V = 1), i.e., clinical case distributions where the hospital is a confounder, it made obvious mispredictions: most notably, it mistook a morphologically unequivocal tumor patch for a normal patch based on its medical center origin (right). Figure 4b confirms that these mispredictions were a structural issue. On average, between 28% (Atlas, Virchow2) and 94% (Phikon-v2) of normal patches from the RUMC medical center were incorrectly predicted as tumors. More importantly, between 19% (Virchow) and 99% (HIPT) of tumor patches from the UMCU medical center were not recognized or misclassified. Notably, here, foundation models with better robustness scores (Virchow2, Atlas, CONCHv1.5, Virchow) yielded significantly lower false negative rates than less robust FMs (e.g., H-optimus-0, Prov-GigaPath, MUSK, Phikon-v2).

We then applied the downstream models to predict all patches of unseen whole-slide images from the UMCU medical center (Fig. 4c). Despite a large number of pre-training images and strong performance on standard benchmarks (https://github.com/bioptimus/releases/tree/main/models/h-optimus/v0), the foundation models with lower robustness ratings (shown for UNI2-h, Phikon-v2 as representatives) did not enable the identification of the tumor areas. With Virchow2 as one of the most robust models in terms of our metrics, in contrast, recognition of larger and smaller tumor areas was still possible, even though the small tumor area (bottom slide) only received low prediction scores. Similarly, the CONCH-based downstream model still managed to highlight all tumor regions. Notice that CONCH achieved high robustness ratings, even though it was pre-trained on a significantly lower amount of whole-slide images and was mostly outperformed by the other three foundation models in regular biological benchmark tasks (see, e.g., ref. 26).

In summary, our results show that — in the typical practical case where homogeneous training data across medical centers cannot be guaranteed — FM-derived downstream models are prone to making severe mistakes that rule out their safe use in high-stakes clinical settings. Using more robust FM representations (e.g., obtained via vision-language alignment) or downstream models, however, profoundly decreases the risk of such errors and thus increases practical utility and safety.

Robustness of weakly supervised slide prediction models

In addition to ROI classification (see above), pathology foundation models serve as backbones for clinically important whole-slide image tasks, including the prediction of complex molecular biomarkers and patient outcomes21,65,66. Note that slide-level downstream tasks typically involve weakly supervised training of model architectures that aggregate thousands of feature vectors into a single slide prediction.

We investigated model robustness in the context of four practically relevant slide-level applications covering tumor detection, disease subtyping, biomarker inference, and overall survival prognosis (see Fig. 5a). Specifically, we trained Attention-based MIL models67 on WSI data with increasing correlation between medical centers and biological targets and measured their out-of-domain generalization performance on slides from unseen hospitals. Note that label imbalances across medical centers, which lead to such spurious correlations, naturally occur in three out of the four datasets that we considered (see Fig. 5a). For example, the mean observed overall survival time (censored data included) of breast cancer patients in TCGA was 81.4 months in Mayo (May) but only 19.4 months Greater Poland Cancer Center (GPCC), incentivizing downstream models to associate medical center features with patient survival risk.

As spurious correlations increased, we again observed that cross-hospital generalization performance almost always deteriorated across tasks and FM backbones (Fig. 5b, top). Notably, performance drops were consistently observed in lung cancer subtyping and homologous recombination deficiency (HRD) biomarker prediction for breast cancer patients. This shows that non-biological features in FM representations also negatively impact biomarker inference and slide-level models in general. For Camelyon tumor detection, the performances for all FMs dropped to (close to) random on fully correlated data (AUCs of 0.47–0.60), even for FMs with relatively high biological representation robustness, such as Virchow2 and CONCHv1.5. Interestingly, the performance drops were stronger than for the patch-level tumor detection models (see results for Camelyon in Fig. 3). In addition, we observed increasing performance variance across the unseen medical centers as spurious correlations increased for three out of four tasks (Fig. 5b, bottom). This indicates that stronger Clever Hans learning also makes downstream models less likely to maintain performance across hospitals. Further FM comparison results are provided in Supplementary Note 7.3.

Another consequence of lacking model robustness to Clever Hans learning was revealed in the context of overall survival prediction for breast cancer patients (Fig. 5, right). Specifically, adding training data from new hospitals with divergent label distributions to an existing training set decreased generalization performance for four out of six FMs (from 0.576 to 0.556 C-Index on average), despite providing the downstream models with 67% more data points to learn from.

In summary, FM-based slide-level downstream models are also prone to making severe mistakes that undermine their safety and trustworthiness, especially when applied to previously unseen medical centers. This is particularly dangerous in high-stakes biomarker and outcome prediction, where model validation is typically even more challenging than for routine diagnostic tasks.

Robustness of unsupervised clustering and case retrieval

Another key application of pathology foundation models is generating insights directly from the FM representations of diagnostic cases, e.g., through clustering (grouping morphological patterns) or retrieval (search for similar samples)19,25,32,68,69. In light of the presented robustness deficits in foundation models, we investigated how medical center signatures might impact these unsupervised algorithms.

Clustering is used, for example, to uncover novel disease subtypes or clinically relevant morphological patterns32,70,71. The underlying assumption is that samples with similar biological characteristics are close to each other in the FM representation space and will therefore form clusters. However, if medical center signatures influence the distances between embeddings, to-be-discovered clusters may be built around non-biological factors instead. Figure 6a showcases this issue qualitatively. In Phikon-v2, clustering is predominantly driven by medical center origin, with Cluster 1 composed mainly of UMCU patches and Cluster 2 of RUMC patches. As a result, the clusters are biologically heterogeneous and do not facilitate subtype discovery. Biologically similar patches from different centers are not grouped together, preventing the identification of biological commonalities across the cohort and medical centers. In contrast, CONCH representations yield biologically coherent clusters that combine patches of both medical centers but are separated by tumor status. This organization facilitates hierarchical analysis of intra-cluster biological similarities and inter-cluster biological differences.

For a quantitative assessment, we define a clustering score that measures the robustness of FM-derived clusters to non-biological features. It favors biologically meaningful groupings while penalizing clusters driven by medical center, and ranges from −1 (pure medical center clusters) to 1 (perfect biological clusters), with 0 implying equal influence of both factors or random clustering. In contrast to the robustness index, which measures the influence of morphology and artifacts on the local neighborhood of each representation, the clustering score evaluates their impact on the global structure of the representation space (see Methods for details).

Figure 6b highlights that, due to the impact of medical center signatures on FM representations, clustering often yields imperfect or poor results with clustering scores close to 0. Therefore, samples coming from the same medical center will frequently end up in the same cluster, even if their morphologies differ. Consequently, the utility of clustering in multi-center datasets, e.g., for identifying new disease subtypes or patterns, is substantially limited for FMs with unrobust representations. Earlier models, such as Ciga, HIPT, CTransPath, and Phikon, tended to be more susceptible to center signatures, while more recent methods like Atlas, CONCHv1.5, and CONCH achieved the highest clustering scores across all datasets. Surprisingly, Virchow2 produced considerably worse clustering scores, indicating that its clusters were heavily influenced by medical centers. Furthermore, the difficulty of clustering appeared to be dataset-dependent. In Tolkach ESCA, biological information was more salient, while in Camelyon, a more balanced influence of both medical center and biological features was observed. TCGA proved particularly challenging, especially LUSC-LUAD, for which none of the FMs produced representations suitable for accurate subtyping.

We furthermore observe a positive correlation between the robustness index and clustering score across datasets (Fig. 6b). This supports the notion that coherent local neighborhoods tend to reflect consistent global structures, suggesting that the robustness index may be predictive of the FMs’ clustering quality in multi-center datasets. However, some models exhibited high robustness indices but clustering indices near zero, meaning that the local structure may reflect biological information, whereas the global clustering is influenced jointly by biological and confounding factors (this is further explored in Supplementary Note 7.6).

We further assessed the efficacy of retrieving histologically similar images from a case database using the foundation model representations in the context of limited robustness (see Supplementary Note 7.4 for details). This clinically relevant application has recently gained attention19,68,69. The issues inherited from non-robust FM representations encountered for downstream models also re-occurred in this setting (Fig. 6c and Supplementary Note 7.4). We observed worse performance and mispredictions when retrieval databases had heterogeneous clinical distributions across medical centers. This means qualitatively that retrieval for multi-site data is more likely to fail if the retrieval database cannot be guaranteed to have balanced data contributions from all medical centers. Quantitatively, however, the retrieval accuracies decreased more gently and did not drop as steeply as in the supervised downstream setting (Fig. 3b), and were stable for out-of-domain patches when using some of the foundation models with a higher robustness index (Atlas, CONCHv1.5, CONCH, MUSK, Virchow2, Virchow, Prov-GigaPath, H-optimus-0). Overall, foundation models with more robust representations led to better retrieval results in heterogeneous databases (Fig. 6c).

Robustification of foundation and downstream models

We will now discuss a framework to alleviate or correct the shortcomings of non-robust FM representations and downstream models. To address this, we explored approaches to robustify FM representation spaces for downstream analysis without requiring FM retraining, as the latter is generally costly and often practically impossible.

We studied three different approaches (Fig. 7a): (i) data robustification (DR), i.e., directly removing medical center signatures (e.g., staining or scanner artifacts) from the images, (ii) representation robustification (RR), i.e., removing signatures after feature extraction in the FM representations, and (iii) training robustification (TR), i.e., preventing downstream models from using the signatures as Clever Hans features for their predictions. We assessed all of these approaches in the context of specific downstream applications, i.e., using task-specific data for robustification.

Specifically, as a typical DR approach, we inspected Reinhard stain normalization47, which harmonizes staining differences in image patches. For RR, we further adopted ComBat batch correction48,49, which fits an empirical Bayesian framework to identify and remove batch effects from high-dimensional vectors. Although originally developed for molecular data, it has recently shown potential for correcting histopathology representations50. Finally, for TR, we assessed domain-adversarial neural network (DANN) training72, which penalizes the use of medical center features in downstream models via an additional loss term (see “Methods” for details).

Improved robustness index

We first re-computed the robustness index on the robustified FM representation spaces (Fig. 7b). Reinhard stain normalization considerably improved biological representation robustness for most foundation models, with relative increases of +19.1% on average. Strikingly, ComBat enhanced it even further (+28.2% on average), suggesting that the approach of representation robustification (RR) holds substantial potential for making FM representations more biologically meaningful. The most robust FM representations were achieved by combining both methods (DR+RR), yielding a robustness index of 0.915 and higher (Virchow2, Atlas, UNI2-h). The largest jumps were observed for the models with initially low robustness index, with a relative rise of up to +70.3% (Phikon), and UNI2-h, H-optimus-0, H0-mini, CONCH, and Prov-GigaPath achieving robustness indices close to the best reported scores (0.885–0.915) after robustification.

Improved clustering

Similar effects can be observed for clustering on the robustified representations (detailed results in Supplementary Note 7.7). Robustification strategies led to an increase in clustering scores and thus clustering quality. ComBat (RR) yielded the most consistent improvements, even transforming the lowest observed clustering score of Phikon-v2 on the Camelyon tumor detection dataset from −0.99 (indicating a clustering purely by medical center) to a significantly improved +0.61 (i.e., biologically more meaningful clusters). Robustification was particularly beneficial for the clustering quality in FMs with originally lower robustness ratings, whereas models that already exhibited high clustering scores without robustification saw only smaller gains.

Improved generalization

To understand to what extent the proposed robustification of FM representations can alleviate the negative consequences of non-robust foundation models, we re-assessed the generalization performance of downstream models (see Fig. 7c). We find that on in-distribution test data, Reinhard stain normalization (DR) could reduce Clever Hans learning and improve the generalization performance for most FMs, on average by 1.11%pt (max. +3.28%pt). Notably, the approach was most effective for training data with strong spurious correlations (see Supplementary Note 7.5). Domain-adversarial training (TR) only increased performance for 12 out of 20 FMs, leading to a weaker positive effect of +0.23%pt on average (max. +2.40%pt). Reinhard and DANN could be combined to yield the highest increase of 1.30%pt on average (max. +3.48%pt), although using Reinhard stain normalization alone was still more effective for some FMs (e.g., UNI2-h, Virchow). After robustification, some FMs that were originally rated lower in representation robustness (i.e., H0-mini, UNI2-h, MUSK) enabled downstream model performance comparable to Virchow2, Atlas, and CONCHv1.5. Similar trends were observed for out-of-distribution test data (see Supplementary Note 7.5).

Interestingly, despite substantially enhancing the robustness index scores, ComBat (RR) did not consistently lead to better downstream models; it even exacerbated the performance drops compared to the setting without robustification. More precisely, despite effectively reducing mispredictions for slightly heterogeneous training data, it likely removed important biological signals when they were strongly correlated with medical center information (see Supplementary Note 7.5). This suggests that ComBat may only be effective if the data contribution of each medical center covers all biological characteristics present in the cohort to be analyzed.

We further find that no method entirely eliminated the generalization performance drops, i.e., supervised downstream models still learned aspects of medical center signatures as Clever Hans features instead of exclusively focusing on biological signals. A closer look into the FM representation spaces provided a potential explanation: medical center information is often entangled with biological features. Specifically, we found that biological and technical information was not encoded in separate linear directions of the FM representations; instead, the same linear directions often contained both types of signals (see Supplementary Note 6.2 for details). Therefore, eliminating medical center signatures also risks damaging important biological signals.

In summary, representation robustification via ComBat has the potential to greatly improve FM biological representation robustness, but may also remove aspects of the biological signals when data distributions are heterogeneous across medical centers. Training robustification via DANN was found to be limited in effect and applicability. Data robustification via stain normalization brought consistent improvements, but cannot achieve complete robustification since it ignores technical factors other than staining. Nonetheless, our results demonstrate that robustifying FM representations can alleviate performance drops without the need for retraining FMs.

Discussion

Foundation models have become a de facto standard across fields1,2,7,8,11,12,19,73,74,75,76,77,78,79,80,81 due to their broad application scope and their capacity to lower data requirements for downstream models. Ideally, their representations reflect the underlying (biomedical) problem well. Recently, however, several studies have demonstrated that the learned representations have the tendency to reflect local correlations well but can only suboptimally model long-range or semantic problem structure82. Moreover, representation learning can be subject to Clever Hans effects due to systematic failures occurring in unregularized unsupervised learning37.

In this work, we have contributed a further aspect to this discussion that has so far not received sufficient attention by the community, namely, the sensitivity to non-biological, technical features that we show to cause critical failure in pathology FMs. We intentionally built our analysis on real-world medical center differences, because they provide more complex non-biological features than controlled variations of staining or scanner. We analyzed 20 current pathology FMs and observed that learned representations entangle biologically meaningful (and desired) information with spurious confounders such as the scanner type or medical center origin. We furthermore see that such confounders can give rise to critical failures when the pathology FMs are applied in diagnostic downstream tasks, biomarker prediction, outcome prognosis, clustering, or retrieval.

Our work has systematically contributed to a better understanding of pathology FMs by (i) establishing multiple measures of robustness, (ii) introducing benchmarks that allow to quantify (non-)robustness, (iii) demonstrating consequences of non-robustness in diagnostic tasks, and finally (iv) proposing a framework to robustify representations without the need to re-train FMs.

Each of our robustness measures comes with advantages and limitations. The robustness index enables a direct, deterministic, and data-efficient assessment of the impact of non-biological features on FM representations. It is independent of downstream modeling choices, which renders it particularly suitable for FM benchmarking. In contrast, the investigation of downstream model robustness to Clever Hans learning (average performance drop) and the robustness of clustering to non-biological features (clustering score) quantifies negative consequences in FM-derived outcomes in medically relevant applications. These can also be affected by variations coming from the downstream model, unrelated to the FM itself. In our study, we analyzed both representation-level and output-level effects and, interestingly, found strong correlations between the representation-level and output-level effects, suggesting that downstream applications inherit robustness deficits from FM representation spaces.

We found that the strength of biological feature encoding relative to the strength of technical feature encoding is more predictive of downstream robustness than confounder insensitivity alone. Intuitively, if the embedding space structure reflects biological features more strongly than confounding ones, downstream models may be more likely to use biological features for their decisions.

Let us put our findings into a broader perspective. First, our results show the importance of FM robustness, suggesting the inclusion of robustness criteria in future foundation model development and improvement to be a promising pathway. Integrating the proposed benchmark metrics into foundation model pre-training or post-training alignment may lead to more robust representation spaces that are better suited for clinical translation and research applications. Robust FM representations are particularly desirable when only a small amount of data from multiple medical centers is available, e.g., for detecting or characterizing rare diseases. Second, our structured approach to analyzing and improving FMs may serve as a blueprint for other biomedical fields, as batch effects are prevalent in most biomedical data34,36,37,38,39,40; subsampling balanced multi-site datasets, computing the robustness index, and measuring generalization performance drops are not conceptually limited to pathology. Thus, our framework could also contribute to improving the robustness of radiology or omics foundation models, among others.

An important debate has revolved around the question of whether making foundation models robust is desirable at all. In the context of managing biases in ML models based on sex or race, Weng et al.83 argued that foundation models should ideally contain as much information as their underlying data, as removal of supposedly irrelevant information could have unintended adverse effects. Extending this thought to medical center information, one could imagine cases where removing them leads to the elimination of relevant biological patterns of subpopulations that are overrepresented in a specific medical center, as center-specific biological information could be mistaken for technical artifacts. Instead, Weng et al. proposed to tackle biases through careful downstream model training and evaluation. However, we argue that collecting sufficient downstream training and evaluation data will often be impossible in medicine. For rare diseases, for example, the few available cases may need to be drawn from various medical centers with highly heterogeneous data characteristics. Here, robustified foundation models could have an enormous impact on downstream model efficacy and, therefore, on the quality of patient outcomes in clinical settings.

Adding to the debate, our findings indicate that robustification of FM representations may yield a disentanglement of biological and confounding features (such as medical center information), which gives rise to improvement in generalization. We stress that this aspect has so far been overlooked and not yet fully understood; thus, more technical efforts should go into this important direction. A promising direction for this could be the use of advanced explainable AI techniques to identify and mitigate undesired entanglement84,85.

Although the datasets and experiments presented in our benchmark are still somewhat limited in complexity, they were highly instructive in revealing limitations and differences between popular foundation models. Yet, as FMs have different strengths and weaknesses, we note that the robustness scores and the observed correlations between them may also differ when considering datasets beyond those already included in PathoROB. Clearly, along with foundation models becoming increasingly robust in the future, the quest for more complex pathology prediction tasks to find more pronounced differences will need to continue. It will be particularly interesting to extend PathoROB to datasets relevant for biomarker or outcome prediction, e.g., by utilizing matched spatial transcriptomics data20,86 or including slide-level evaluations. Furthermore, the subsampling of data to have exactly one biological class and one medical center dimension, with carefully designed proportions, served the purpose of furthering our understanding of FM representations. More complex scenarios reflecting general clinical scenarios will be useful to gain more insights into FM robustness. To extend the concept of a biological class to multi-dimensional or continuously-valued patch labels (e.g., gene expression measurements), biologically consistent patch groupings could be derived via clustering, for example.

Finally, we focused our efforts on patch-level vision foundation models. Yet, practical applications often require slide-level predictions, which can be obtained via aggregation models (i.e., multiple instance learning)67,87,88 or pre-trained slide representation models66,89,90. Further, vision-language foundation models enable use cases beyond the ones considered here, including image captioning or zero-shot predictions. However, patch representations are an essential building block in all of these applications, leading us to believe that the patch-level encoders are most relevant to evaluate for robustness. Nonetheless, robustness evaluation and comparison of slide-level and vision-language foundation models form a logical extension for further work.

Paving the way towards more robust foundation models, we believe that future research should focus on furthering pre-training strategies — particularly on how to favor alignment with clinical semantics while mitigating biases and ensuring transferability to downstream tasks. Beyond refining the pre-training process, our study highlights a promising complementary research direction: post-hoc robustness improvements of FM representation spaces, especially via alignment. We presented evidence that post-hoc vision-language alignment may reduce robustness issues. In addition, similarly to how modern LLM and vision training pipelines employ instruction tuning, reinforcement learning from human feedback (RLHF), or alignment to human representations as a crucial post-training step to mitigate harmful outputs82,91,92, future pathology foundation models may benefit from analogous post-training procedures to reduce the impact of undesired non-biological features. This could offer a practical alternative to the extensive retraining of new pathology foundation models on the path to robust, trustworthy, and reliable AI models.

Methods

Dataset subsampling

We subsampled datasets from clinical sources that enable a comparison between biological signals and real-world medical center signatures in the representations of foundation models. For this, we considered patch-level data where each patch represents a distinct biological class, e.g., a patch from the tumor area of a lung adenocarcinoma or a lung squamous cell carcinoma. In addition, we required that the patches originate from different medical centers. A key idea was to make the subsampled datasets complete and balanced: ideally, every medical center contributes the same number of cases, slides, and patches per biological class. Furthermore, we tried to ensure that systematic differences between medical centers are solely due to biologically irrelevant factors by sampling such that known medical and demographic characteristics are comparable across medical centers.

Following these principles, we subsampled four datasets from three public sources: (i) A subset from the Camelyon cohorts for tumor detection51,52,53, with two biological classes (normal vs. tumor) and two medical centers, plus three additional medical centers for out-of-domain (OOD) generalization. (ii) Two subsets from TCGA-UT54,55 for tumor type prediction. The first (TCGA 4 × 4) contains 4 biological classes (BRCA, COAD, LUAD, LUSC) and four medical centers (plus four OOD medical centers). The second (TCGA 2 × 2) comprises 94 class-class-center-center quartets, consisting of two biological classes and two medical centers each. (iii) One subset from the Tolkach ESCA resource56,57 for tissue compartment classification in oesophageal resections, with six biological classes (tumor, mucosa, muscularis propria, etc.) and three medical centers (plus one OOD cohort). Note that we did not consider sex, gender, or ethnicity in the study design, as we anticipate that technical biases (e.g., from tissue processing, staining, scanning) are relevant for any patient subgroup. Nevertheless, our datasets represent female and male patients (self-reported gender). Specifically, Camelyon contains only female patients, while the Tolkach-ESCA test set that we sampled from contains 88% male patients and 12% female patients. The TCGA data used in PathoROB represent 1024 male and 998 female patients. Dataset statistics and sampling details are provided in Supplementary Note 1.

Foundation models and feature extraction

In total, we evaluated 20 foundation models in our study that encompass a diverse range of architectures (from convolutional networks to vision transformers), utilize various pre-training objectives (including SSL objectives such as SimCLR, DINO, SRCL, iBOT, DINOv2 and image/text models such as CONCH and MUSK), vary greatly in dataset size (from 6k to 3.1 million WSIs), and span multiple scales in model capacity (from 11 million to 1.1 billion parameters) (see Table 1). For each SSL ViT model, we extracted the CLS and mean pooled patch token representations and concatenated them to a final representation, which was used throughout all analyses in the paper. For the image/text models, we used the recommended layers and settings on the respective Hugging Face site. For CNN models (Ciga, RetCCL), we simply used the global average pooling representation at the end of the encoder.

Representation space visualization and target prediction

To visualize foundation model representation spaces, we used the default t-SNE implementation from the sklearn package (https://scikit-learn.org/stable/modules/generated/sklearn.manifold.TSNE.html) on extracted representation vectors from the Camelyon dataset for each of the four cancer/normal × RUMC/UMCU categories. For the computation of the t-SNE, we chose the perplexity as the optimal k value derived during the robustness index calculation (see Supplementary Note 2.2). For measuring biological class and medical center prediction accuracies per foundation model, we trained linear probing models (aka. logistic regression), i.e., a simple neural network head without hidden layers, to predict either biological class or medical center (training details in Supplementary Note 4). For this, we employed the subsampled PathoROB datasets of Camelyon, TCGA 4 × 4, and Tolkach ESCA. The data were split into approximately 0.6/0.1/0.3 train/validation/test on the slide level. We report the test set accuracies averaged over 20 repetitions and the three datasets with 95% confidence intervals (± t0.975,59 ⋅ SE), corrected to remove common variance due to dataset (Masson & Loftus93).

Robustness index

The robustness index is inspired by k-nearest neighbor (KNN) classification, one of the earliest, simplest, and most widely used machine learning methods for classifying samples based on feature vectors94,95. KNN classification and related methods find the k neighbors closest to a sample as determined by the distances between their feature vectors. We consider which neighbors have the same biological class and which have the same confounding class. This divides the neighbors of a sample into four groups; see Fig. 8.

The idea behind the robustness index is to express the degree to which models capture biological rather than confounding information. Neighbors with both the same biological and confounding class (SS) are irrelevant, however, as there is no way to determine whether the sample is close due to having the same biological or the same confounding class. The same goes for the OO class. We exclude the SS and OO combinations, therefore, and thus restrict the analysis to the neighbors that have either the same biological class (SO) or the same confounding class (OS).

For each sample in a given evaluation dataset D containing n samples, we obtain the k nearest neighbors. From this set of n ⋅ k neighbors, we select the subsets of neighbors with either the Same biological class as the sample of which it is a neighbor and the Other confounding class (SOk), or the Other biological class and the same confounding class (OSk). The robustness index \({{{\mathcal{R}}}}_{k}\) is then defined as:

The robustness index expresses the degree to which biological rather than confounding features dominate the neighborhood of a sample in the representation space. The robustness index provides an easily interpretable metric that is shown in this work to capture a relevant dimension of model performance. For further results and details regarding the robustness index, see Supplementary Note 2.

Visual explanation: from category frequencies to robustness index

To illustrate how the frequencies of these four categories determine robustness index values, we visualize the computation using real data. Figure 9 shows the robustness index calculation for CTransPath, an earlier, often-cited pathology foundation model, and CONCHv1.5, a recent, more robust Vision-Language Model, on TCGA 2 × 2.

Figure 9 a shows the frequencies of all four categories (SS, SO, OS, OO) for both models, representing only the neighbors at each specific distance rank k. The frequencies of SS (blue) and OO neighbors (red) are similar for both models and do not play a role in the calculation. Figure 9b aggregates this information over the k nearest neighbors, resulting in more stable curves. Large differences are seen for the SO and OS lines in both a and b; the CONCHv1.5 model places SO neighbors nearby much more often than CTransPath, as shown by the higher orange line for k values below the midpoint of 300, and correctly places OS neighbors further away, as seen by the lower green line.

Figure 9c focuses on the SO and OS lines, and normalizes these to sum to one. The resulting normalized SO curve equals the robustness index. The higher prevalence of SO neighbors and the lower frequency of OS neighbors combine to yield a substantially higher robustness index for CONCHv1.5 across the full range of k values.

Additional robustness metrics

The diagram in Fig. 8 defines the robustness index and several additional metrics of robustness. Below, we describe these in more detail. Further metrics are defined in Supplementary Note 2.

  • Confounder Insensitivity purely measures robustness in the common sense of insensitivity to confounding features. This can be seen in different ways:

    • Figure 9b reveals the signature of confounding bias: among the 400 nearest neighbors sharing the same biological class, unrobust models heavily overrepresent same-confounding-class neighbors (SS, blue) relative to other-confounding-class neighbors (SO, orange). A fully robust model shows no such bias — SS and SO neighbors appear with equal frequency, as do OS and OO neighbors. The robustness metric reflects the ratio between these frequencies and equals 1 for robust models.

    • The overrepresentation of same-confounding-class neighbors implies that the neighborhood composition is predictive of the evaluation sample’s confounding class. The relative frequency of same-confounding-class neighbors (SS + OS)/(SO + OO) is a metric of this predictability. The inverse of this predictability (SO + OO)/(SS + OS) measures insensitivity to confounding features and is thus a measure of robustness.

    $$\,{{\rm{Confounder}}}\; {{\rm{Insensitivity}}}\,=\frac{SO+OO}{SS+OS}$$(3)
  • The Bio-vs-confounding ratio measures the relative dominance of same-biological-class neighbors over same-confounding-class neighbors, corresponding to our definition of biological representation robustness. This equals the earlier robustness index variant proposed in ref. 42.

    $$\,{{\rm{Bio}}} - {{\rm{vs}}} - {{\rm{confounding}}}\; {{\rm{ratio}}}\,=\frac{SS+SO}{SS+OS}$$(4)
  • The robustness index proposed here is a simplified variant that removes SS neighbors from the equation, since their proximity may reflect either biological or confounding similarity, and they contribute to both numerator and denominator.

ROI prediction model training and evaluation

Split generation

We subsampled training splits from our PathoROB datasets (Camelyon, TCGA 4 × 4, Tolkach ESCA) in which the medical center is increasingly correlated with the target to be predicted. The first training split (Split 1, Cramér’s V = 0) was fully balanced, i.e., every medical center evenly contributed the same number of patches per disease class. In Split 2, each medical center had one or more overrepresented disease classes, for which more data points were available than for the other classes, introducing a weak correlation between the medical center and the prediction target. This correlation was gradually increased until the final split (Cramér’s V = 1). Here, each medical center contributes data for some but not for all other classes, resulting in a strong (or even perfect) correlation between the medical center and the biological target. In each split, the total number of training data points was the same overall, per disease class, and per medical center; only the dataset composition differed. For patches of each of the datasets, we ensured there was no slide overlap between the training and test sets. We held out a balanced in-distribution (ID) test set (different cases from the same medical centers) and a balanced out-of-distribution (OOD) data (different cases from different medical centers), allowing us to assess the generalization performance on data where the model cannot rely on correlations between medical centers and biological targets. Further details and the complete split statistics are provided in Supplementary Note 4 and Supplementary Figs. 7–9.

Downstream model training and evaluation

As a downstream model, we trained linear probing (aka. logistic regression), i.e., a simple neural network head without hidden layers to predict the biological classes (training details in Supplementary Note 4). We note that linear probing is also one of the most prevalent evaluation protocols in computational pathology13,14,15,20,23,89. For evaluation, we calculated the accuracy on the ID and OOD test sets per dataset, model, and training split, and reported the average over the 20 repetitions with 95% confidence intervals (± t0.975,19 ⋅ SE). For FM comparison, we aggregated the results into an average performance drop (APD) per model: For a dataset D, model M, and repetition r we define this as

An APD of 0% corresponds to no generalization performance drops across splits, implying that the downstream model predictions did not rely on medical center information despite training data correlations incentivizing this. Increasingly negative APDs indicate steeper generalization performance drops, suggesting greater vulnerability to Clever Hans learning. We report the average APD over 20 repetitions and 3 datasets with 95% confidence intervals (± t0.975,59 ⋅ SE), corrected to remove common variance due to dataset (Masson & Loftus93).

Slide prediction model training and evaluation

Datasets and splits

We investigated four clinically relevant and frequently studied slide-level prediction tasks, i.e., slide-level tumor detection in Camelyon16 / Camelyon1751,52,53 and lung cancer subtyping (adenocarcinoma vs. squamous cell carcinoma) from the TCGA-LUAD and TCGA-LUSC studies for comparison with the patch-level prediction results, as well as homologous recombination deficiency (HRD) and overall survival time prediction from breast cancer slides of TCGA-BRCA, representing challenging molecular biomarker prediction and outcome prognosis tasks18,65,96,97,98. For Camelyon, NSCLC, and BRCA HRD, we created training splits with controlled correlations between labels and medical centers to measure the effects of Clever Hans learning, using the medical centers with the highest case numbers (see Supplementary Note 4 and Supplementary Fig. 10 for split statistics). For survival prediction with non-categorical labels, where controlled correlations are difficult to construct, we chose a more practical approach. In the first split, training and validation sets were composed from all slides of the three highest-contributing centers, all of which have similar overall survival time distributions. For the second split, we used all slides of the six highest-contributing centers; note that the three additional centers have visibly different survival time distributions from the first three, introducing a potential confounder (see Fig. 5a). For computational reasons, we restricted the feature extraction to six foundation models (Virchow2, CONCHv1.5, H0-mini, UNI2-h, CTransPath, Phikon-v2) that represent different training paradigms and levels of biological representation robustness.

Model training

We trained Attention-based multiple instance learning (MIL) models67, which is arguably the most established slide-level prediction approach. For training, we sampled bags of 2048 patch representations per slide. For validation and testing, we simultaneously exposed all patches of a slide to the model to avoid sampling biases and statistical instabilities. For the survival prediction task, we adopted a discrete-time survival modeling approach, where the survival time span is divided into four time intervals, and the model outputs a risk score based on interval-specific hazard predictions99,100. To account for variations due to splitting, we repeated all Attention-based MIL model trainings 20 times by resampling the training, validation, and test data. Further training details are provided in Supplementary Note 4.

Clustering analysis

Measuring clustering quality

While the robustness index captures the local neighborhood structure of each representation, the clustering score extends this analysis to a global level. It evaluates whether embedding vectors form distinct clusters, and to what extent they are driven by diagnostically relevant factors or misled by similarities in medical centers. We used unsupervised K-means clustering with cosine distances, selecting the number of clusters K by maximizing the silhouette score101. This setup mimics real-world exploratory analyses where prior cluster information is unknown (e.g., discovering morphological subtypes). The quality of clustering is evaluated by comparing the predicted cluster assignments \(\widehat{{{\mathcal{C}}}}=\{{\widehat{C}}_{1},\ldots,{\widehat{C}}_{K}\}\) to the true biological \({{{\mathcal{C}}}}_{{{\rm{bio}}}}\) and confounding labels \({{{\mathcal{C}}}}_{{{\rm{mc}}}}\) (potentially of different sizes than K) using the adjusted Rand index (ARI)102. The ARI measures the agreement between two clusterings primarily by the number of pairs of samples that are correctly clustered together or apart, accounting for chance. A perfect clustering is solely based on biological information and not influenced by the medical center origin. Hence, we define the clustering score as the difference between these ARIs:

The clustering score ranges within approximately [ −1, 1], where values near zero suggest that the clustering is influenced by both biological and confounding information (or neither), positive values indicate a clustering aligned with medically relevant features, and negative values reflect a medical center-driven structure. Supplementary Note 5 provides a more detailed description of the clustering score, and Supplementary Note 7.6 discusses the clustering results with optimally chosen K, which reveals the effect of the silhouette score-based selection.

Clustering experiments

Clustering experiments are conducted on balanced 2 × 2 configurations (two biological classes and two medical centers). For Camelyon and Tolkach ESCA, the same subsampled 2 × 2 datasets as in the robustness index evaluation are used, while for TCGA, all 2 × 2 pairs derived from the 4 × 4 setting are considered. For each value of K ∈ [2, 30], K-means clustering is performed, and the optimal K is selected based on the silhouette score. The results of the final clustering, with the selected K, are compared with the true biological labels and the medical center origins using the proposed clustering score. An average clustering score and its standard deviation are estimated based on 50 repetitions of the final clustering with different random initializations.

Foundation model robustification without FM retraining

Robustification methods

While output-level robustness improvement for standalone pathology models has been studied (see, e.g., ref. 103), we view representation- and output-level robustness improvement for foundation models as a major current open challenge for the field.

To remove medical center signatures in image space, we applied Reinhard47 and Macenko104 stain normalization to each patch individually before extracting features from the foundation models. As a normalization target, we used the average statistics of 500 sampled patches from the TCGA-LUSC project. As we found that Reinhard stain normalization consistently achieved better robustification results than Macenko stain normalization, we only report the Reinhard results.

In molecular biology, data are represented in numerical vectors that describe cell or bulk expression profiles. Although potentially differently distributed, molecular expression vectors are structurally similar to FM feature vectors, motivating the application of ComBat48, one of the most popular batch correction methods in molecular biology. We used the PyComBat implementation (https://pypi.org/project/inmoose). For experiments with held-out test data, we first applied ComBat to the training data, and then used the corrected training data as a single reference batch to normalize the test sets without leaking information into the training set, as done by Murchan et al.50.

Domain-Adversarial Neural Networks (DANNs)72 aim to learn a modified feature representation in which all domains (e.g., medical centers) are indistinguishable, while simultaneously optimizing for a prediction task. Let \(D={({x}_{i},{y}_{i},{c}_{i})}_{i}\) be the dataset consisting of the feature vectors x, the biological target y, and the medical center c. Further, let \(\phi :x\mapsto x{\prime}\) be a learned projection function on top of the original representation, \({f}_{{\mathsf{CL}}}:\phi (x)\mapsto \widehat{y}\) a classification head, and \({f}_{{\mathsf{DA}}}:\phi (x)\mapsto \widehat{c}\) a domain discriminator on top of ϕ. The DANN objective can be defined as two competing loss parts:

where \(\widehat{y},\widehat{c}\) are the independent logits of the two separate heads, \({f}_{{\mathsf{CL}}}\) and \({f}_{{\mathsf{DA}}}\), on top of the learned feature space ϕ. The first term, \({{{\mathcal{L}}}}_{{\mathsf{CL}}}\), is used as a standard classification loss, whereas \({{{\mathcal{L}}}}_{{\mathsf{DA}}}\) is used to align the domains (both, e.g., using cross-entropy), and λ is a weight to balance the loss terms. Training details and hyper-parameter choices are provided in Supplementary Note 4.

Notice that ComBat and DANN require knowledge of the medical center origin of each patch, whereas Reinhard stain normalization does not. Further, DANN can only be applied when training a downstream model for a specific prediction task. Also note that the site stratification method proposed by Howard et al.31 cannot be usefully applied if training data are imbalanced — balancing features of interest across folds is not possible if all samples of a biological class came exclusively from one medical center.

Experimental evaluation

We re-ran the experiments, training supervised downstream models on top of robustified feature spaces (see Supplementary Note 4 for details). For each foundation model and robustness improvement method, we computed the average performance drops (see above) with respect to the averaged unnormalized baseline performance on the balanced Split 1 (Cramér’s V = 0), i.e.,

for dataset D, model M, and repetition r. This enables a comparison of the robustification methods with the baseline setting: an average performance drop closer to zero than the baseline means that the method reduced the harm of Clever Hans learning. We report the averages per FM over all repetitions and datasets.

Statistical information

Due to the small sample size (n = 20), p-values of the Spearman rank-order correlation ρ were estimated based on a two-sided paired permutation test with 50,000 permutations. We computed 95% confidence intervals for the correlation coefficients using Fisher’s z-transformation105.

Software

For data collection, benchmarking experiments, and results analyses, we used Python 3.10 and common Python libraries (numpy 2.2.6, pandas 2.3.2, pillow 11.3.0, torch 2.8.0, timm 1.0.19, scipy 1.15.3, scikit-learn 1.7.2, matplotlib 3.10.6, seaborn 0.13.2). For reading and processing whole-slide images into image patches for the Camelyon data resources, we used OpenSlide 4.0.0 (https://openslide.org/). For stain normalization, we used an adapted version of torchstain 1.4.1 (https://github.com/EIDOSLAB/torchstain). For ComBat batch correction, we used pyComBat / InMoose 0.7.8 (https://github.com/epigenelabs/inmoose). For slide-level downstream model training and evaluation, we used the xMIL repository (v0.0.1): https://github.com/bifold-pathomics/xMIL. Our custom codes from the PathoROB benchmark are publicly available at: https://github.com/bifold-pathomics/PathoROB106.

Reporting summary

Further information on research design is available in the Nature Portfolio Reporting Summary linked to this article.

Data availability

The PathoROB benchmark data used in this study are available on Hugging Face under: https://huggingface.co/collections/bifold-pathomics/pathorob. All PathoROB data are processed data that originate from public repositories (Camelyon52: https://camelyon17.grand-challenge.org/Data, License: Public Domain: CC0; TCGA UT54: https://zenodo.org/records/5889558, License: Non-Commercial Use: CC-BY-NC-SA 4.0; Tolkach ESCA57: https://zenodo.org/records/7548828, License: Custom, Non-Commercial Use). The original data are publicly accessible via the links provided. The TCGA slide-level experiment data used in this study are available in the Genomic Data Commons (GDC) database under: https://portal.gdc.cancer.gov/. The foundation model weights were downloaded from public repositories and from Hugging Face (https://huggingface.co/), except for the proprietary models RudolfV and Atlas developed by Aignostics, Mayo Clinic and Charité. Source data are provided in this paper.

Code availability

The code for the PathoROB benchmark is publicly available at: https://github.com/bifold-pathomics/PathoROB (License: BSD-3-Clause)106. Slide-level experiments were conducted using the publicly available xMIL repository: https://github.com/bifold-pathomics/xMIL (License: MIT). Stain normalization was computed with code from the publicly available torchstain repository: https://github.com/EIDOSLAB/torchstain (License: MIT). ComBat batch normalization was performed with the publicly available InMoose package: https://pypi.org/project/inmoose (License: GPL-3.0).

References

  • Bommasani, R. et al. On the opportunities and risks of foundation models. Preprint at https://doi.org/10.48550/arXiv.2108.07258 (2021).

  • Azad, B. et al. Foundational models in medical imaging: A comprehensive survey and future vision. Preprint at https://doi.org/10.48550/arXiv.2310.18689 (2023).

  • Khan, W. et al. A comprehensive survey of foundation models in medicine. In Proceedings of the IEEE Reviews in Biomedical Engineering (2025).

  • Bilal, M. et al. Foundation models in computational pathology: A review of challenges, opportunities, and impact. Preprint at https://doi.org/10.48550/arXiv.2502.08333 (2025).

  • Ciga, O., Xu, T. & Martel, A. Self supervised contrastive learning for digital histopathology. Mach. Learn. Appl. 7, 100198 (2022).

  • Campanella, G. et al. A clinical benchmark of public self-supervised pathology foundation models. Nat. Commun. 16, 3640 (2025).

  • Tiu, E. et al. Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning. Nat. Biomed. Eng. 6, 1399–1406 (2022).

  • Zhang, S. et al. A multimodal biomedical foundation model trained from fifteen million image–text pairs. NEJM AI 2, AIoa2400640 (2025).

  • Lee, J. et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36, 1234–1240 (2019).

  • Gu, Y. et al. Domain-specific language model pretraining for biomedical natural language processing. ACM Trans. Comput. Healthc. 3, 1–23 (2021).

  • Theodoris, C. V. et al. Transfer learning enables predictions in network biology. Nature 618, 616–624 (2023).

  • Fu, X. et al. A foundation model of transcription across human cell types. Nature 637, 965–973 (2025).

  • Zimmermann, E. et al. Virchow2: Scaling self-supervised mixed magnification models in pathology. Preprint at https://doi.org/10.48550/arXiv.2408.00738 (2024).

  • Alber, M. et al. Atlas: A novel pathology foundation model by Mayo Clinic, Charité, and Aignostics. Preprint at https://doi.org/10.48550/arXiv.2501.05409 (2025).

  • Chen, R. J. et al. Towards a general-purpose foundation model for computational pathology. Nat. Med. 30, 850–862 (2024).

  • Vorontsov, E. et al. A foundation model for clinical-grade computational pathology and rare cancers detection. Nat. Med. 30, 1–12 (2024).

  • Dippel, J. et al. AI-based anomaly detection for clinical-grade histopathological diagnostics. NEJM AI 1, AIoa2400468 (2024).

  • Schirris, Y., Gavves, E., Nederlof, I., Horlings, H. M. & Teuwen, J. DeepSMILE: Contrastive self-supervised pre-training benefits MSI and HRD classification directly from H&E whole-slide images in colorectal and breast cancer. Med. Image Anal. 79, 102464 (2022).

  • Dippel, J. et al. Rudolfv: A foundation model by pathologists for pathologists. Preprint at https://doi.org/10.48550/arXiv.2401.04079 (2024).

  • Jaume, G. et al. HEST-1k: A dataset for spatial transcriptomics and histology image analysis. Adv. Neural Inf. Process. Syst. 37, 53798–53833 (2024).

  • Campanella, G. et al. Real-world deployment of a fine-tuned pathology foundation model for lung cancer biomarker detection. Nat. Med. 31, 3002–3010 (2025).

  • Mahmood, F. A benchmarking crisis in biomedical machine learning. Nat. Med. 31, 1060–1060 (2025).

  • kaiko.ai, Gatopoulos, I., Känzig, N., Moser, R. & Otálora, S. eva: Evaluation framework for pathology foundation models. in Medical Imaging with Deep Learning (2024).

  • Zhang, A., Jaume, G., Vaidya, A., Ding, T. & Mahmood, F. Accelerating data processing and benchmarking of AI models for pathology. Preprint at https://doi.org/10.48550/arXiv.2502.06750 (2025).

  • Vaidya, A. et al. Molecular-driven foundation model for oncologic pathology. Preprint at https://doi.org/10.48550/arXiv.2501.16652 (2025).

  • Ma, J. et al. Pathbench: A comprehensive comparison benchmark for pathology foundation models towards precision oncology. Preprint at https://doi.org/10.48550/arXiv.2505.20202 (2025).

  • Lee, J., Lim, J., Byeon, K. & Kwak, J. T. Benchmarking pathology foundation models: Adaptation strategies and scenarios. Comput. Biol. Med. 190, 110031 (2025).

  • Mulliqi, N. et al. Foundation models a panacea for artificial intelligence in pathology? Preprint at https://doi.org/10.48550/arXiv.2502.21264 (2025).

  • Bareja, R. et al. Evaluating vision and pathology foundation models for computational pathology: A comprehensive benchmark study. Preprint at https://doi.org/10.1101/2025.05.08.25327250 (2025).

  • Neidlinger, P. et al. Benchmarking foundation models as feature extractors for weakly supervised computational pathology. Nat. Biomed. Eng. https://doi.org/10.1038/s41551-025-01516-3 (2025).

  • Howard, F. M. et al. The impact of site-specific digital histology signatures on deep learning model accuracy and bias. Nat. Commun. 12, 4423 (2021).

  • Klauschen, F. et al. Toward explainable artificial intelligence for precision pathology. Annu. Rev. Pathol. Mech. Dis. 19, 541–570 (2024).

  • Leek, J. T. et al. Tackling the widespread and critical impact of batch effects in high-throughput data. Nat. Rev. Genet. 11, 733–739 (2010).

  • Goh, W. W. B., Wang, W. & Wong, L. Why batch effects matter in omics data, and how to avoid them. Trends Biotechnol. 35, 498–507 (2017).

  • Goh, W. W. B., Yong, C. H. & Wong, L. Are batch effects still relevant in the age of big data?. Trends Biotechnol. 40, 1029–1040 (2022).

  • DeGrave, A. J., Janizek, J. D. & Lee, S.-I. AI for radiographic COVID-19 detection selects shortcuts over signal. Nat. Mach. Intell. 3, 610–619 (2021).

  • Kauffmann, J. et al. Explainable AI reveals Clever Hans effects in unsupervised learning models. Nat. Mach. Intell. 7, 412–422 (2025).

  • Finak, G. et al. Standardizing flow cytometry immunophenotyping analysis from the human immunophenotyping consortium. Sci. Rep. 6, 20686 (2016).

  • Čuklina, J. et al. Diagnostics and correction of batch effects in large-scale proteomic studies: a tutorial. Mol. Syst. Biol. 17, e10240 (2021).

  • Zindler, T., Frieling, H., Neyazi, A., Bleich, S. & Friedel, E. Simulating ComBat: how batch correction can lead to the systematic introduction of false positive results in DNA methylation microarray studies. BMC Bioinform. 21, 1–15 (2020).

  • Kömen, J., Marienwald, H., Dippel, J. & Hense, J. Do histopathological foundation models eliminate batch effects? A comparative study. In Advancements In Medical Foundation Models: Explainability, Robustness, Security, and Beyond (AIM-FM Workshop at NeurIPS, 2024).

  • de Jong, E. D., Marcus, E. & Teuwen, J. Current pathology foundation models are unrobust to medical center differences. Preprint at https://doi.org/10.48550/arXiv.2501.18055 (2025).

  • Filiot, A. et al. Distilling foundation models for robust and efficient models in digital pathology. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention 162–172 (2025).

  • Gustafsson, F. K. & Rantalainen, M. Evaluating computational pathology foundation models for prostate cancer grading under distribution shifts. Preprint at https://doi.org/10.48550/arXiv.2410.06723 (2024).

  • Ji, X. et al. Physical color calibration of digital pathology scanners for robust artificial intelligence–assisted cancer diagnosis. Mod. Pathol. 38, 100715 (2025).

  • Drexlin, D. J. et al. Medi: Metadata-guided diffusion models for mitigating biases in tumor classification. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention 379–388 (2025).

  • Reinhard, E., Adhikhmin, M., Gooch, B. & Shirley, P. Color transfer between images. IEEE Comput.Graph. Appl. 21, 34–41 (2001).

  • Johnson, W. E., Li, C. & Rabinovic, A. Adjusting batch effects in microarray expression data using empirical bayes methods. Biostatistics 8, 118–127 (2007).

  • Behdenna, A. et al. pyComBat, a Python tool for batch effects correction in high-throughput molecular data using empirical Bayes methods. BMC Bioinform. 24, 459 (2023).

  • Murchan, P., Broin, P. Ó, Baird, A.-M., Sheils, O. & Finn, S. P. Deep feature batch correction using ComBat for machine learning applications in computational pathology. J. Pathol. Inform. 15, 100396 (2024).

  • Bejnordi, B. E. et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA 318, 2199–2210 (2017).

  • Litjens, G. et al. 1399 H&E-stained sentinel lymph node sections of breast cancer patients: the CAMELYON dataset. GigaScience 7, giy065 (2018).

  • Bándi, P. et al. From detection of individual metastases to classification of lymph node status at the patient level: The CAMELYON17 challenge. IEEE Trans. Med. Imaging 38, 550–560 (2019).

  • Komura, D. & Ishikawa, S. Histology images from uniform tumor regions in TCGA whole slide images (TCGA-UT)https://doi.org/10.5281/zenodo.5889558 (2021).

  • Komura, D. et al. Universal encoding of pan-cancer histology by deep texture representations. Cell Rep. 38, 110424 (2022).

  • Tolkach, Y. et al. Artificial intelligence for tumour tissue detection and histological regression grading in oesophageal adenocarcinomas: a retrospective algorithm development and validation study. Lancet Digit. Health 5, e265–e275 (2023).

  • Tolkach, Y. Datasets for publication “Artificial intelligence for tumor detection and histological regression grading in oesophageal adenocarcinomas: a retrospective algorithm development and validation study”. Zenodohttps://doi.org/10.5281/zenodo.7548828 (2023).

  • Lapuschkin, S. et al. Unmasking Clever Hans predictors and assessing what machines really learn. Nat. Commun. 10, 1096 (2019).

  • Geirhos, R. et al. Shortcut learning in deep neural networks. Nat. Mach. Intell. 2, 665–673 (2020).

  • Hermann, K., Mobahi, H., Fel, T. & Mozer, M. C. On the foundations of shortcut learning. In Proceedings of the International Conference on Learning Representations (ICLR, 2024).

  • Vaidya, A. et al. Demographic bias in misdiagnosis by computational pathology models. Nat. Med. 30, 1174–1190 (2024).

  • Brown, A. et al. Detecting shortcut learning for fair medical AI using shortcut testing. Nat. Commun. 14, 4314 (2023).

  • De Angeli, K. et al. Class imbalance in out-of-distribution datasets: Improving the robustness of the textcnn for the classification of rare cancer types. J. Biomed. Inform. 125, 103957 (2022).

  • Guerrero, R. E. D., Carvalho, L., Bocklitz, T., Popp, J. & Oliveira, J. L. A data augmentation methodology to reduce the class imbalance in histopathology images. J. Imaging Inform. Med. 37, 1767–1782 (2024).

  • Song, A. H. et al. Multimodal prototyping for cancer survival prediction. In Proceedings of the International Conference on Machine Learning (ICML, 2024).

  • Wang, X. et al. A pathology foundation model for cancer diagnosis and prognosis prediction. Nature 634, 970–978 (2024).

  • Ilse, M., Tomczak, J. & Welling, M. Attention-based deep multiple instance learning (2018). In Proceedings of the International Conference on Machine Learning (ICML).

  • Tizhoosh, H. & Pantanowitz, L. On image search in histopathology. J. Pathol. Inform. 15, 100375 (2024).

  • Alfasly, S. et al. Validation of histopathology foundation models through whole slide image retrieval. Sci. Rep. 15, 3990 (2025).

  • Zhang, Y. et al. Histopathology images-based deep learning prediction of prognosis and therapeutic response in small cell lung cancer. Npj Digit. Med. 7, 15 (2024).

  • Chelebian, E., Avenel, C., Ciompi, F. & Wählby, C. Depicter: Deep representation clustering for histology annotation. Comput. Biol. Med. 170, 108026 (2024).

  • Ganin, Y. et al. Domain-adversarial training of neural networks. J. Mach. Learn. Res. 17, 59:1–59:35 (2016).

  • Radford, A. et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML, 2021).

  • Jia, C. et al. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the International Conference on Machine Learning (ICML, 2021).

  • Achiam, J. et al. Gpt-4 technical report. Preprint at https://doi.org/10.48550/arXiv.2303.08774 (2023).

  • Zhou, C. et al. A comprehensive survey on pretrained foundation models: A history from bert to chatgpt. Int. J. Mach. Learn. Cybern. 16, 9851–9915 (2024).

  • Singhal, K. et al. Large language models encode clinical knowledge. Nature 620, 172–180 (2023).

  • Kabylda, A. et al. Molecular simulations with a pretrained neural network and universal pairwise force fields. J. Am. Chem. Soc. 147, 33723–33734 (2025).

  • Kovács, D. P., Batatia, I., Arany, E. S. & Csányi, G. Evaluation of the mace force field architecture: From medicinal chemistry to materials science. J. Chem. Phys. 159, 044118 (2023).

  • Awais, M. et al. Foundation models defining a new era in vision: A survey and outlook. IEEE Trans. Pattern Anal. Mach. Intell. 47, 2245–2264 (2025).

  • Shaban, M. et al. A foundation model for spatial proteomics. Preprint at https://doi.org/10.48550/arXiv.2506.03373 (2025).

  • Muttenthaler, L. et al. Aligning machine and human visual representations across abstraction levels. Nature 647, 349–355 (2025).

  • Weng, W.-H. et al. An intentional approach to managing bias in general purpose embedding models. Lancet Digit. Health 6, e126–e130 (2024).

  • Chormai, P., Herrmann, J., Müller, K.-R. & Montavon, G. Disentangled explanations of neural network predictions by finding relevant subspaces. IEEE Trans. Pattern Anal. Mach. Intell. 46, 7283–7299 (2024).

  • Huben, R., Cunningham, H., Smith, L. R., Ewart, A. & Sharkey, L. Sparse autoencoders find highly interpretable features in language models. In Proceedings of the International Conference on Learning Representations(ICLR, 2024).

  • Gindra, R. H. et al. A large-scale benchmark of cross-modal learning for histology and gene expression in spatial transcriptomics. In Proceedings of the IEEE/CVF International Conference on Computer Vision 1182–1192 (2025).

  • Campanella, G. et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat. Med. 25, 1301–1309 (2019).

  • Wagner, S. J. et al. Transformer-based biomarker prediction from colorectal cancer histology: A large-scale multicentric study. Cancer Cell 41, 1650–1661 (2023).

  • Xu, H. et al. A whole-slide foundation model for digital pathology from real-world data. Nature 630, 181–188 (2024).

  • Ding, T. et al. A multimodal whole-slide foundation model for pathology. Nat. Med. 31, 3749–3761 (2025).

  • Ouyang, L. et al. Training language models to follow instructions with human feedback. Adv. Neural Inf. Process. Syst. 35, 27730–27744 (2022).

  • Bai, Y. et al. Constitutional AI: Harmlessness from AI feedback. Preprint at https://doi.org/10.48550/arXiv.2212.08073 (2022).

  • Masson, M. E. J. & Loftus, G. R. Using confidence intervals for graphically based data interpretation. Can. J. Exp. Psychol. 57, 203–220 (2003).

  • Fix, E. & Hodges, J. Discriminatory Analysis: Nonparametric Discrimination: Consistency Properties (USAF School of Aviation Medicine, 1951).

  • Cover, T. & Hart, P. Nearest neighbor pattern classification. IEEE Trans. Inf. Theory 13, 21–27 (1967).

  • Loeffler, C. M. L. et al. Prediction of homologous recombination deficiency from routine histology with attention-based multiple instance learning in nine different tumor types. BMC Biol. 22, 225 (2024).

  • El Nahhas, O. S. et al. Regression-based deep-learning predicts molecular biomarkers from pathology slides. Nat. Commun. 15, 1253 (2024).

  • Chen, R. J. et al. Pan-cancer integrative histology-genomic analysis via multimodal deep learning. Cancer Cell 40, 865–878 (2022).

  • Zadeh, S. G. & Schmid, M. Bias in cross-entropy-based training of deep survival networks. IEEE Trans. pattern Anal. Mach. Intell. 43, 3126–3137 (2020).

  • Chen, R. J. et al. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR, 2021).

  • Rousseeuw, P. J. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20, 53–65 (1987).

  • Hubert, L. & Arabie, P. Comparing partitions. J. Classif. 2, 193–218 (1985).

  • Jahanifar, M. et al. Domain generalization in computational pathology: survey and guidelines. ACM Comput. Surv. 57, 1–37 (2025).

  • Macenko, M. et al. A method for normalizing histology slides for quantitative analysis. In Proceedings of the IEEE International Symposium on Biomedical Imaging: From Nano to Macro (ISBI).

  • Lane, D. M. Confidence interval on pearson’s correlation.https://onlinestatbook.com/2/estimation/correlation_ci.html (2018).

  • Kömen, J. et al. Towards robust foundation models for digital pathology. GitHub repository: PathoROB https://doi.org/10.5281/zenodo.18961604 (2026).

  • Chen, R. J. et al. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR, 2022).

  • Wang, X. et al. Retccl: Clustering-guided contrastive learning for whole-slide image retrieval. Med. Image Anal. 83, 102645 (2023).

  • Wang, X. et al. Transformer-based unsupervised contrastive learning for histopathological image classification. Med. Image Anal. 81, 102559 (2022).

  • Kang, M., Song, H., Park, S., Yoo, D. & Pereira, S. Benchmarking self-supervised learning on diverse pathology datasets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR, 2023).

  • Aben, N. et al. Towards large-scale training of pathology foundation models. Preprint at https://doi.org/10.48550/arXiv.2404.15217 (2024).

  • Filiot, A. et al. Scaling self-supervised learning for histopathology with masked image modeling. Preprint at https://doi.org/10.1101/2023.07.21.23292757 (2023).

  • Filiot, A., Jacob, P., Mac Kain, A. & Saillard, C. Phikon-v2, a large and public feature extractor for biomarker prediction. Preprint at https://doi.org/10.48550/arXiv.2409.09173 (2024).

  • Lu, M. Y. et al. A visual-language foundation model for computational pathology. Nat. Med. 30, 863–874 (2024).

  • Xiang, J. et al. A vision–language foundation model for precision oncology. Nature 638, 769–778 (2025).

Acknowledgements

We thank Florian C.F. Schulz and Augustin Krause for support with the foundation model representation extraction library. We thank our colleagues Laure Ciernik, Elias Eulig, Alexander Möllers, Hans Pinckaers and Stephan Tietz for their valuable feedback on earlier versions of this manuscript, which helped improve this work. We would like to express our gratitude to Yuri Tolkach for sharing additional case metadata for the Tolkach ESCA dataset. The results shown here are in part based upon data generated by the TCGA Research Network: https://www.cancer.gov/tcga.

Funding

This work was partly funded by the German Ministry for Education and Research (under refs 01IS14013A-E, 01GQ1115, 01GQ0850, 01IS18056A, 01IS18025A, 13GW0744D, and BIFOLD25B). Furthermore, K.-R.M. was partly supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2019-II190079, Artificial Intelligence Graduate School Program, Korea University) and grant funded by the Korea government (MSIT) (No. RS-2024-00457882, AI Research Hub Project).

Author information

Authors and Affiliations

Contributions

Conceptualization and methodology: J.K., E.D.J., J.H., H.M., J.D., P.N., and K.-R.M. Development of the robustness index: E.D.J. Development of the downstream experiments: J.K., J.H., H.M., and J.D. Data curation, code creation, and experiments: J.K., E.D.J., J.H., H.M., J.D., and P.N. Analysis of results: J.K., E.D.J., J.H., H.M., J.D., and P.N. Project administration: J.H. Supervision: E.M., L.R., M.A., J.T., F.K., and K.-R.M. Writing: J.K., E.D.J., J.H., H.M., J.D., P.N., E.M., L.R., M.A., F.K., and K.-R.M.

Corresponding authors

Ethics declarations

Competing interests

E.D.J., J.D., L.R., and M.A. are employees of Aignostics. M.A., F.K., and K.-R.M. are co-founders of Aignostics. F.K. is lead medical advisor and board member of Aignostics. K.R.M. is lead technical advisor to Aignostics. E.D.J. was previously employed at kaiko.ai. Aignostics and kaiko.ai developed foundation models that were assessed in this study (Atlas, RudolfV, Kaiko ViT-B/8). J.T. is co-founder of Glass Medical AI. The remaining authors declare no competing interests.

Peer review

Peer review information

Nature Communications thanks Guillaume Jaume, Issam El Naqa and the other anonymous reviewer(s) for their contribution to the peer review of this work. A peer review file is available.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Source data

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.

About this article

Cite this article

Kömen, J., de Jong, E.D., Hense, J. et al. Towards robust foundation models for digital pathology. Nat Commun 17, 5218 (2026). https://doi.org/10.1038/s41467-026-73923-2

  • DOI: https://doi.org/10.1038/s41467-026-73923-2