Prism · azureml v2 ↗
- source model id
- azureml/Prism@2
- hosting
- foundry-managed-compute
- platforms
- Microsoft Foundry
- license
- custom
- architecture
- input_modalities
- output_modalities
- capabilities
PRISM is a multi-modal generative foundation model for slide-level analysis of H&E-stained histopathology images. Utilizing Virchow tile embeddings and clinical report texts for pre-training, PRISM c…
azureFoundry/azureml/Prism@2
PRISM is a multi-modal generative foundation model for slide-level analysis of H&E-stained histopathology images. Utilizing Virchow tile embeddings and clinical report texts for pre-training, PRISM combines these embeddings into a single slide embedding and generates a text-based diagnostic report. These can be used for tasks such as cancer detection, sub-typing, and biomarker identification. The model's slide encoder can be fine-tuned for specific classification tasks, leveraging both image and text data to enhance diagnostic performance and robustness. This model is available solely for non-commercial research and evaluation purposes. It is not designed for clinical use and should not be employed to diagnose any diseases. The reports generated by PRISM are intended for assessing the model's quality and may contain errors. Therefore, using the generated reports in clinical settings is strictly prohibited. The generated reports do not reflect the model's true performance for zero-shot or finetuning benchmark diagnostic tasks.
PRISM supports several modes of use:
• text report generation to describe tissue in H&E whole slide images • zero-shot cancer detection and sub-typing using text prompts • adaptation to new tasks via PRISM finetuning, or linear classifier on the slide embedding
Developed by: Paige.AI, Inc., New York, NY, USA and Microsoft Research, Cambridge, MA, USA Model Type: Vision-Language Encoder-Decoder Model Stats: Params (M): 558 Architecture: Encoder: Perceiver (https://doi.org/10.48550/arXiv.2103.03206) Decoder: BioGPT (https://huggingface.co/microsoft/biogpt) Model inputs: tile image embeddings and text captions Tile image encoder: Virchow V1 (https://huggingface.co/paige-ai/Virchow) Training Details: Objective: CoCa (https://doi.org/10.48550/arXiv.2205.01917) Precision: Mixed precision (fp16) Paper: PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathology: https://arxiv.org/abs/2405.10254 Pretraining Dataset: Internal dataset of 587 thousand whole slide images and 195 thousand clinical reports from Memorial Sloan Kettering Cancer Center. License: CC-BY-NC-ND-4.0
Direct use PRISM is a vision-language model that can analyze whole slide images using the following methods:
• CLIP-style zero-shot classification via zero_shot method, or • generate a tissue description in the image via generate method.
The model takes whole slide images in the form of tile embeddings from our Virchow model. Please see https://huggingface.co/paige-ai/Virchow for instructions on how to use it to generate embeddings for your whole slide image.
Downstream use You can use PRISM to compute slide embedding for downstream tasks such as slide-level classification. The slide embedding can be further adapted to new tasks by finetuning the slide encoder of PRISM on slide-level labels, e.g. biomarkers.
Slide embeddings are accessible via slide_representations method.
Terms of use
Terms of use The Prism Model and associated code are released under the CC-BY-NC-ND 4.0 license and may only be used for non-commercial, academic research purposes with proper attribution. Any commercial use, sale, or other monetization of the Prism Model and its derivatives, which include models trained on outputs from the Prism Model or datasets created from the Prism Model, is prohibited and requires prior approval. Please note that the primary email used must match your institutional email to receive approval. By downloading/deploying the Prism Model, you attest that all information (affiliation, research use) is correct and up-to-date. By downloading/deploying the Prism model, you agree not to distribute, publish or reproduce a copy of the Prism Model. If another user within your organization wishes to use the Prism Model, they must register as an individual user and agree to comply with the terms of use. If you are a commercial entity, please contact the corresponding author.
Further, by downloading/deploying the PRISM Model, you agree you will only use the PRISM Model for academic research purposes and will not use, or allow others to use, the PRISM Model to:
Further, you agree that you will appropriately disclose to end users any known dangers of your AI system.
Citation
Please cite the following work if you use the PRISM Model in your research. Shaikovski, George, Adam Casson, Kristen Severson, Eric Zimmermann et al. "PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathology." arXiv preprint arXiv:2405.10254 (2024). https://doi.org/10.48550/arXiv.2405.10254
@article{shaikovski2024prism,
title={PRISM: A Multi-Modal Generative Foundation Model for Slide-Level Histopathology},
author={Shaikovski, George and Casson, Adam and Severson, Kristen and Zimmermann, Eric and Wang, Yi Kan and Kunz, Jeremy D and Retamero, Juan A and Oakley, Gerard and Klimstra, David and Kanan, Christopher and others},
journal={arXiv preprint arXiv:2405.10254},
year={2024}
}
Disclaimer
PRISM has been developed for research purposes and is not intended for diagnosis of real patients or projection/prediction of future disease possibilities. Fairness evaluation cannot be completed due to limitations in the metadata. Underlying biases of the training datasets may not be well characterized and may not be representative of all demographics.
Acknowledgements The results shown here (specifically, in the section "Sample inference code") are in whole or part based upon data generated by the TCGA Research Network: http://cancergenome.nih.gov/.
Example input for zero-shot image classification task:
{
"input_data": {
"columns": [
"embeddings",
"neg_prompts",
"pos_prompts"
],
"index":[0],
"data": [["url", "lobular carcinoma, invasive", "ductal carcinoma, invasive"]]
}
}
Note: 'url' will be a publically accessible url linking to a file containing Virchow embeddings.
[{"output": [0.99, 0.01]}]
Example input for tile description generation task:
{
"input_data": {
"columns": [
"embeddings",
],
"index":[0],
"data": [["url"]]
}
}
[{"output": "</s>Diagnosis: Moderately differentiated invasive ductal carcinoma with micropapillary features in breast tissue. </s>"}]
1–1 / 1
1–3 / 3
Microsoft Foundry · Model page ↗
1–1 / 1
These charts contain model-linked results. The charts above also include unlinked source profiles. Compare scores within each evaluation. Model quality, usage, transparency and system performance measure different things.