Bigpicture: Exploring the potential of AI foundation models

In this article, Francesco Ciompi (WP4 Lead) and Geert Litjens (Former WP4 Lead) discuss the project's current contributions, and share their visions for how Bigpicture could enable the development of foundation models and multimodal AI in the future.

Geert Litjens, Professor of AI for medical image analysis at Radboud University Medical Center and member of Bigpicture’s workpackage 4, highlights the importance of the repository: “Bigpicture is laying the foundation for significant advancements in pathology by enabling the development of generic AI methods applicable to various tasks. Our focus has been on creating tools like quality control algorithms and content-based image retrieval, which enhance the usability of the dataset.”

What are foundation models?

Foundation models are large AI systems designed to tackle multiple tasks by learning from vast amounts of data, often spanning various types such as images, text, and numerical information. In the context of pathology, such models have the potential to perform tasks like detecting patterns in images, predicting clinical outcomes, and identifying biomarkers. However, developing these models requires significant computational resources and large-scale datasets—a vision that aligns with Bigpicture’s long-term ambitions but extends beyond its immediate deliverables.

Bigpicture’s current contributions lie in facilitating content-based image retrieval and developing AI-assisted tools for tasks such as artifact detection and quality control. These foundational tools aim to streamline workflows and maximize the utility of its growing dataset, which will eventually include approximately 3 million images.

Overcoming obstacles in data availability

The scale and diversity of Bigpicture’s repository make it uniquely valuable for advancing pathology research. However, building such a repository has not been without challenges. For instance, the process of finalizing data sharing agreements (DSAs) delayed the availability of data for large-scale experimentation. Geert reinforces this point: “The availability of high-quality, diverse data is crucial for AI advancements. With Bigpicture, we are not just creating a repository but enabling researchers to ask complex questions and develop innovative AI methods that could shape the future of pathology.”

A new task force to explore foundation models

Francesco Ciompi, WP4 lead, and Professor of AI at Radboud University Medical Center, adds: “While Bigpicture has made significant progress in overcoming these legal and logistical hurdles, the actual building and training of foundation models would require additional resources and external funding.”

This is where the newly formed task force comes into play. Set to begin work in early 2025, the group will explore opportunities to build AI models using data available through Bigpicture. Initial activities will focus on drafting a memorandum of understanding among interested Bigpicture partners, outlining the structure and contribution of each partner in the task force, and ensuring alignment with Bigpicture’s collaborative ethos.

A vision for multimodal AI

A key aspect of Bigpicture’s vision is its contribution to multimodal AI. While the current consortium focuses on pathology images, the broader goal is to create tools that integrate additional data types, such as patient records and genetic information. This approach could open the door to more comprehensive analyses and new discoveries in disease diagnosis and treatment planning.

Francesco elaborates: “Although Bigpicture’s immediate focus is on creating a repository of vision-based data, this could become a cornerstone for future multimodal models that combine pathology with other data modalities. Such models would have the potential to power clinical predictions and help standardize pathology research across Europe.”

Preparing for the Future

The task force’s efforts to explore foundation models reflect Bigpicture’s commitment to advancing pathology research while maintaining a realistic scope for its current activities. “We are not overpromising what Bigpicture itself can deliver,” Francesco emphasizes. “Instead, we are positioning Bigpicture as a key enabler of future innovations, creating the infrastructure and collaborations needed to realize these possibilities.”