The software behind Bigpicture’s AI ambition

Being a pioneering project, it is only logical that Bigpicture, especially in its first years, is mainly about finding solutions, and on top of that.. finding more challenges that need solutions. Raphaël: “We are doing many things for the first time. Although the existing software is successfully being integrated at institutes throughout Europe, it has never before been used on such a large scale.”

The integration of Cytomine in Bigpicture brings two main challenges that need to be considered. First of all, it needs further development to accommodate the millions of slides that are expected to be uploaded. Secondly, Bigpicture has different end-users with different requirements. Therefore, the user interface needs modifications, and there’s a need for more complex algorithms. “Improving the visualisation speed in the core architecture of Cytomine is not quite there yet. There are many technical aspects to consider in order to improve the existing database to the scale of Bigpicture’s huge datasets. In parallel, we also continue to develop more interactive algorithms fitting the needs of Bigpicture’s end-users.”

The birth of Cytomine

With a background in Computer Science and a PhD in Machine Learning, Raphaël’s first experience working with medical researchers was when he started at GIGA Interdisciplinary Biomedical Research Center, University of Liège, in Belgium. “Collaborating with biomedical researchers and pathologists helped me understand what tools were missing for them. It was the first time we discussed digital pathology. I started working on a project with the aim to deliver an algorithm to recognize abnormal cells, but then realized that more was missing, e.g. to ease the visualisation and collaborative analysis of large images that exists in digital pathology.”

As a solution, Raphaël initiated the development of Cytomine in 2010. A web-based, open-source software platform for the uploading, annotation, sharing and analyzing of pathology imaging data. One of the most important features is that Cytomine allows people to collaborate over the web, meaning that for instance teachers and students can work on their own algorithms and images remotely. At the start, Cytomine was mainly used within Belgian universities linked to histology and pathology. But today, thanks to the work and services of the Cytomine organization, the open-source software is used by institutes and universities in France, Norway, Sweden, etc. Its amount of users is growing rapidly. “When I started developing this platform in 2010, I wanted it to be freely available for as many people as possible. To see it’s being used by many institutions throughout Europe is a great achievement. And I will be very proud when we have Cytomine up and running for Bigpicture and have all the partners connected to the platform.”

What makes Cytomine unique

Most of the existing pathology software is desktop-based, meaning that each user installs the software on his/her computer. Sharing the work and communicating through the software is a challenge. Cytomine is web-based, meaning that you can access it from anywhere, making remote collaboration much easier and more efficient. Cytomine does not only support the visualisation of the images, but also the annotation is an important feature in the open-source software. “One user can annotate an image and another user can see it immediately. It is a collaborative way of analyzing by sharing images and annotations. Additionally, we can take advantage of these annotations as they will help the development of machine learning and deep learning models, and with that mimic the expertise of pathologists.”

As the software is open-source it’s also constantly evolving. “Open-source means that you have access to everything. The source code is freely available and computer scientists can improve the code. If a company or institution would miss features, they can take the code and make modifications. This is what makes Cytomine is so interesting for Bigpicture.”

The collaboration between Bigpicture and Cytomine

When the call of IMI came to submit the project, Raphaël met with Jeroen van der Laak and Geert Litjens and decided to collaborate and merge their projects to make one common proposal. Raphaël: “As we already did a lot of the software development work for Cytomine, it seemed logical that I would be involved in work package 4 (WP4). It’s better to develop from what we already have, than to start from scratch.”

Bigpicture integration: upscaling and variability

The software, although used broadly, was not yet ready to deal with the millions of images that Bigpicture would include, nor the complex algorithms that would need to process these amounts of images and annotations. “It’s going to be the first time that our software will handle these amounts of data.” And therein lies one of the challenges for Raphael and the team: scaling up.

Up to now, similar projects such as Bigpicture only dealt with small datasets collected by one specific collaborator or one hospital. With the Bigpicture platform being developed, many new roads must be taken, resulting in new challenges and solutions along the way. For instance, data will be collected from multiple sides. Hospital a will not use the exact same protocols and annotation settings as hospital b. “We will get a large variety in images and it’s still challenging to deal with that. We are not yet sure if the existing machine learning models are robust enough to handle this. Thus, the challenge is not just the upscaling to host the huge amounts of data, but it is also about variability.”

Bigpicture integration: user interface

Besides dealing with large amounts of data, there is another challenge to consider; the user interface. Cytomine is currently used for research and education user cases. For this purpose the interface is already user-friendly. But Bigpicture’s end-users are (toxicological) pathologists and AI researchers in both public and private institutions, each with their own needs. “Workflows and user cases are very different for these target groups. The current AI algorithms need more development to make it user-friendly and interactive for Bigpicture’s end-users. And we need to extend our user interfaces to meet the needs of these new workflows and integrate new kinds of algorithms.”

Collaboration

As mentioned before, the scalability and user interface are the main challenges for the integration of Cytomine, and only through collaboration is it possible to successfully overcome those. “We are only a small team ofAI developers, but with the 45 partners connected to the Bigpicture project and their AI teams, each with their own algorithm development, there’s also a lot of opportunity.”

The complexity and scale of the Bigpicture project, obviously requires good collaboration between all stakeholders. “A project of this scale is completely new. No one can do that without collaboration between the partners, software development teams, and other work packages and task forces. We are all connected and depending on each other. For instance, WP2 develops the repository and will provide tools to deal with the data. WP3 collects the data, so naturally they are involved in how to describe the WSI (metadata). And we in WP4 need that input for the algorithms. It is so important that we all speak the same language.”

Learning to speak the same language

When communicating with so many different stakeholders, as is the case within the Bigpicture project, there can be many differences in objectives, requirements and terminology. This can vary throughout countries, disciplines, and even within one discipline in different institutions. After 1.5 year of Bigpicture, Raphaël thinks that the teams are now at a point where they really understand each other and speak the same language. “I think this was one of the first things we really needed to get sorted. To get on the same page. For instance, when we develop AI algorithms, we as AI researchers would have the objective to make it as accurate as possible. We would evaluate how well the algorithm is doing and if we see opportunity to improve with 1% more accuracy, we would do it. However, a pathologist might not see the 1% accuracy improvement as a priority. He or she would mainly look at the amount of extra work it would take to correct the algorithm. In this example, we don’t have the same objective. In computer science we would ‘only’ look at accuracy, but not directly at the time spent on a slide. We now know that we need to provide tools that reduce time, not only getting that 1% improvement on accuracy.”

Excited for what’s to come

Being a pioneer and doing something for the first time comes with a fair amount of challenges, but is above all very exciting and brings maybe even more opportunities. “Up to now we worked on algorithms based on relatively small amounts of data. Bigpicture’s huge repository will definitely open new doors for all of us. We have the chance to work on new, complex algorithms, that can eventually support and improve a pathologists’ work.