Algorithm Submissions - PANTHER Challenge

Training data and baseline code are now available ! Open development phase opens soon !

Submission of Inference Containers to the Open Development and Testing Phase 👩‍💻🗳️¶

Grand Challenge Algorith Template¶

Algorithm submissions for both the Open Development Phase and Closed Testing Phase of the PANTHER challenge must be provided as inference containers. These Docker containers package your trained AI algorithms—including both the architecture and model weights—along with all necessary components to load an MRI case, generate segmentation predictions, and save the output file for evaluation.

Example code for these submissions is available on the PANTHER's GitHub repository. This repository features example implementations and includes detailed, step-by-step instructions for integrating and packaging your algorithm into a Docker container.

We recommend that you initially upload a GC algorithm using the unaltered template. This preliminary submission is crucial, as it allows you to troubleshoot any issues, which may require multiple iterations to resolve. In the following sections, you will find a comprehensive walkthrough to guide you through this process.

Installing Docker on Local System🐳¶

You must have Docker installed and running on your system for the following steps to work. If you are using Windows, we recommend installing Windows Subsystem for Linux (WSL). For more details, please watch the official tutorial by Microsoft for installing WSL 2 with GPU support.

Building, Testing and Exporting Algorithm Container on Local System 👷¶

Please use the provided shell scripts (.sh files) for all subsequent commands. If you are using a Windows system, we strongly recommend installing the Windows Subsystem for Linux (WSL) with an Ubuntu distribution. We also recommend using Docker Desktop to manage your Docker containers and images. Note that to build the baseline algorithm image using Ubuntu in WSL, you must first enable integration with additional distros in Docker Desktop for Windows (navigate to Settings → Resources → WSL Integration). Alternatively, you may install the Docker Engine directly on Ubuntu by following the provided installation steps.

Clone the template repository (as referenced above) to your local system using Git with LFS initialized (this can be done via GitHub Desktop). Build and test your inference Docker container by executing do_test_run.sh, which will build your container and run it on images located in the ./test/ folder. Once testing is complete, export your container using do_save.sh. Your final container image will be generated as a single .tar.gz file.

For more details, please refer to this comprehensive tutorial on creating GC algorithms: Create your own algorithm.

Uploading algorithm container to Grand Challenge ⬆️¶

Before uploading your container, we invite you to create a Team so that collaborators can join your project. Once your team is set up, navigate to Submit > Open Development Phase of either Task 1 or Task 2. Please note that if you wish to participate in both tasks, you must create and submit a separate algorithm container for each one. Complete the form on “Manage your algorithms” for each task accordingly.

Once complete, click the "Save" button. At this point, your GC algorithm has been created and you are on its homepage. Now, click the "Containers" tab on the left panel, and then the "Upload a Container" button. Select your container image (.tar.gz file) for upload. After the upload is complete, click "Save". It typically takes 20-60 minutes till your container has been activated (depending on the size of your container). After its status is "Active", test out your container with, unprocessed cases from the Open Development Phase.

Create Containers for your AI algorithm 📦¶

Following these steps, you can encapsulate your AI algorithm in an inference container by modifying one of our baseline algorithm templates. Begin by editing the functions in ./inference.py to implement your solution. Any additional packages should be added to ./requirements.txt, and any extra files or folders must be explicitly copied using commands in the ./Dockerfile. When it’s time to update your algorithm, simply test and export your new Docker container, then update the container image for the GC algorithm on grand-challenge.org with the new version. Please note that your container will not have internet access during execution on grand-challenge.org, so all necessary model weights and resources must be prepackaged within your container image.

There are two methods for including model weights in your setup. One option is to incorporate the model weights directly into your Docker container by placing them in a ./resources directory. Alternatively, you can upload the model weights as a separate tarball via the Grand Challenge interface (navigate to Algorithm > Models), where the resources in the tarball will be extracted to /opt/ml/model at runtime. This second option is particularly advantageous if you discover an error in your code and need to rebuild your container. By keeping the model weights separate, you can rebuild the container more quickly without having to package the potentially large weights each time.

You can verify that your container is fully self-contained by testing it locally using the --network=none option with docker run. If you encounter any issues during this process, please feel free to post in the forum for assistance.

Submission to Open Development Phase ♻️¶

Once your trained AI model is uploaded as a fully functional GC algorithm, you’re ready to submit it for the Development Phase of the challenge. Head over to the Open Development Phase Submissions page, select the task you want to participate in, choose your algorithm, and click “Save” to submit. Please note that each team is allowed up to 10 submissions during this phase. Provided that no errors occur and the evaluation completes successfully, your performance will appear on the leaderboard—typically within 24 hours.

⚠️ Please double-check all rules to make sure that your submission is compliant. Invalid submissions will be removed and teams repeatedly violating any/multiple rules will be disqualified. Also, please try-out your uploaded GC algorithm with a sample case before making a full submission to the leaderboard.

Submission to Final Test Phase-Testing Leaderboard 🔚¶

Once you have developed and thoroughly tested your model(s), you are invited to submit your best-performing model for the Final Test Phase, following the same procedure outlined previously. Please note the following important details for this phase:

One Submission per Team: Only one submission is permitted per team. Multiple submissions from the same team will result in disqualification.
Open Development Phase Remains Active: The Open Development Phase will remain open during this period, providing you with additional time to test your model prior to your final submission.
Detailed Method Explanation Required: Along with your submission, please complete this form to provide a clear and comprehensive explanation of your methodology. Your explanation should be detailed enough to allow others to reproduce your solution. If you have an arXiv preprint that explains your method, you can include its link in the form. Additionally, if you have a GitHub repository containing your method details, please share that link as well.

Results for all error-free submissions that comply with the participation rules will first be computed in a hidden setting. Once processed, the metrics results will be publicly displayed on the leaderboard and final ranking will be announced.

We eagerly await your submissions to the PANTHER Challenge. Best of luck! 🍀