Continuous Registration Banner

What is the Continuous Registration Challenge?

Continuous Registration Challenge (CRC) is a challenge for registration of lung- and brain images inspired by modern software development practices. Participants implement their algorithm using the open source SuperElastix C++ API (documentation) which comes with a continuous integration-, experiments-, and evaluation system out of the box. All submissions are automatically compiled, tested, and benchmarked weekly on eight different data sets. Participants are listed on the owner's page, and performances are listed on the leaderboard page. If you use SuperElastix for other work you should cite the owner of the algorithm, not SuperElastix.

The challenge focuses on pairwise registration of lungs and brains, two problems frequently encountered in clinical settings. We have collected seven open-access data sets and one private data set (3+1 lung data sets, 4 brain data sets). For this challenge, a submission consists of code and only code. Benchmarking is done in a fully automated manner and will continue to do so even after the challenge ends. The goal is to provide an open source, collaborative platform where you and your peers develop, compare, and continuously improve algorithms and parameter settings.

Important dates

  • July 1 2018: The Continuous Registration Challenge is now open for contributions. There is no deadline. The challenge runs indefinitely.

FAQ

  • How do I participate? Implement your registration method in SuperElastix and commit your code via a pull request (PR) to the SuperElastix repository on Github (see an example implementation here). Then make a parameter file that uses your component and commit it in the challenge folder in the SuperElastix repository. See a more detailed description in the documentation. Read more about SuperElastix here. Read the full developer documentation here. Read the SuperElastix paper here. 
  • Can I submit a dataset? Yes you can! Just add a Python dataset class to this file and send us a PR on Github along with the data so we can add it to the leaderboard.
  • How do I get attribution for my submission? You will be invited to co-author a journal paper describing the results of the challenge and you will be listed as an owner of your component here. Users are asked to cite your paper everytime a registration is run that uses some of your code.
  • Do I have to write an entire registration algorithm to participate? No! You can also tune an existing registration method in SuperElastix and just submit a parameter file!  
  • Will you help me implement my method in SuperElastix? Yes, we will! We have a slack channel and a mailing list that you can join.
  • Will other participants be able to see my code in the GitHub repository? Yes, everyone can see each others work, learn from each other, and improve from each others experience. That is open source.
  • Why should I participate? Your method will accessible to users and researchers via the SuperElastix library. You will be cited every time a user uses your method for academic work (SuperElastix prints your citing information everytime your submissions is used). You will get a build system, continuous integration (CI) testing system, and have your method automatically tested on many different data sets and continuously benchmarked against other registration algorithms during and after the challenge. Participating in this challenge is much easier that doing experiments yourself!
  • What is the expected outcome? The primary outcome is a high-quality, open source, software library that makes many different registration algorithms easily accessible to experts in other domains that just uses image registration as a tool. Specifically, we have launched this challenge to develop
    • An open source, collaborative platform where researchers develop, compare, and continuously improve image registration algorithms.
    • A high-quality codebase, command line interface, and clean library interface to state-of-the-art registration algorithms.
    • A repository of benchmarks that allow user to more easily choose an algorithm for their application, and a platform that allows researchers to compare algorithms with minimal amount of work.
  • What is SuperElastix? SuperElastix is a joint community effort to provide an open source, cross-platform image registration toolbox. It is designed for researchers to compare algorithms in standardized ways, and for users to easily be able to run many different registration algorithms. SuperElastix's C++ API therefore serves as the gateway to this challenge. Submissions to the challenge are implemented as SuperElastix "components". Components are isolated, modular code blocks that send and receive data via a predefined set of interfaces. Interface compatibility is enforced by the compiler, which guarantees mathematical compatibility between connected components, and allow experiments to be automated. The core framework consists of a thin layer that knows how to instantiate components, setup a pipeline of components described by a parameter file, and handle input/output data. A component can be a complete registration algorithm or subcomponent thereof. At the start of the challenge, SuperElastix contains elastix, niftyreg and the ITKv4 registration toolboxes, which also serves as example code for participants to use, and as baseline scores. Full documentation is here.

Participation guidelines

Read the rules and check out the documentation!