What is the Continuous Registration Challenge?

Continuous Registration Challenge (CRC) is a challenge for registration of lung- and brain images inspired by modern software development practices. Participants implement their algorithm using the open source SuperElastix C++ API (documentation) which comes with a continuous integration-, experiments-, and evaluation system out of the box. All submissions are automatically compiled, tested, and benchmarked weekly on eight different data sets. Participants are listed on the owner's page, and performances are listed on the leaderboard page. If you use SuperElastix for other work you should cite the owner of the algorithm, not SuperElastix.

The challenge focuses on pairwise registration of lungs and brains, two problems frequently encountered in clinical settings. We have collected seven open-access data sets and one private data set (3+1 lung data sets, 4 brain data sets). The challenge results will be presented and discussed at the upcoming Workshop On Biomedical Image Registration (WBIR 2018). All participants are invited to co-author a journal paper describing the results of the challenge. We will submit the paper to a leading journal in the field.

For this challenge, a submission consists of code and only code. Benchmarking is done in a fully automated manner and will continue to do so even after the challenge ends. The goal is to provide an open source, collaborative platform where you and your peers develop, compare, and continuously improve algorithms and parameter settings.

Important dates
  • February 5th 2018: Announcement of challenge and release of SuperElastix 1.0.
  • Introductory webinars every Monday from February 12th to May 6th at 16:00 CET. Join today at meet.google.com/spz-nkmu-cyy!
  • Technical Q&A webinars every Tuesday from February 20th to June 5th 2018 16:00 CET. Link will be posted here 15 min before the webinar starts (we use Google Hangouts so Chrome is required).
  • June 14th 2018: Final commit to the SuperElastix repository for challenge participants.
  • June 28/29 2018: Results presented at the Workshop of Biomedical Image Registration 2018.
  • June 30 2018: SuperElastix Hackathon by Matt McCormick from Kitware.
FAQ
  • How do I participate? Implement your registration method in SuperElastix and commit your code via a pull request (PR) to the SuperElastix repository on Github (see an example implementation here). Then make a paremeter file that uses your component and commit it in the challenge folder in the SuperElastix repository. See a more detailed description in the documentationRead more about SuperElastix hereRead the full developer documentation hereRead the SuperElastix paper here
  • How do I get attribution for my submission? You will be invited to co-author a journal paper describing the results of the challenge and you will be listed as an owner of your component here. Users are asked to cite your paper everytime a registration is run that uses some of your code.
  • Do I have to write an entire registration algorithm to participate? No! You can also tune an existing registration method in SuperElastix and just submit a parameter file!  
  • Will you help me implement my method in SuperElastix? Yes, we have weekly introductory webinars and weekly technical Q&A webinars as long as the challenge runs. In addition, you can join our slack channelopen an issue on Github, or join the challenge's mailing list.
  • Will other participants be able to see my code in the GitHub repository? Yes, everyone can see each others work, learn from each other, and improve from each others experience. That is open source.
  • Why should I participate? Your method will accessible to users and researchers via the SuperElastix library even after the challenge ends. You will be cited every time a user uses your method for academic work (SuperElastix prints your citing information everytime your submissions is used). You will get a build system, continuous integration (CI) testing system, and have your method automatically tested on many different data sets and continuously benchmarked against other registration algorithms during and after the challenge. Participating in this challenge is much easier that doing experiments yourself!
  • What is the expected outcome? The primary outcome is a high-quality, open source, software library that makes many different registration algorithms easily accessible to experts in other domains that just uses image registration as a tool. Specifically, we have launched this challenge to develop
    • An open source, collaborative platform where researchers develop, compare, and continuously improve image registration algorithms.
    • A high-quality codebase, command line interface, and clean library interface to state-of-the-art registration algorithms.
    • A repository of benchmarks that allow user to more easily choose an algorithm for their application, and a platform that allows researchers to compare algorithms with minimal amount of work.
  • What is SuperElastix? SuperElastix is a joint community effort to provide an open source, cross-platform image registration toolbox. It is designed for researchers to compare algorithms in standardized ways, and for users to easily be able to run many different registration algorithms. SuperElastix's C++ API therefore serves as the gateway to this challenge. Submissions to the challenge are implemented as SuperElastix "components". Components are isolated, modular code blocks that send and receive data via a predefiend set of interfaces. Interface compatibility is enforced by the compiler, which guarantees mathematical compatibility between connected components, and allow experiments to be automated. The core framework consists of a thin layer that knows how to instantiate components, setup a pipeline of components described by a parameter file, and handle input/output data. A component can be a complete registration algorithm or subcomponent thereof. At the start of the challenge, SuperElastix contains elastix, niftyreg and the ITKv4 registration toolboxes, which also serves as example code for participants to use, and as baseline scores. Full documentation is here.
Participation guidelines
This challenge is held as part of the WBIR 2018 conference. Read the Rules!
Timeline