Skip to search boxSkip to navigationSkip to main content

Why is the Winner the Best?

  • M. Eisenmann
    ,
  • A. Reinke
    ,
  • V. Weru
    ,
  • M. D. Tizabi
    ,
  • F. Isensee
    ,
  • T. J. Adler
  • German Cancer Research Center
    ,
  • Heidelberg University 
    ,
  • University of Leeds
    ,
  • University of Lausanne
    ,
  • University of Applied Sciences Western Switzerland
    ,
  • Technische Hochschule Ingolstadt
Research Output:
Chapter in Book/Report/Conference proceeding
Conference contribution
Research
Peer-review

Abstract

International benchmarking competitions have become fundamental for the comparative performance assessment of image analysis methods. However, little attention has been given to investigating what can be learnt from these competitions. Do they really generate scientific progress? What are common and successful participation strategies? What makes a solution superior to a competing method? To address this gap in the literature, we performed a multicenter study with all 80 competitions that were conducted in the scope of IEEE ISBI 2021 and MICCAI 2021. Statistical analyses performed based on comprehensive descriptions of the submitted algorithms linked to their rank as well as the underlying participation strategies revealed common characteristics of winning solutions. These typically include the use of multi-task learning (63%) and/or multi-stage pipelines (61%), and a focus on augmentation (100%), image preprocessing (97%), data curation (79%), and post-processing (66%). The 'typical' lead of a winning team is a computer scientist with a doctoral degree, five years of experience in biomedical image analysis, and four years of experience in deep learning. Two core general development strategies stood out for highly-ranked teams: the reflection of the metrics in the method design and the focus on analyzing and handling failure cases. According to the organizers, 43% of the winning algorithms exceeded the state of the art but only 11% completely solved the respective domain problem. The insights of our study could help researchers (1) improve algorithm development strategies when approaching new problems, and (2) focus on open research questions revealed by this work.

Research Output Information

Output type

Research Output:
Chapter in Book/Report/Conference proceeding
Conference contribution
Research
Peer-review

Original language

English

Pages from-to (Number of pages)

Pages 19955-19966 (12 pages)

Publication milestones

  • Published - 2023

Publication status

Published - 2023

Publisher

IEEE Computer Society, United States

Publication series

  • Publication series name: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
    ISSN (Print): 1063-6919
    Volume: 2023-June

ISBN (Electronic)

9798350301298

Publication IDs

  • Scopus: 85173128601

Host publication title

Proceedings - 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023

Publication metrics

Metrics

SciVal
FWCI
1.55
SciVal
Author count
125
SciVal
Paper percentile
82
SciVal
citations
23
Scopus
citations

PlumX, opens in new tab

Captures
72
Citation count
23