RadNLP 2024 shared task: Natural Language Processing for Radiology (NTCIR-18)

RadNLP 2024

RadNLP 2024 (Natural Language Processing for Radiology) is a shared task focusing on the application of natural language processing in radiology. The organizing team prepares the tasks, publishes the dataset, and calls for solutions from participants.

The aim of RadNLP 2024 is to create open medical data and to contribute insights back to medical and informatic communities.

RadNLP 2024 is held as a core task of the international conference NTCIR-18, organized by the National Institute of Informatics in Japan.

@radnlp

Tweets by RadNLP

News

Task overview

Objective

The objective of RadNLP 2024 is to automatically determine the stage (i.e., the degree of progression) of lung cancer from radiology reports.

Radiology reports are clinical documents authored by radiologists and sent to referring clinicians, in which medical images such as CT and MRI are described and interpreted.

Radiology reports are rich in information related to cancer staging, which can be essential for clinical or research purposes. However, radiology reports do not always specify the stage of the cancer explicitly¹, which imposes extra workload on human experts to read them through and extract information manually.

RadNLP 2024 aims to aid clinical practice by automating cancer staging from radiology reports using the natural language processing technique.

¹ Sexauer R et al. Towards more structure: comparing TNM staging completeness and processing time of text-based reports versus fully segmented and annotated PET/CT data of non-small-cell lung cancer. Contrast Media Mol Imaging 2018:5693058.

Dataset

[UPDATED] RadNLP 2024 datasets contain 243 English or Japanese radiology reports:

English Track

The dataset consists of 243 English radiology reports.

Japanese Track

The dataset consists of 243 Japanese radiology reports.

Participants are welcome to join either the English track, the Japanese track, or both. Scoring and ranking will be conducted independently for each track.

Our datasets contain NO personal health information. The radiology reports are not derived from real medical institutions but are created with crowdsourcing by diagnosing de-identified images on Radiopaedia². Task participants requires no complex applications to use our datasets.

² Nakamura Y et al. Clinical Comparable Corpus Describing the Same Subjects with Different Expressions. Stud Health Technol Inform 2022:290:253-257.

RadNLP 2024 is a multi-label document classification task to correctly determine T, N, and M categories for each radiology report.

Gold standard labels for the training and validation datasets are provided as CSV tables. Each table has four columns (ID, T, N, and M):

ID: unique integers assigned to each radiology report.
T: gold standard for the T category.
N: gold standard for the N category.
M: gold standard for the M category.

ID	T	N	M
1	3	3	1
2	1	0	0
3	3	1	0
4	2	0	0
...	...	...	...

We provide another empty CSV table to fill in the prediction results for the test set. Task participants should fill in the table and submit it to the task organizer team in the task period. After the submission deadline, submission scores are calculated and sent back to the task participants.

Important dates

~~May 29, 2024: Kick-off event~~
July 2024: Release of the training and validation datasets
November 2024: Registration deadline
November 2024: Release of the test dataset
January 2025: Submission deadline of the prediction results
February 2025: Return of scores
March 2025: Submission deadline of the system paper draft
May 2025: Submission deadline of the camera ready version of the system paper
June 10–13, 2025 (JST): NTCIR-18 conference at the National Institute of Informatics, Tokyo, Japan