NTCIR-16 Real-MedNLP

 With the advent of the internet age, medical records are increasing being written in electronic formats in place of paper, which leads to a higher importance of information processing techniques in medical fields. However, the amount of privacy-free medical text data is still small in non-English languages, such as Japanese and Chinese. In such a situation, we had proposed a series of previous four medical natural language processing (MedNLP) tasks, MedNLP-1, MedNLP-2, MedNLPDoc, and MedWeb. This task will yield promising technologies to develop practical computational systems for supporting a wide range of medical services.



6.11 日本語版ホームページを公開しました
6.18 Our English website is available


Real-MedNLP is a shared task workshop for medical language processing using actual medical documents (case reports and radiographic reports). The goal of this task is to promote the development of practical systems that support various medical services.

The Real-MedNLP task has two corpus-based tracks (MedTxt-CR Track and MedTxt-RR Track), each with three subtasks.

Guideline:coming soon


【MedTxt-CR Corpus】 
 This dataset comprises a set of open-access case reports available at CiNii. Typically, open-access case reports are biased in the reporting of patients and diseases, due to differences in the policies various medical societies have towards open-access publications.
 To reduce such bias caused by each medical society’s publication policy, we select case reports based on actual frequencies of patients and diseases.
 - Training set: 100 reports
 - Test set: 100 reports

 A case report is detailed description of a patient’s medical condition for research purposes. They track the onset and temporal progression of the patient’s disease and are large in quantity as medical societies typically have dedicated submission tracks for consolidating these reports. Considering these advantages, case reports possess enormous potential as a rich source of information. Furthermore, the format of a case report is similar to that of a discharge summary, which is frequently used in healthcare contexts. Techniques developed here for case report analysis could also be applied to analyze discharge summaries.

【MedTxt-RR Corpus】 
 This dataset comprises a set of 15 cases, in which 9 different radiologists describe the findings for each report. In total, 135 texts will be made available.
 - Training set: 72 texts
 - Test set: 63 texts

 A radiology report is a type of clinical document that is written by a radiologist. Basically, they focus on a single radiography image and describes all potential findings (including potential diseases) that can be expected from the image. While reports and target images are paired, most research on radiographic reports tends to focus only on images, due to the hype surrounding image-based AI  (such as automatic diagnosis of X-rays, CT, and MRI). One of the biggest problems when handling radiographic reports is in the variety of writing styles. Although a diagnosis can be written in a variety of ways (diversity of expression), conventionally, only one report is created per image. As such, simply collecting reports from medical institutions may not yield enough information on the variability in reporting styles for the same diagnosis. Consequently, we included independent reports from multiple doctors for the same CT image.


MedTxt-CR sample

MedTxt-RR sample


Task Overview

1. Few-resources NER challenge

Since NER is arguably the most fundamental information extraction problem for MedNLP, we designed challenges pertaining to NER on our real-world clinical documents, with a sample of only 100-200 documents. This scale of corpus size tends to be regarded as a “few-resource machine learning”, which is a de facto standard for any sort of MedNLP in general.

Subtask 1 : Just 100 Training

  • NER using the training set consisting of 100 documents
  • This subtask is equivalent to standard supervised learning with few resources

Subtask 2 : Guideline Learning

  • NER using the example text for each tag in Annotation Guideline
  • This simulates the training of human annotators, who often learn from the annotation guidelines provided by researchers.


2. Applications

This task is an application-focused subtask designed for each corpus track from a practical point-of-view.

Subtask 3:Application

  • Adverse Drug Event detection (ADE) for MedTxt-CR Track
     Extract adverse drug event (ADE) information from case reports and create a table
  • Same Patient Identification (SPI) for MedTxt-RR Track
     Identify the radiographic-report for the same patient (case).



  • September 2021: Datasets release
  • September-November 2021: Dry Run
  • December 2021-February 2022: Formal Run
  • February 1, 2022: Evaluation Result Release
  • February 1, 2022: Draft Task Overview Paper Release
  • March 1, 2022: Draft Participant Paper Submission Due
  • May 1, 2022: All Camera-ready Paper Submission Due
  • June 14-17,  2022: NTCIR-16 Conference (NII, Tokyo, Japan) (An option for online presentation will be available)


Entry Form

Please apply through the NTCIR-16 website.



Y’s Reading inc.

Social Computing Laboratory


If you have any questions, please contact the Real-MedNLP office.

 Real-MedNLP office email:real-mednlp[at]is.naist.jp