REAL document-based MEDical Natural Language Processing


Recently, more and more medical records are written in electronic format in place of paper, which leads to a higher importance of information processing techniques in medical fields. However, the amount of privacy-free medical text data is still small in non-English languages, such as Japanese and Chinese. In such a situation, we had proposed a series of previous four medical natural language processing (MedNLP) tasks, MedNLP-1, MedNLP-2, MedNLPDoc, and MedWeb. In MedNLP-1 as one of the NTCIR-10 pilot tasks, we proposed a fundamental task, named entity recognition (NER), using dummy medical records created by medical doctors. In MedNLP-2 as an NTCIR-11 core task, we provided a term normalization task using dummy medical records created by medical doctors. In MedNLPDoc, one of the NTCIR-12 core tasks, we designed a complete task starting from a medical record from a medical textbook to give a proper disease name represented by the ICD code. In MedWeb as an NTCIR-13 core task, a disease tweet classification task was designed to stimulate the use of social media data in medical/healthcare domains, and dummy Twitter data was created in Japanese and translated into English and Chinese. Thus, we did not utilize the real data and relied only on dummy data.

In this proposed pilot task, we re-design the scheme for our ultimate goal that promotes and supports developing practical tools and systems applicable in the medical industry, which will help medical doctors and co-medicals to make medical decisions and treatments (what is called medical AI tasks). To do so, this pilot task provides the two core resources; (1) Case-Report dataset and (2) Radiographic-Report dataset. More importantly, we prepare the real data in Japanese and translate the original reports into English, enabling us to develop the first benchmark for multi-language medical NLP. Participants are supposed to retrieve important information from real medical documents in Japanese and translated them into English. The task is two folds; (I) Named Entity Recognition subtask and (II) document classification (ADE (adverse drug event detection cor Case-Report; TMN classification for Radiographic reports) subtask for the two resources, which were designed from the practical viewpoint. This task will yield promising technologies to develop practical computational systems for supporting a wide range of medical services.

Data Set

  • Case-Report
    •  Named Entity Recognition
      • Diagnosis, disease & Symptoms(病名・症状)
      • Drug name
      • Body region (Human body part)
    • Document Classification
      • Outcome (death / in hospital / follow up)(転帰)
      • ADE (adverse drug event)(副作用有無)
  • Radiographic-Report
    • Named Entity Recognition
      • Diagnosis, disease & Symptoms(病名・症状)
      • Body region (Human body part)
    • Document Classification
      • TNM classification(腫瘍分類)


Participants are supposed to extract information from medical reports written by physicians.

(1) Case-Report

[What is a case report] A case report is a kind of medical research paper written for a patient. The case report analysis has potentially two advantages; the case report covers most of the disease timeline or a history of the target disease, and the number of case reports is greater than the other papers because each medical society usually has a submission truck for case reports. Considering these advantages, the case reports could be a rich information resource. Note that the format of a case report is similar to the one of a discharge summary, which is a kind of medical report. Therefore, techniques for the case report analysis could be expanded to analyze discharge summaries.

[Dataset] This dataset comprises a set of 100 open-access case reports available at CiNii. Because the number of medical societies that make publications open-access is limited, types of patients and diseases reported in open-access case reports are highly biased. To reduce the bias caused by each medical society’s publication policy, we select 100 case reports based on actual frequencies of patients and diseases.

An example of named entity-annotated case reports (styled using CSS and HTML)

Sample (in Japanese)

(2) Radiographic-Report

[What is a radiographic report] A radiographic report is a kind of clinical document written by a radiologist. Basically, it focuses on a single radiography image and describes all findings (including potential diseases) expected from the image. While a report and its target image are a pair, most research on radiographic report analysis tends to focus only on images because an image-based AI draws much attention, such as automatic diagnosis of X-rays, CT, and MRI. One of the biggest problems when handling radiographic reports is a variety of writing styles. Although the diagnosis can be written in a variety of ways (diversity of expression), usually only one report is created for one image. For this reason, simply collecting reports from medical institutions could not give us enough information about how people write texts for the same diagnosis. To solve this problem, multiple (8 or more) doctors make their reports independently for the same CT image.

[Dataset] This dataset comprises a set of 15 cases, in which 9 different radiologists describe the findings for each report. In total 135 texts would be available.

Sample (in Japanese)



Sep 1, 2021: Data-set Release (We will provide samples before the date)
Sep-Nov 2021: Dry Run
Dec 2021: Formal Run
Feb 1, 2022: Evaluation Result Release
Feb 1, 2022: Draft Task Overview Paper Release
Mar 1, 2022: Draft Participant Paper Submission Due
May 1, 2022: All Camera-ready Paper Submission Due