Covid-19 MLIA @ Eval organizes a community evaluation effort aimed at accelerating the creation of resources and tools for improved MultiLingual Information Access (MLIA) in the current emergency situation with a reference to a general public use case:
“Sofia has heard that a drug has been experimented in different countries and she would like to have a consolidated and trustworthy view of the main findings, whether the drug is effective or not, and whether there are any adverse effects”.
Distillation for the general public also implies a level of specialist-non-specialist communication, when the aggregated sources contain both disseminative and specialised sources. Therefore, the general public would need to understand medical expertise by using their correspondent in the “everyday ” language or by using an appropriately calibrated language for the communication to be effective.
Covid-19 MLIA @ Eval adopts an incremental and iterative evaluation methodology to enable the release of intermediate (but functional) resources and to progressively (iteration-after-iteration) move towards finally consolidated tools and resources. We plan for three rounds, tentatively one and half-month long each.
Each round consists of the following phases:
* Data release.
* Submission of participants’ runs.
* Ground-truth creation (when needed).
* Scoring of participants’ runs.
* Writing of the rolling technical report.
* Round wrap virtual meeting.
An integral part of the Covid-19 MLIA @ Eval approach is the sharing of information and ideas among the participants. This happens via the rolling technical report where participants describe their solutions as the evaluation rounds progress and the virtual meetings at the end of each round where participants briefly present the main highlights of what worked and what did not work and interactively discuss together in order to share ideas and improve for the next round.
Moreover, to facilitate the exchange of resources and components and to jointly work together for improving MLIA technologies for Covid-19, participants are provided with a dedicated git repository where to push and share the outcomes of your participation in the different rounds, i.e. runs, code, (language) resources, and a technical report. To facilitate sharing and re-use, all the contents of the repositories are released under the Creative Commons Attribution-ShareAlike 4.0 International License.
Machine Translation Task
The goal of the Machine Translation task is to evaluate systems focused on the Covid-19 related text. The Covid-19 MT task addresses the following language pairs:
* English-German.
* English-French.
* English-Spanish.
* English-Italian.
* English-Modern Greek.
* English-Swedish.
* English-Arabic.
All languages pairs only in the direction translating from English to the other language. The main challenge is that the text to be translated is specialized on the new and high-relevant topic of Covid-19. The task is open for beginners and established research groups from any area of interest in the scientific community, the public administration and the industry. At the end of each round, participants will write/update an incremental report explaining their system. The report will highlight which methods data have been used.
Organizers
Overall
* Khalid Choukri, ELDA, France.
* Nicola Ferro, University of Padua, Italy.
Data Acquisition and Engineering
* Miltos Deligiannis, ILSP/Athena RC, Greece.
* Marwa Hadj Salah, ELRA/ELDA, France.
* Guillaume Jacquet, JRC, Italy.
* Vassilis Papavassiliou, ILSP/Athena RC, Greece.
* Stelios Piperidis, ILSP/Athena RC, Greece.
* Prokopis Prokopidis, ILSP/Athena RC, Greece.
Information Extraction
* Cyril Grouin, LISN, France.
* Thierry Declerck, DFKI, Germany.
* Pierre Zweigenbaum, LISN, France.
Multilingual Semantic Search
* Giorgio Maria Di Nunzio, University of Padua, Italy.
* Maria Eskevich, CLARIN ERIC.
Machine Translation
* Francisco Casacuberta, Universitat Politècnica de València, Spain.
* Miguel Domingo, Universitat Politècnica de València, Spain.
* Mercedes García-Martínez, Pangeanic, Spain.
* Manuel Herranz, Pangeanic, Spain.
Supporters
Covid-19 MLIA @ Eval is an evaluation effort promoted by several communities which are closely working together.
* European Commission.
* European Language Resources Coordination (ELRC).
* European Language Resources Association (ELRA).
* European Research Infrastructure for Language Resources and Technology (CLARIN).
* CLEF Initiative.
You can find additional information at http://eval.covid19-mlia.eu/task3/.