SIGNLL (ACL's Special Interest Group on Natural Language Learning) invites proposals for the CoNLL Shared Task 2017.
Since 1999, CoNLL (the Conference on Computational Natural Language Learning) has included a shared task in which training and test data are provided by the organizers, which allows participating systems to be evaluated and compared in a systematic way. Descriptions of the participating systems and an evaluation of their performance are presented both at the conference and in the proceedings. Previous shared tasks have been devoted to chunking (1999, 2000), clausing (2001), named entity recognition (2002, 2003), semantic role labeling (2004, 2005), dependency parsing (2006, 2007), joint parsing of syntactic and semantic dependencies (2008, 2009), hedge detection (2010), co-reference resolution (2011, 2012), grammatical error correction (2013, 2014), and shallow discourse parsing (2015, 2016).
For CoNLL-2017, proposals are invited for any shared task involving natural language learning, including not only natural language processing tasks and applications but also the modeling of human language learning. The CoNLL-2017 conference will be co-located with ACL in Vancouver, Canada, from 30 July 2017 to August 4 2017.
To ease reproducibility, CoNLL Shared Tasks have been hosted on the TIRA platform (http://www.tira.io/) since 2015. We strongly encourage the use of this platform for the next shared task.
Proposals should contain the following information:
- Names and affiliations of the members of the organizing committee (including a designated chair and contact person)
- Description of the shared task, including its relevance for the study of computational natural language learning
- Description of the process for selecting, acquiring and preparing the data for training and evaluation, including conditions of availability
- Description of the procedure for evaluating the performance of participating systems, including the availability of evaluation software
- Any other relevant information, including previous evaluations for the proposed task
A tentative timeline for the CoNLL Shared Task 2017 is as follows:
- October 15, 2016: general announcement (made by SIGNLL)
- December 2016: call for interest in participation, including a description and a schedule
- February 2017: task website (first version with general information, task definition, examples and calendar), trial datasets (plus documentation and scorer), and registration for the task (plus corpora licenses)
- March 2017: training and development sets available
- May 2017: test sets available and final evaluations
Proposals should make clear that a similar schedule can be implemented, especially for the preparation of resources (trial/training/development/test data sets).
The deadline to submit a proposal is August 31, 2016. Proposals should be in PDF format and the main content should be limited to 3 pages. This said, we encourage proposers to include annexes with examples that help understanding the data and type of evaluation. Proposals should be submitted by e-mail to proposals@signll.org, from the email of the contact person, with email subject “CoNLL-2017 Shared Task Proposal”. Proposals will be reviewed by the SIGNLL steering committee and proposers will be notified about the selection before October 1, 2016. Proposers might be contacted during the reviewing period to provide further information.
For any question related to this selection process, please email proposals@signll.org.