shared task

GermEval 2015: LexSub

ACL-Sponsored Event: 
No
Event Notification Type: 
Call for Participation
Location: 
University of Duisburg-Essen, Campus Essen
Event Dates: 
Tuesday, 29 September 2015
Submission Deadline: 
Wednesday, 15 July 2015
Contact: 
Tristan Miller
Contact Email: 
miller [at] ukp.informatik.tu-darmstadt.de
City: 
Essen
Country: 
Germany
Description/Details: 

Word sense disambiguation (WSD) has been a core research problem in computational linguistics since the very inception of the field. In recent years, there has been considerable interest in using lexical substitution as an extrinsic evaluation of WSD systems. This has led to a number of mono- and crosslingual evaluation competitions at SemEval and EVALITA. We now invite all researchers and industry professionals to participate in GermEval 2015: LexSub, the first lexical substitution task for the German language. The task is associated with the GSCL 2015 conference in Essen, and will take place as a workshop there on 29 September 2015.

Task description

Lexical substitution is the task of identifying an appropriate substitute for a target word in a given context. For example, in the sentence "She's a bright kid who excels academically," an appropriate substitute for "bright" might be "smart", whereas an inappropriate one would be "glowing". Automatically identifying substitution candidates, and selecting those which best match the context, requires intelligent application of lexical-semantic knowledge and word sense disambiguation techniques. However, unlike traditional WSD tasks, lexical substitution does not mandate the use of any particular sense inventory.

The data for the GermEval 2015: LexSub task is described by Cholakov et al. in "Lexical substitution dataset for German" (Proc. LREC, 2014). All together it consists of 2040 sentences from the German Wikipedia, each containing a target word and a list of substitutions proposed by human annotators. There are 153 unique target words, equally distributed across parts of speech (nouns, verbs, and adjectives) and three frequency groups. About half of this data (26 nouns, 26 verbs, and 26 adjectives in 1040 sentence contexts) forms the training set, which will be made available to participants in advance. The remainder forms the test set, which will be used for the evaluation and published in full only after the shared task is completed.

Participants need not rely on any particular language resources, but if they wish they can employ the sense-linked lexical-semantic resource UBY and JoBimText distributional semantics models. UBY also provides an interface to GermaNet. Industrial users will be eligible to a special GermaNet licence to be obtained from Eberhard-Karls Universität Tübingen. Please refer to our web pages on how to obtain the data sets and resources.

Systems' performance will be measured by comparing their substitutes against those selected by the human annotators; for this we will use the "best", "out of ten", and "generalized average precision" metrics. The organizers will provide a scoring system and the output of some baseline systems.

Practical information

* 23 January 2015: Availability of training data
* 1 July 2015: Availability of test data
* 15 July 2015: Deadline for initial submission of papers and results
* 1 August 2015: Notification of acceptance and shared task results
* 15 August 2015: Deadline for camera-ready papers
* 30 September–2 October 2015: GSCL 2015

Submissions will consist of a file providing the substitutions for each instance of the target data and a paper of up to four pages (including references) describing the approach and analyzing the performance. Papers should follow the GSCL 2015 style guide, and will be reviewed and published in an online volume of workshop proceedings. (We may ask participants to peer-review other submissions.) Participants are expected to present summaries of their systems at the GermEval 2015: LexSub workshop at GSCL 2015.

Organizing committee

* Sallam Abualhaija, Technische Universität Hamburg-Harburg
* Darina Benikova, LT Group, Technische Universität Darmstadt
* Chris Biemann, LT Group, Technische Universität Darmstadt
* Judith Eckle-Kohler, UKP Lab, Technische Universität Darmstadt
* Iryna Gurevych, UKP Lab, Technische Universität Darmstadt
* Tristan Miller, UKP Lab, Technische Universität Darmstadt

To contact the organizing committee, please post to the GermEval 2015: LexSub mailing list, or for private communication e-mail miller [at] ukp.informatik.tu-darmstadt.de (Tristan Miller).

Acknowledgements

This shared task is supported by the DFG-funded project “Integrating Collaborative and Linguistic Resources for Word Sense Disambiguation and Semantic Role Labeling” (InCoRe, GU 798/9-1), the BMBF-funded CLARIN F-AG7, and the LOEWE research cluster “Digital Humanities”.

CoNLL-2014 Shared Task: Grammatical Error Correction

ACL-Sponsored Event: 
No
Event Notification Type: 
Call for Participation
Event Dates: 
Thursday, 26 June 2014 to Friday, 27 June 2014
Submission Deadline: 
Wednesday, 19 March 2014
Contact: 
Hwee Tou Ng
Siew Mei Wu
Ted Briscoe
Christian Hadiwinoto
Raymond Hendy Susanto
Contact Email: 
conll14st [at] gmail.com
City: 
Baltimore
State: 
Maryland
Country: 
USA
Description/Details: 

CoNLL-2014 Shared Task: Grammatical Error Correction
Baltimore, Maryland, USA
June 26-27, 2014

http://www.comp.nus.edu.sg/~nlp/conll14st.html

**NEW**: 35 teams have registered so far to participate in the shared task.
We are also planning for a journal special issue on grammatical error correction after the conclusion of the shared task.

============================
Final Call for Participation
============================

CoNLL-2014 will continue the CoNLL tradition of having a high profile shared task in natural language processing. This year's shared task will be grammatical error correction, a continuation of the CoNLL shared task in 2013. A participating system in this shared task is given short English texts written by non-native speakers of English. The system detects the grammatical errors present in the input texts, and returns the corrected essays. The shared task in 2014 will require a participating system to correct all errors present in an essay (i.e., not restricted to just five error types in 2013). Also, the evaluation metric will be changed to F0.5, weighting precision twice as much as recall.

The grammatical error correction task is impactful since it is estimated that hundreds of millions of people in the world are learning English and they benefit directly from an automated grammar checker. However, for many error types, current grammatical error correction methods do not achieve a high performance and thus more research is needed.

Participating teams will be provided with common training data in which grammatical errors have been annotated. Blind test data will be used to evaluate the outputs of the participating teams using a common scoring software and evaluation metric.

Registration
------------

Registration for the shared task has begun. To participate in the shared task, please download the license form
(http://www.comp.nus.edu.sg/~nlp/conll14st/nucle_license.pdf) for usage of the training data. Print the form, sign, and have the scanned PDF file of the signed form ready. You may then proceed to register your team through our registration page
(http://sterling8.ddns.comp.nus.edu.sg/conll14st/register/) by providing the particulars (name, affiliation, and email address) of the main contact person and all the other team members. Please upload your scanned PDF file of the *signed* form through the same page.

After you have registered, we will get back to you within two days. If your registration is complete, you will be provided with the training data and the scorer for the shared task. You will also be added to a Google Group where discussions of the details of the shared task will take place. As such, please register early so that you are informed of the details of the shared task. If you encounter problems in registering or you do not hear from us after two days, please inform us through the shared task contact email address conll14st [at] gmail.com.

Important Dates
---------------

* November 22, 2013: announcement of shared task
* December 5, 2013: set up of shared task website
* December 27, 2013: registration begins and release of training set and scorer
* January 22, 2014: registration deadline
* March 16, 2014: test set available
* March 19, 2014: systems' outputs collected
* March 26, 2014: system results due to participants
* April 2, 2014: shared task system papers due
* April 11, 2014: reviews due
* April 14, 2014: notification of acceptance
* April 21, 2014: camera ready version of shared task system papers due
* June 26-27, 2014: CoNLL-2014 conference (Baltimore, Maryland, USA)

Shared Task Organizers
----------------------

* Hwee Tou Ng (Chair), National University of Singapore
* Siew Mei Wu, National University of Singapore
* Ted Briscoe, University of Cambridge
* Christian Hadiwinoto, National University of Singapore
* Raymond Hendy Susanto, National University of Singapore

Contact
-------

Questions about the CoNLL-2014 shared task can be sent to conll14st [at] gmail.com.

CoNLL-2014 Shared Task: Grammatical Error Correction

ACL-Sponsored Event: 
No
Event Notification Type: 
Call for Participation
Event Dates: 
Thursday, 26 June 2014 to Friday, 27 June 2014
Submission Deadline: 
Wednesday, 19 March 2014
Contact: 
Hwee Tou Ng
Siew Mei Wu
Ted Briscoe
Christian Hadiwinoto
Raymond Hendy Susanto
Contact Email: 
conll14st [at] gmail.com
City: 
Baltimore
State: 
Maryland
Country: 
USA
Description/Details: 

================================================================================

CoNLL-2014 Shared Task: Grammatical Error Correction
Baltimore, Maryland, USA
June 26-27, 2014

http://www.comp.nus.edu.sg/~nlp/conll14st.html

================================================================================
Call for Participation
================================================================================

CoNLL-2014 will continue the CoNLL tradition of having a high profile shared
task in natural language processing. This year's shared task will be grammatical
error correction, a continuation of the CoNLL shared task in 2013. A
participating system in this shared task is given short English texts written by
non-native speakers of English. The system detects the grammatical errors
present in the input texts, and returns the corrected essays. The shared task in
2014 will require a participating system to correct all errors present in an
essay (i.e., not restricted to just five error types in 2013). Also, the
evaluation metric will be changed to F0.5, weighting precision twice as much as
recall.

The grammatical error correction task is impactful since it is estimated that
hundreds of millions of people in the world are learning English and they
benefit directly from an automated grammar checker. However, for many error
types, current grammatical error correction methods do not achieve a high
performance and thus more research is needed.

Participating teams will be provided with common training data in which
grammatical errors have been annotated. Blind test data will be used to evaluate
the outputs of the participating teams using a common scoring software and
evaluation metric.

Registration
------------

Registration for the shared task has begun. To participate in the shared task,
please download the license form
(http://www.comp.nus.edu.sg/~nlp/conll14st/nucle_license.pdf) for usage of the
training data. Print the form, sign, and have the scanned PDF file of the signed
form ready. You may then proceed to register your team through our registration
page (http://sterling8.ddns.comp.nus.edu.sg/conll14st/register/) by providing
the particulars (name, affiliation, and email address) of the main contact
person and all the other team members. Please upload your scanned PDF file of
the *signed* form through the same page.

After you have registered, we will get back to you within two days. If your
registration is complete, you will be provided with the training data and the
scorer for the shared task. You will also be added to a Google Group where
discussions of the details of the shared task will take place. As such, please
register early so that you are informed of the details of the shared task. If
you encounter problems in registering or you do not hear from us after two days,
please inform us through the shared task contact email address
conll14st [at] gmail.com.

Important Dates
---------------

* November 22, 2013: announcement of shared task
* December 5, 2013: set up of shared task website
* December 27, 2013: registration begins and release of training set and scorer
* January 22, 2014: registration deadline
* March 16, 2014: test set available
* March 19, 2014: systems' outputs collected
* March 26, 2014: system results due to participants
* April 2, 2014: shared task system papers due
* April 11, 2014: reviews due
* April 14, 2014: notification of acceptance
* April 21, 2014: camera ready version of shared task system papers due
* June 26-27, 2014: CoNLL-2014 conference (Baltimore, Maryland, USA)

Shared Task Organizers
----------------------

* Hwee Tou Ng (Chair), National University of Singapore
* Siew Mei Wu, National University of Singapore
* Ted Briscoe, University of Cambridge
* Christian Hadiwinoto, National University of Singapore
* Raymond Hendy Susanto, National University of Singapore

Contact
-------

Questions about the CoNLL-2014 shared task can be sent to conll14st [at] gmail.com.

1st Shared Task on Native Language Identification

Abbreviated Title: 
NLI2013
ACL-Sponsored Event: 
No
Event Notification Type: 
Call for Papers
Location: 
co-located with BEA8 Workshop
Event Dates: 
Thursday, 13 June 2013 to Friday, 14 June 2013
Submission Deadline: 
Monday, 11 March 2013
Contact: 
Joel Tetreault
Aoife Cahill
Daniel Blanchard
Contact Email: 
nlisharedtask2013 [at] gmail.com
City: 
Atlanta
State: 
GA
Country: 
USA
Description/Details: 

SHARED TASK DESCRIPTION

We are excited to organize the first shared task in Native Language Identification (NLI) which is the task of identifying the native language (L1) of a writer based solely on a sample of their writing. The task is framed as a classification problem where the set of L1s is known a priori. Most work has focused on identifying the native language of writers learning English as a second language. This problem has been growing in popularity and has motivated several ACL, NAACL and EMNLP papers, as well as a master's and doctorate thesis.

Native Language Identification can be useful for a number of applications. First, it can be used in educational settings to provide more targeted feedback to language learners about their errors. It is well known that speakers of different languages make different kinds of errors when learning a language. A writing tutor system which can detect the native language of the learner will be able to tailor the feedback about the error and contrast it with common properties of the learner's language. Second, native language is often used as a feature that goes into authorship profiling, which is frequently used in forensic linguistics.

The goal of this task is to provide a space to evaluate different techniques and approaches to Native Language Identification. To date, it has been difficult to compare approaches due to issues with training and testing data and a lack of consistency in evaluation standards. In this shared task, we provide a new data set as well provide a framework where different NLI systems can be finally compared. The shared task will be co-located with the 8th Workshop on Innovative Use of NLP for Building Educational Applications on June 13 or 14 in Atlanta, USA:

http://www.cs.rochester.edu/~tetreaul/naacl-bea8.html

DATA

Educational Testing Service (ETS) is making public 11,000 English essays from the Test of English as a Foreign Language(TOEFL) through the LDC with the motivation to create a larger and more reliable data set for researchers to conduct Native Language Identification experiments on. This set, henceforth TOEFL11, comprises 11 L1s with 1,000 essays per L1. The 11 native languages covered by our corpus are: Arabic, Chinese, French, German, Hindi, Italian, Japanese, Korean, Portuguese, Spanish, Telugu, and Turkish. Furthermore, each essay in the TOEFL11 is labeled with an English language proficiency level (high, medium, or low) based on the judgments of human assessment specialists. The essays are usually 300 to 400 words long. 90% of this set will be sequestered as the training data and the remaining 10% will be released as test data.

EVALUATION

The shared task will have three sub-tasks:

* Closed-Training: The first and main task will be the 11 way classification task using only the TOEFL11 for training.
* Open-Training-1: The second task will be to allow the use of any amount or type of training data excluding the TOEFL11.
* Open-Training-2: The third task will be to allow the use of any amount or type of training data.

The same test data will be used for all sub-tasks.

REGISTRATION

If you would like to participate in the NLI Shared Task, you need to formally register in order to obtain the training and test data. To register, please send the following information to nlisharedtask2013 [at] gmail.com:

* Name of Institution or other label appropriate for your team
* Name of contact person for your team
* Email address of contact person for your team

SCHEDULE

January 14 - Training Data Release
March 11 - Test Data Release
March 18 - Submissions Due
March 25 - Results Announcement
April 08 - Papers Due
April 10 - Revision Requests Sent
April 12 - Camera Ready Version Due
June 13 or 14 - NLI Shared Task Presentations @ BEA8 Workshop

ORGANIZERS

Joel Tetreault, Nuance Communications, USA
Aoife Cahill, Educational Testing Service, USA
Daniel Blanchard, Educational Testing Service, USA

Contact email: nlisharedtask2013 [at] gmail.com

Generation Challenges 2012

Abbreviated Title: 
GenChal 2012
ACL-Sponsored Event: 
No
Event Notification Type: 
Call for Papers
Location: 
Starved Rock
Event Dates: 
Wednesday, 30 May 2012 to Friday, 1 June 2012
Submission Deadline: 
Sunday, 15 April 2012
Contact: 
Albert Gatt
Anja Belz
Alexander Koller
Kristina Striegnitz
Contact Email: 
albert.gatt [at] um.edu.mt
a.s.belz [at] brighton.ac.uk
State: 
Illinois
Country: 
USA
Description/Details: 

GENERATION CHALLENGES 2012
--------------------------

To be held in conjunction with INLG 2012, 30 May-01 June 2012, Starved Rock, IL, USA.

Over the past five years, there has been a lot of activity in
connection with shared tasks in Natural Language Generation (NLG).
Six separate sets of shared tasks each with its own data and team of
organisers have so far been run: TUNA (Gatt et al.), GREC (Belz et
al.), GIVE (Koller et al.), QG (Rus et al.), HOO (Dale and Kilgarriff)
SR (Belz, White et al.). The Pilot Attribute Selection for
Generating Referring Expressions (ASGRE) Challenge took place in 2007
(TUNA); the Referring Expression Generation (REG) Challenge in 2008
(TUNA, GREC); the first, second and third Generation Challenges umbrella
events encompassing diverse sets of shared tasks, took place in 2009
(TUNA, GREC and GIVE), 2010 (QG, GREC and GIVE) and 2011 (HOO, SR);
other shared tasks are being prepared. More information about all of these activities can be found via the links on the Generation Challenges 2012 homepage:
http://www.nltg.brighton.ac.uk/research/genchal12.

In order to continue to provide a common forum for these activities,
we are organising Generation Challenges 2012 (GenChal'12), an umbrella
event designed to bring together a variety of shared-task evaluation
efforts that involve the generation of natural language. For some of the
tasks, evaluation results and participating systems will be presented at
the Generation Challenges 2012 Special Session at the 7TH International
Natural Language Generation Conference (INLG'12). The session will follow the
format of previous GenChal results sessions, with presentations of
results by the organisers of the different shared tasks, as well as
presentations of proposals for new shared tasks in the Task Proposals
Track. As at GenChal'10 and GenChal'11, we envisage having a poster session where
participants in the various shared tasks will have the opportunity to
present their approaches and results. In addition, we are planning to
have break-out sessions for individual tasks, providing further
opportunity for current and future task participants to exchange ideas
and feed back to the organisers.

1. Shared Tasks:
- - - - - - - -

The GenChal'12 meeting will be held at INLG'12 in Starved Rock, IL, USA,
30 May-01 June 2012, but the timetables of the individual tasks are not
completely aligned, and they will be organised independently (for
details, please refer to the Call for Participation for each
task). The following is the list of tasks that are running in 2012:

1. Surface Realisation Task 2012: (White, Wanner, Bohnet, Hogan, Stent,
Belz et al.): Generation of sentences from shallow and deep input
represenations. There will be an update report presented at GenChal'12.

2. Helping Our Own (HOO) (Dale): Text-to-text generation
task aiming to improve the quality of texts, in particular those
written by non-native speakers. Call for Registration has been
posted: http://clt.mq.edu.au/research/projects/hoo/hoo2012/callforregistration.html.
The official HOO results session will be hosted at BEA'12. A summary report
will be presented at GenChal'12.

2. QG'12 (Rus et al.): Question Generation task aiming to generate questions
given answers in the form of Natural Language text. Although the QG task will not
be running in 2012, there will be an update report presented at GenChal'12.

2. Track on Proposals for Future Tasks:
- - - - - - - - - - - - - - - - - - - -

We invite submissions of papers describing ideas for future shared
tasks in the general area of language generation. Proposed tasks can
be in the area of core NLG, or in other research areas in which
language is generated, e.g. text-to-text generation (including MT and
summarisation), combining core NLG and MT, or combining core NLG and
text summarisation. Submissions should describe possible future tasks
in detail, including information regarding organisers, task
description, motivating theoretical interest and/or application
context, size and state of completion of data to be used, and
evaluation plans.

3. Instructions for Paper Submissions:
- - - - - - - - - - - - - - - - - - -

Submissions in the Task Proposals Track should be no more than 4 (four)
pages long excluding citations, and should
follow the ACL'12 guidelines using the style files provided via the
ACL 2012 homepage. Papers should be sent in PDF format by email to
nlg-stec [at] itri.brighton.ac.uk.
Task Proposal papers should be submitted no later than 15 April 2012.
Note that this deadline has been extended.

Submissions will be reviewed by at least 3 members of the Generation
Challenges Steering Committee (see below). As reviewing will not be
blind, there is no need to anonymise papers. This is not intended to be
a selective process, but the organisers reserve the right to reject papers
which do not fall within the scope of the GenChal initiative, or which do
not follow guidelines.

Accepted submissions will be included in the INLG'12 proceedings, but
the page limit for camera-ready versions has not yet been finalised
(the final limit will depend on how many submissions there are).

6. Dates:
- - - - -

Paper submission 15th April 2012
Notification of acceptance 20th April 2012
Submission of camera-ready papers 27th April 2012
INLG'12 30 May-01 June 2012

NB: Submission deadlines for the GenChal'12 Shared Tasks differ and can be
found in the calls for participation for the tasks.

7. Generation Challenges Steering Committee:
- - - - - - - - - - - - - - - - - - - - - -

Anja Belz, NLTG, University of Brighton, UK
Albert Gatt, University of Malta and University of Aberdeen, UK
Alexander Koller, Saarland University, Germany
Robert Dale, Macquarie University, Australia
Kevin Knight, ISI, University of Southern California, USA
Chris Mellish, University of Aberdeen, UK
Johanna Moore, University of Edinburgh, UK
Amanda Stent, AT&T Labs, USA
Kristina Striegnitz, Union College, USA

8. Organisation:
- - - - - - - - -

Anja Belz, NLTG, University of Brighton, UK
Albert Gatt, University of Malta and University of Aberdeen, UK
Alexander Koller, Saarland University, Germany
Kristina Striegnitz, Union College, US

Generation Challenges 2012 homepage: http://www.nltg.brighton.ac.uk/research/genchal12
Queries can be emailed directly to Albert Gatt (albert.gatt [at] um.edu.mt)
or Anja Belz (a.s.belz [at] brighton.ac.uk).

*SEM Shared Task 2012

Abbreviated Title: 
*Sem Shared Task 2012
ACL-Sponsored Event: 
No
Event Notification Type: 
Call for Proposals
Submission Deadline: 
Tuesday, 15 November 2011
Contact: 
Suresh Manandhar
Deniz Yuret
Contact Email: 
suresh [at] cs.york.ac.uk
dyuret [at] ku.edu.tr
City: 
Montreal
Country: 
Canada
Description/Details: 

*SEM Shared Task 2012: Call for Proposals

ACL special interest groups SIGLEX and SIGSEM invites proposals for the *SEM Shared Task 2012.

*SEM is the first joint conference on lexical and computational semantics. The main goal of *SEM is to provide a stable forum for the growing number of NLP researchers working on different aspects of semantic processing, which has been scattered over a large array of small workshops and conferences. The first edition of *SEM will be a two-day conference collocated with the North American Chapter of the Association for Computational Linguistics and Human Language Technologies 2012 conference (NAACL HLT), which will take place in Montreal, Canada.

Every year *SEM is planning to include a shared task in which training and test data are provided by the organizers, which allows participating systems to be evaluated and compared in a systematic way. Descriptions of the participating systems and an evaluation of their performance will be presented both at the conference and in the proceedings. The same or similar shared tasks may be chosen for several years in a row to focus the research attention of the community on a particular semantic task. For *SEM 2012, proposals are invited for any shared task related to lexical or computational semantics.

Note that this shared task will be held in addition to the SemEval tasks. SemEval task organizers are welcome to submit their task for the shared task. If accepted, their task will be moved from SemEval to the shared task of *SEM.

Proposals should contain the following information:
- Names and affiliations of the members of the organizing committee (including a designated contact person)
- Description of the shared task, including its relevance for the study of computational and lexical semantics
- Description of the process for selecting, acquiring and preparing the data for training and evaluation, including conditions of availability
- Description of the procedure for evaluating the performance of participating systems, including the availability of evaluation software
- Any other relevant information

Proposals should be in PDF format, maximum 2 pages, and sent by e-mail to (StarSemSharedTask [at] cs.york.ac.uk):

Important Dates:

Shared Task Proposals due: 15 November 2011
Accept/Reject decision due: 25 November 2011

Any questions regarding the proposals can be sent to (StarSemSharedTask [at] cs.york.ac.uk).

The expected timeline for the *SEM shared task 2012 is as follows:
- Dec 1: general announcement
- Dec 15: call for interest in participation
- Dec 15: task website (first version with general information,
task definition, examples and calendar)
- Dec 15: trial datasets (plus documentation and scorer)
- Jan 13: registration for the task (plus corpora licenses)
- Jan 15: training and development sets available
- Feb 21: test set available
- Mar 1: systems' outputs collected

*SEM Shared Task Committee:

*SEM General Chair: Eneko Agirre
*SEM co-chairs : Johan Bos, Mona Diab
Shared Task Co-chairs : Suresh Manandhar and Deniz Yuret

Further details will be available from the *SEM website:

http://ixa2.si.ehu.es/starsem

GIVE-2.5 - Generating Instructions in Virtual Environments - Last Call for System Submissions

Abbreviated Title: 
GIVE-2.5 - Last Call for System Submissions
ACL-Sponsored Event: 
No
Event Notification Type: 
Other
Event Dates: 
Wednesday, 28 September 2011 to Friday, 30 September 2011
Submission Deadline: 
Monday, 23 May 2011
Contact: 
Kristina Striegnitz
Contact Email: 
striegnk [at] union.edu
City: 
Nancy
Country: 
France
Description/Details: 

GIVE-2.5: Last Call for Participation/System Submissions
http://www.give-challenge.org/research/

Part of Generation Challenges 2011
Endorsed by SIGGEN and SIGSEM

We invite you to participate in the next edition of the Challenges on Generating Instructions in Virtual Environments (GIVE) by submitting an NLG system for the GIVE scenario. In this scenario, a human user performs a "treasure hunt" task in a virtual 3D environment. The NLG system's job is to generate, in real time, a sequence of natural-language instructions that will help the user perform this task.

GIVE-1 was organized in 2008-09. We evaluated five natural language generation systems using almost 1200 user interactions. GIVE-2, run in 2009-10, evaluated seven systems on more than 1800 user interactions. The public evaluation period for GIVE-2.5 will be in June and July 2011. The task in GIVE-2.5 will be basically identical to the one in GIVE-2; this is so GIVE-2 systems can be improved based on experiences from the evaluation, and to allow more people to participate in the GIVE-2 task. For more information and to try out the GIVE-2 software, see http://www.give-challenge.org/research.

If you are interested in participating, please email Kristina Striegnitz at striegnk [at] union.edu so that we can send you further information and keep you updated on any new developments.

Overview of the GIVE challenges

The Challenge on Generating Instructions in Virtual Environments (GIVE) is a novel approach to the notoriously hard problem of evaluating NLG systems. In this scenario, a human user performs a "treasure hunt" task in a virtual 3D environment. The NLG system's job is to generate, in real time, a sequence of natural-language instructions that will help the user perform this task. The crucial thing is that users connect to the generation systems over the Internet. By logging how well they were able to follow the system's instructions, we can evaluate the quality of these instructions in
terms of task completion rates and times, subjective measures such as helpfulness and friendliness, and runtime performance. Because the user and the system don't need to be physically in the same place, access to experimental subjects over the Internet becomes easy.

GIVE is a theory-neutral, end-to-end evaluation effort for NLG systems. It involves research opportunities in text planning, sentence planning, realization, and situated communication. One particularly interesting aspect of situating the generation problem in a virtual environment is that spatial and relational expressions play a bigger role than in other NLG tasks. Beyond NLG, GIVE can be interesting as a testbed for improving the NLG components of dialogue systems, and for computational semanticists working on spatial language.

GIVE-1 and GIVE-2

In the GIVE-1 Challenge (2008-09) five NLG systems were evaluated using data from almost 1200 game runs. To our knowledge, this made GIVE-1 the largest ever NLG evaluation effort in terms of the number of experimental subjects. We presented the results of the evaluation at the ENLG Workshop, and have verified that these results are consistent with (but more detailed than) the results that could be obtained from a traditional lab-based evaluation.

In GIVE-2 seven systems were evaluated by more than 1800 users from 39 countries. The results were presented at INLG 2010 in Ireland. The main novelty in GIVE-2 is that where GIVE-1 used discrete worlds (which were based on square tiles, and the user could only jump from the center of one tile to the center of the next, and turn in 90 degree steps), GIVE-2 permits free, continuous movements in the worlds. This makes the generation task more challenging because simple instructions of the form "walk three steps forward" are no longer possible.

Anyone is invited to submit an NLG system to participate in the GIVE-2.5 Challenge; this includes contributions from students and student teams. To get an idea of what this involves, you may want to go to the GIVE website mentioned above and take a look at our EACL 2009 demo paper describing the software architecture, or download the GIVE-2 software and look at it in more detail.

Timeline

GIVE-2.5 will use the same software as GIVE-2. Therefore, development for GIVE-2.5 can begin immediately. The systems have to be ready by the end of May. At the end of May, we will have a one week internal testing period, during which we ensure that all systems satisfy some minimal quality standards. After the internal testing we will distribute the evaluation worlds and the participants will have one more week to respond to our comments and to make final adaptations of their system to the evaluation worlds. This will be followed by the public evaluation until the end of July. The results will be presented as part of Generation Challenges 2011 at ENLG 2011 in Nancy, France, Sep. 28-30, 2011, and non-peer-reviewed papers describing each participating system will be included in the ENLG proceedings. Participants are encouraged to also submit more in depth papers about their approach to ENLG or other peer-reviewed publications through their normal submission procedures.

Important dates for participants

May 23: systems running for internal testing
June 3: notification of acceptance
June 8: final systems running, start of public evaluation
July 1 & Aug. 1: snapshots of evaluation results made available
Sep. 1: camera ready system descriptions due
Sep. 28-30: presentation at Generation Challenges 2011

GIVE 2.5 Organizing committee

Alexandre Denis, Loria
Andrew Gargett, United Arab Emirates University
Konstantina Garoufi, Saarland University
Alexander Koller, University of Potsdam
Kristina Striegnitz, Union College
Mariet Theune, University of Twente

GIVE steering commitee

Donna Byron, Northeastern University
Justine Cassell, Carnegie Mellon University
Robert Dale, Macquarie University
Alexander Koller, University of Potsdam
Johanna Moore, University of Edinburgh
Jon Oberlander, University of Edinburgh
Kristina Striegnitz, Union College

Patent Machine Translation Task (PatentMT) at NTCIR-9

Abbreviated Title: 
PatentMT @ NTCIR-9
ACL-Sponsored Event: 
No
Event Notification Type: 
Call for Participation
Location: 
National Institiute of Informatics (NII)
Event Dates: 
Tuesday, 6 December 2011 to Friday, 9 December 2011
Submission Deadline: 
Thursday, 20 January 2011
Contact: 
Chinese-English: Benjamin K. Tsou (Hong Kong Institute of Education, China)
Japanese-English: Isao Goto (NICT, Japan)
City: 
Tokyo
Country: 
Japan
Description/Details: 

-------------------------------------------------------------------------

Patent Machine Translation Task (PatentMT) at NTCIR-9
Call for Participation

December 6-9, 2011, Tokyo, Japan
http://ntcir.nii.ac.jp/PatentMT/

- A Chinese to English subtask newly added
- Human evaluations will be carried out
- Parallel corpora provided: 1 million Chinese-English and
3 million Japanese-English sentence pairs

-------------------------------------------------------------------------

Interested participants are invited to the Patent Machine Translation
task (PatentMT) at NTCIR-9.

Patents constitute one of the challenging domains for machine
translation because the sentences can be quite long and contain complex
structures. Moreover, there is a significant practical need for machine
translation of patent documents. Let us cultivate this challenging and
significant practical research field with patent machine translation!
Participants can use a large-scale patent parallel corpus for research
and will benefit from reliable human evaluation of their MT quality.

PatentMT, while cast in a framework of friendly competition, has the
ultimate goal to foster scientific cooperation. In this context, the
organizers propose a research task and an open experimental
infrastructure for the scientific community working on machine
translation research.

PatentMT at NTCIR-9 offers three subtasks:
- Chinese to English
- Japanese to English
- English to Japanese
Participants choose the subtasks in which they would like to
participate.

Patent parallel corpora from mostly patent descriptions and patent
monolingual corpora will be provided for training. The training
resources are as follows:
- Chinese to English subtask:
* 1 million Chinese-English parallel sentences
* Large-scale monolingual patent documents in English
- Japanese to English subtask:
* 3 million Japanese-English parallel sentences
* Large-scale monolingual patent documents in English
- English to Japanese subtask:
* 3 million Japanese-English parallel sentences
* Large-scale monolingual patent documents in Japanese
Moreover, blind test sets of patent descriptions will be released.
Participants are requested to machine translate the test sets.
Use of the data will be governed by NTCIR-9 agreement which is under
preparation.

The submitted translation results will be evaluated by human evaluation
and automatic evaluation. The primary evaluation is human evaluation.
So the MT systems can be tested in an all-round and human manner.
Result feedback will be released by the organizers after the evaluation
of the run submissions.

Participants are requested to submit a paper describing the MT system,
the utilized resources, and their results using the provided test data,
and are requested to present their papers at the workshop.

=== Important Dates

- Training data release January 5, 2011
- Task registration due January 20, 2011
- Test data release May 9, 2011
- Translation results submission due May 22, 2011
- Evaluation results release August 19, 2011
- MT system description due September 20, 2011
- Camera-ready due November 4, 2011
- NTCIR-9 workshop December 6-9, 2011

=== Organizers

Chinese-English Side:
- Benjamin K. Tsou (Hong Kong Institute of Education/
City University of Hong Kong, China)
- Kapo Chow (Hong Kong Institute of Education, China)
- Bin Lu (City University of Hong Kong, China)

Japanese-English side:
- Isao Goto (NICT, Japan)
- Eiichiro Sumita (NICT, Japan)

For more information, please visit the NTICR-9 PatentMT web site:
http://ntcir.nii.ac.jp/PatentMT/

-------------------------------------------------------------------------

Pages

Subscribe to RSS - shared task