Text Analysis Conference -- Knowledge Base Population 2016

Event Notification Type: 
Call for Participation
Abbreviated Title: 
Location: 
Monday, 14 November 2016 to Wednesday, 16 November 2016
State: 
Maryland
Country: 
USA
Contact Email: 
City: 
Gaithersburg
Contact: 
Submission Deadline: 
Friday, 15 July 2016

Text Analysis Conference
Knowledge Base Population 2016

Evaluation: February-November, 2016
Workshop: November 14-15, 2016

http://www.nist.gov/tac/2016/KBP/

Conducted by:
U.S. National Institute of Standards and Technology (NIST)

With support from:
U.S. Department of Defense

INTRODUCTION

The Text Analysis Conference (TAC) is a series of shared tasks, evaluations and workshops organized to promote research in Natural Language Processing and related applications, by providing a large test collection, common evaluation procedures, and a forum for organizations to share their results. The goal of TAC Knowledge Base Population (KBP) is to develop and evaluate technologies for populating structured knowledge bases (KBs) from unstructured text.

You are invited to participate in TAC KBP 2016. Organizations may choose to participate in any or all of the TAC KBP 2016 tracks. NIST provides test data for each KBP task, and participants run their NLP systems on the data and return their results to NIST for evaluation. TAC KBP culminates in a November workshop at NIST in Gaithersburg, Maryland, USA.

All results submitted to NIST are archived on the TAC web site, and all evaluations of submitted results are included in the workshop proceedings. Dissemination of TAC work and results other than in the workshop proceedings is welcomed, but the conditions of participation specifically preclude any advertising claims based on TAC results.

TRACKS

1) Cold Start KBP
* http://www.nist.gov/tac/2016/KBP/ColdStart/
The Cold Start KBP track builds a knowledge base from scratch using a given document collection and a predefined schema for the entities and relations that will comprise the KB. In addition to an end-to-end KB Construction task, Cold Start KBP includes a Slot Filling (SF) task to fill in values for predefined slots (attributes) for a given query entity.

2) Validation/Ensembling
* http://www.nist.gov/tac/2016/KBP/SFValidation/
The Validation/Ensembling track focuses on the refinement of output from slot filling systems by either combining information from multiple slot filling systems, or applying more intensive linguistic processing to validate individual candidate slot fillers.

3) Entity Discovery and Linking
* http://nlp.cs.rpi.edu/kbp/2016/
The Entity Discovery and Linking (EDL) track aims to extract entity mentions from a source collection of textual documents, and link them to an existing reference Knowledge Base (KB); an EDL system is also required to cluster mentions for those entities that don't have corresponding entries in the reference KB.

4) Event
* http://www.nist.gov/tac/2016/KBP/Event/
The goal of the Event track is to extract information about events such that the information would be suitable as input to a knowledge base. The track includes Event Nugget (EN) tasks to detect and link events, and Event Argument (EA) tasks to extract event arguments and link arguments that belong to the same event.

5) Belief and Sentiment
* http://www.cs.cornell.edu/home/cardie/best-eval-2016/
The Belief and Sentiment track detects belief and sentiment of an entity toward another entity, relation, or event.

WHAT'S NEW

1) All tasks operate on the same combined English, Chinese, and Spanish source document collection; all tracks also allow participation in a single language.
2) Entity Discovery and Linking operates at a larger scale (90K documents) and extends nominal mentions to all entity types.
3) New Belief and Sentiment track on detecting belief (also known as factuality) and sentiment from sources to targets.

REGISTRATION

Organizations wishing to participate in any of the TAC KBP 2016 tracks are invited to register online by July 15, 2016. Participants are advised to register and submit all required agreement forms as soon as possible in order to receive timely access to evaluation resources, including any sample and training data. Registration for a track does not commit you to participating in the track, but is helpful to know for planning. Late registration will be permitted only if resources allow. Any questions about conference participation may be sent to the TAC project manager: tac-manager [at] nist.gov.

Track registration: http://www.nist.gov/tac/2016/KBP/registration.html

WORKSHOP

The TAC 2016 workshop will be held November 14-15, 2016, in Gaithersburg, Maryland, USA. The workshop is a forum both for presentation of results (including failure analyses and system comparisons), and for more lengthy system presentations describing techniques used, experiments run on the data, and other issues of interest to NLP researchers. KBP track participants who wish to give a presentation during the workshop will submit a short abstract in October describing the experiments they performed. As there is a limited amount of time for oral presentations, the abstracts will be used to determine which participants are asked to speak and which will present in a poster session.

PRELIMINARY SCHEDULE

July 15 Deadline for registration for track participation (late registration by permission only)
August-October Track evaluation windows (varies by track)
October 10 Deadline for short system descriptions
October 10 Deadline for workshop presentation proposals
By mid October Release of individual evaluated results to participants (varies by track)
October 20 Notification of acceptance of presentation proposals
November 1 Deadline for system reports (workshop notebook version)
November 14-15 TAC 2016 workshop in Gaithersburg, Maryland, USA
February 2017 Deadline for system reports (final proceedings version)

ORGANIZING COMMITTEE

Hoa Trang Dang (U.S. National Institute of Standards and Techonology)
Jason Duncan (MITRE)
Joe Ellis (Linguistic Data Consortium)
Marjorie Freedman (BBN Technologies)
Ralph Grishman (New York University)
Eduard Hovy (Carnegie Mellon University)
Heng Ji (Rensselaer Polytechnic Institute)
James Mayfield (Johns Hopkins University)
Teruko Mitamura (Carnegie Mellon University)
Boyan Onyshkevych (U.S. Department of Defense)
Shahzad Rajput (U.S. National Institute of Standards and Techonology)
Owen Rambow (Columbia University)
Zhiyi Song (Linguistic Data Consortium)
Stephanie Strassel (Linguistic Data Consortium)