================= FINAL CALL FOR PAPERS ================
4REAL Workshop 2018
Workshop on Replicability and Reproducibility of Research Results
in Science and Technology of Language
http://4real2018.di.fc.ul.pt
12 May 2018
Miyazaki, Japan
Collocated with LREC 2018
11th Language Resources and Evaluation Conference
http://lrec2018.lrec-conf.org
Important dates:
15 January 2018: Deadline for submissions to 4REAL workshop
9 February 2018: Notification of authors of submissions to 4REAL workshop
1 March 2018: Deadline for camera-ready
9-11 May 2018: LREC main conference (Wednesday to Friday)
12 May 2018: 4REAL workshop (Saturday)
Call for papers:
Reproduction and replication of research results are at the heart of the validation
of scientific knowledge and of the scientific endeavor. Reproduction of results
entails arriving at the same overall conclusions that is, to appropriately validate
a set of results, scientists should strive to reproduce the same answer to a given
research question by different means, e.g. by reimplementing an algorithm or
evaluating it on a new dataset, etc. Replication has a more limited aim, typically
involving running the exact same solution or approach under the same conditions
in order to arrive at the same output result.
Despite their key importance, reproduction and replication have not been sufficiently
encouraged given the prevailing procedures and priorities for the reviewing,
selection and publication of research results.
The immediate motivation for the increased interest on reproducibility and
replicability is to be found in a number of factors, including the realization that
for some published results, their replication is not being obtained (e.g. Prinz et
al., 2011; Belgley and Ellis, 2012); that there may be problems with the commonly
accepted reviewing procedures, where deliberately falsified submissions, with
fabricated errors and fake authors, get accepted even in respectable journals
(e.g. Bohannon, 2013); that the expectation of researchers vis a vis misconduct,
as revealed in inquiries to scientists on questionable practices, scores higher than
one might expect or would be ready to accept (e.g. Fanelli, 2009); among several
others.
This workshop seeks to foster the discussion and the advancement on a topic that has
been given insufficient attention in the research area of language processing tools
and resources and that has been an important topic emerging in other scientific
areas, continuing the objectives of the first edition of the 4REAL workshop, at LREC
2016. We are thus inviting submissions of articles that present cases, either with
positive or negative results, of actual replication or reproduction exercises
of previous published results in our area.
Specific topics within the scope of the call include, but are not limited to:
– Reproduction and replication of previously published systems, resources,
results, etc.
– Analysis of reproducibility challenges in system-oriented and user-oriented
evaluation.
– Reproducibility challenges on private or proprietary data.
– Reproducibility challenges on ephemeral data, like streaming data,
tweets, etc.
– Reproducibility challenges on online experiments.
– Reproducibility in evaluation campaigns.
– Evaluation infrastructures and Evaluation as a Service (EaaS).
– Experiments on data management, data curation, and data quality.
– Reproducible experimental workflows: tools and experiences.
– Data citation: citing experimental data, dynamic data sets, samples, etc.
We are interested also in articles discussing the challenges, the risk factors,
the appropriate procedures, etc. specific to our area or that should be adopted,
or adapted from other neighboring areas, including methodologies for monitoring,
maintaining or improving citation of language resources and tools and to assess
the importance of data citation for research integrity. This includes also of course
the new risks raised by the replication articles themselves and their own integrity,
in view of the preservation of the reputation of colleagues and works whose results
are reported has having been replicated, etc.
“Replicability Focus” in Language Resources and Evaluation Journal:
Best papers will be invited to be submitted to the Language Resources and
Evaluation Journal under its new “Replicability Focus”.
Organization committee:
António Branco (University of Lisbon)
Nicoletta Calzolari (ILC)
Khalid Choukri (ELRA)
Program committee:
Aljoscha Burchardt (DFKI)
António Branco (University of Lisbon)
Gertjan van Noord (University of Groningen)
Joseph Mariani (CNRS/LIMSI)
Khalid Choukri (ELRA)
Maria Gavrilidou (ILSP)
Marko Grobelnik (Jozef Stefan Institute)
Marko Tadic (University of Zagreb)
Nancy Ide (Vassar College)
Nicoletta Calzolari (ILC)
Patrick Paroubek (CNRS/LIMSI))
Piek Vossen (VU University Amsterdam)
Senja Pollak (Jozef Stefan Institute)
Simon Krek (Jozef Stefan Institute)
Stelios Piperidis (ILSP)
Thierry Declerck (DFKI)
Yohei Murakami (Language Grid Japan)
“Identify, Describe and Share your LRs!” initiative:
Describing your Language Resources (LRs) in the LRE Map is now a normal practice
in the submission procedure of LREC (introduced in 2010 and adopted by other
conferences). To continue the efforts initiated at LREC 2014 about “Sharing LRs”
(data, tools, web-services, etc.), when submitting a paper, authors will have
the possibility to upload LRs in a special LREC repository. This effort of
sharing LRs, linked to the LRE Map for their description, may become
a new “regular” feature for conferences in our field, thus contributing to creating
a common repository where everyone can deposit and share data.
As scientific work requires accurate citations of referenced work so as to allow
the community to understand the whole context and also replicate the experiments
conducted by other researchers, LREC 2018 endorses the need to uniquely Identify LRs
through the use of the International Standard Language Resource Number (ISLRN,
www.islrn.org), a Persistent Unique Identifier to be assigned to each Language
Resource. The assignment of ISLRNs to LRs cited in LREC papers will be offered
at submission time.
References:
Begley and Ellis, 2012, Drug development: Raise standards for preclinical
cancer research, Nature
Bohannon, John, 2013, Who’s Afraid of Peer Review?, Science
Fanelli,2009 How Many Scientists Fabricate and Falsify Research?
Prinz, et al., 2011, Believe it or not: how much can we rely on published
data on potential drug targets?, Nature Reviews Drug Discovery 10, 712.