Review Data Collection at *ACL - Revision history

Iryna Gurevych at 11:09, 25 August 2022

2022-08-25T11:09:09Z

← Older revision		Revision as of 11:09, 25 August 2022
Line 18:		Line 18:

	Prof. Dr. Iryna Gurevych, [mailto:gurevych@ukp.informatik.tu-darmstadt.de gurevych@ukp.informatik.tu-darmstadt.de] <br />		Prof. Dr. Iryna Gurevych, [mailto:gurevych@ukp.informatik.tu-darmstadt.de gurevych@ukp.informatik.tu-darmstadt.de] <br />
	Dr. Ilia Kuznetsov, [mailto:kuznetsov@ukp.informatik.tu-darmstadt.de kuznetsov@ukp.informatik.tu-darmstadt.de]		Dr. Ilia Kuznetsov, PhD [mailto:kuznetsov@ukp.informatik.tu-darmstadt.de kuznetsov@ukp.informatik.tu-darmstadt.de]

Iryna Gurevych at 11:08, 25 August 2022

2022-08-25T11:08:51Z

← Older revision		Revision as of 11:08, 25 August 2022
Line 18:		Line 18:

	Prof. Dr. Iryna Gurevych, [mailto:gurevych@ukp.informatik.tu-darmstadt.de gurevych@ukp.informatik.tu-darmstadt.de] <br />		Prof. Dr. Iryna Gurevych, [mailto:gurevych@ukp.informatik.tu-darmstadt.de gurevych@ukp.informatik.tu-darmstadt.de] <br />
	Ilia Kuznetsov, [mailto:kuznetsov@ukp.informatik.tu-darmstadt.de kuznetsov@ukp.informatik.tu-darmstadt.de]		Dr. Ilia Kuznetsov, [mailto:kuznetsov@ukp.informatik.tu-darmstadt.de kuznetsov@ukp.informatik.tu-darmstadt.de]

Iryna Gurevych at 11:07, 25 August 2022

2022-08-25T11:07:57Z

← Older revision		Revision as of 11:07, 25 August 2022
Line 1:		Line 1:
	''This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL.''		''This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL.''
	''The documents ~~has~~ been prepared by Iryna Gurevych and Ilia Kuznetsov as a result of discussions with the ACL Reviewing committee and ACL conference officer Yusuke Miyao (conceptualization), program chairs and community of COLING-2020 (conceptualization, pilot, community feedback), as well as legal counseling by Dorothy Deng (license agreement and consent).''		''The documents have been prepared by Iryna Gurevych and Ilia Kuznetsov as a result of discussions with the ACL Reviewing committee and ACL conference officer Yusuke Miyao (conceptualization), program chairs and community of COLING-2020 (conceptualization, pilot, community feedback), as well as legal counseling by Dorothy Deng (license agreement and consent).''

	=== Updates ===		=== Updates ===

Iryna Gurevych at 10:47, 5 April 2022

2022-04-05T10:47:08Z

← Older revision		Revision as of 10:47, 5 April 2022
Line 1:		Line 1:
	''This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL.''		''This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL.''
	''The documents has been prepared by Iryna Gurevych and Ilia Kuznetsov as a result of discussions with the ACL Reviewing committee and ACL conference officer Yusuke Miyao (conceptualization), program chairs and community of COLING-2020 (conceptualization, pilot, community feedback), as well as legal counseling by Dorothy Deng (license agreement and consent).''		''The documents has been prepared by Iryna Gurevych and Ilia Kuznetsov as a result of discussions with the ACL Reviewing committee and ACL conference officer Yusuke Miyao (conceptualization), program chairs and community of COLING-2020 (conceptualization, pilot, community feedback), as well as legal counseling by Dorothy Deng (license agreement and consent).''

			=== Updates ===
			* March 2022: [[Media:Review_Data_Collection_ 2022_ Report_Proposal.pdf\|Progress report and amendments]] to the original proposal in terms of data collection, access and storage.

	=== Summary ===		=== Summary ===

Iryna Gurevych at 10:11, 30 March 2021

2021-03-30T10:11:24Z

← Older revision		Revision as of 10:11, 30 March 2021
Line 6:		Line 6:

	=== Documents ===		=== Documents ===
	# The [[Media:1._Review_collection_at_ACL_(slides).pdf\| slide deck]] briefly summarizes the two proposals;		# The [[Media:1._Review_collection_at_ACL_(slides).pdf\|slide deck]] briefly summarizes the two proposals;
	# The [[Media:2._Review_collection_at_ACL.pdf\| full document]] elaborates on the proposals in further detail. In addition, we provide		# The [[Media:2._Review_collection_at_ACL.pdf\|full document]] elaborates on the proposals in further detail. In addition, we provide
	# the example License agreements for [[Media:3._ACL_License_Blind_Submission_v1.pdf\| ~~peer reviewing~~ data]]		# the example License agreements for [[Media:3._ACL_License_Blind_Submission_v1.pdf\|blind submission data]]
	# and [[Media:4._ACL_License_Peer_Reviews_v2.pdf\| ~~blind submission~~ data]].		# and [[Media:4._ACL_License_Peer_Reviews_v2.pdf\|peer reviewing data]].

	=== Contact ===		=== Contact ===

Iryna Gurevych at 10:09, 30 March 2021

2021-03-30T10:09:17Z

← Older revision		Revision as of 10:09, 30 March 2021
Line 6:		Line 6:

	=== Documents ===		=== Documents ===
	# The [[~~:File~~:1._Review_collection_at_ACL_(slides).pdf\| slide deck]] briefly summarizes the two proposals;		# The [[Media:1._Review_collection_at_ACL_(slides).pdf\| slide deck]] briefly summarizes the two proposals;
	# The [[~~:File~~: 2._Review_collection_at_ACL.pdf\| full document]] elaborates on the proposals in further detail. In addition, we provide		# The [[Media:2._Review_collection_at_ACL.pdf\| full document]] elaborates on the proposals in further detail. In addition, we provide
	# the example License agreements for [[~~:File~~: 3._ACL_License_Blind_Submission_v1.pdf\| peer reviewing data]]		# the example License agreements for [[Media:3._ACL_License_Blind_Submission_v1.pdf\| peer reviewing data]]
	# and [[:~~File: 3~~.~~_ACL_License_Blind_Submission_v1~~.pdf\| blind submission data]].		# and [[Media:4._ACL_License_Peer_Reviews_v2.pdf\| blind submission data]].

	=== Contact ===		=== Contact ===

Iryna Gurevych at 09:47, 30 March 2021

2021-03-30T09:47:39Z

← Older revision		Revision as of 09:47, 30 March 2021
Line 1:		Line 1:
	''This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL.''		''This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL.''
	''The documents has been prepared by Iryna Gurevych and Ilia Kuznetsov as a result of discussions with the ACL Reviewing committee and ACL conference officer Yusuke Miyao (conceptualization), program chairs and community of COLING-2020 (conceptualization, pilot, community feedback), as well as legal counseling by Dorothy Deng (license agreement and consent).''		''The documents has been prepared by Iryna Gurevych and Ilia Kuznetsov as a result of discussions with the ACL Reviewing committee and ACL conference officer Yusuke Miyao (conceptualization), program chairs and community of COLING-2020 (conceptualization, pilot, community feedback), as well as legal counseling by Dorothy Deng (license agreement and consent).''


	=== Summary ===		=== Summary ===
	We propose two initiatives aiming at facilitating ethical, consent-driven research in NLP for peer reviewing, and peer reviewing for NLP. The first initiative '''concerns the collection of peer review and manuscript data from ACL events'''. We cover the major challenges in this process and propose solutions, as well as report on a pilot study conducted at COLING-2020, where some of those solutions were implemented. The second initiative aims to '''provide a structured way to study and compare peer reviewing workflows''' in the ACL community. For this we propose to create a centralized repository of anonymized numerical data from ACL conferences. This repository can serve as an anchor point for future conference organizers and provide a bridge to the meta-science and science-of-science communities that have been working on similar issues in other disciplines and research fields.		We propose two initiatives aiming at facilitating ethical, consent-driven research in NLP for peer reviewing, and peer reviewing for NLP. The first initiative '''concerns the collection of peer review and manuscript data from ACL events'''. We cover the major challenges in this process and propose solutions, as well as report on a pilot study conducted at COLING-2020, where some of those solutions were implemented. The second initiative aims to '''provide a structured way to study and compare peer reviewing workflows''' in the ACL community. For this we propose to create a centralized repository of anonymized numerical data from ACL conferences. This repository can serve as an anchor point for future conference organizers and provide a bridge to the meta-science and science-of-science communities that have been working on similar issues in other disciplines and research fields.

	=== ~~Package contents~~ ===		=== Documents ===
	# The slide deck briefly summarizes the two proposals;		# The [[:File:1._Review_collection_at_ACL_(slides).pdf\| slide deck]] briefly summarizes the two proposals;
	# The full document elaborates on the proposals in further detail. In addition, we provide		# The [[:File: 2._Review_collection_at_ACL.pdf\| full document]] elaborates on the proposals in further detail. In addition, we provide
	# ~~The~~ example License agreements for peer reviewing data		# the example License agreements for [[:File: 3._ACL_License_Blind_Submission_v1.pdf\| peer reviewing data]]
	# and blind submission data		# and [[:File: 3._ACL_License_Blind_Submission_v1.pdf\| blind submission data]].

	=== Contact ===		=== Contact ===

Iryna Gurevych: Created page with "''This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL.'' ''The documents has been prepared by Iryna Gurevych and Ilia..."

2021-03-30T09:33:49Z

Created page with "''This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL.'' ''The documents has been prepared by Iryna Gurevych and Ilia..."

New page

''This wiki page contains several documents related to our proposals on peer reviewing data collection at *ACL.''
''The documents has been prepared by Iryna Gurevych and Ilia Kuznetsov as a result of discussions with the ACL Reviewing committee and ACL conference officer Yusuke Miyao (conceptualization), program chairs and community of COLING-2020 (conceptualization, pilot, community feedback), as well as legal counseling by Dorothy Deng (license agreement and consent).''

=== Summary ===
We propose two initiatives aiming at facilitating ethical, consent-driven research in NLP for peer reviewing, and peer reviewing for NLP. The first initiative '''concerns the collection of peer review and manuscript data from *ACL events'''. We cover the major challenges in this process and propose solutions, as well as report on a pilot study conducted at COLING-2020, where some of those solutions were implemented. The second initiative aims to '''provide a structured way to study and compare peer reviewing workflows''' in the *ACL community. For this we propose to create a centralized repository of anonymized numerical data from ACL conferences. This repository can serve as an anchor point for future conference organizers and provide a bridge to the meta-science and science-of-science communities that have been working on similar issues in other disciplines and research fields.

=== Package contents ===
# The slide deck briefly summarizes the two proposals;
# The full document elaborates on the proposals in further detail. In addition, we provide
# The example License agreements for peer reviewing data
# and blind submission data

=== Contact ===
If you have commentaries, questions or suggestions regarding the proposal, do not hesitate to contact:

Prof. Dr. Iryna Gurevych, [mailto:gurevych@ukp.informatik.tu-darmstadt.de gurevych@ukp.informatik.tu-darmstadt.de] <br />
Ilia Kuznetsov, [mailto:kuznetsov@ukp.informatik.tu-darmstadt.de kuznetsov@ukp.informatik.tu-darmstadt.de]