Knmnyn: /* Recommendations for future conferences / committees */

2021-07-11T03:22:47Z

Recommendations for future conferences / committees

← Older revision		Revision as of 03:22, 11 July 2021
Line 88:		Line 88:
	ACs should be flagging papers at the point of assigning reviewers to papers. I.e., the papers to the ethics review committee should be flagged for EAC review at the point of reviewer assignment. This will help address the time pressure for the EAC review process.		ACs should be flagging papers at the point of assigning reviewers to papers. I.e., the papers to the ethics review committee should be flagged for EAC review at the point of reviewer assignment. This will help address the time pressure for the EAC review process.

	'''Standard areas of concern'''. We recommend that future EAC chairs take special attention to specific areas that have outsized exposure to areas that produce more ethical and privacy concerns These include NLP Applications, Language Resources, Computational Social Science, and Generation. This could be reflected in a better balancing between number of papers loaded to reviewers and/or chairs of the EAC when reviewing papers in these special areas.		'''Standard areas of concern'''. We recommend that future EAC chairs take special attention to '''specific areas''' that have outsized exposure to areas that produce more ethical and privacy concerns These include NLP Applications, Language Resources, Computational Social Science, and Generation. This could be reflected in a better balancing between number of papers loaded to reviewers and/or chairs of the EAC when reviewing papers in these special areas.

	In line with the above, there were many "standard" concerns that we addressed by constructing boilerplate review texts that we applied to consistently message common concerns, such as dual use, crowd worker compensation, data privacy, as well as copyright and IP.		In line with the above, there were many '''"standard" concerns''' that we addressed by constructing boilerplate review texts that we applied to consistently message common concerns, such as dual use, crowd worker compensation, data privacy, as well as copyright and IP.

	We recommend that future EAC chairs also categorise such concerns, and prepare common texts, scoping and workflow for handling such cases. We are happy to provide our resources for this to future chairs.		We recommend that future EAC chairs also categorise such concerns, and prepare common texts, scoping and workflow for handling such cases. We are happy to provide our resources for this to future chairs.

Knmnyn at 03:21, 11 July 2021

2021-07-11T03:21:24Z

Show changes

Knmnyn at 03:17, 11 July 2021

2021-07-11T03:17:15Z

← Older revision		Revision as of 03:17, 11 July 2021
Line 1:		Line 1:
	Executive Summary		== Executive Summary ==

	For the ACL-IJCNLP ethics review process, the EAC successfully supported the processing of 247 papers, leading to a total of 28 Conditional Accept recommendations and no rejections from the EAC on ethical grounds. Our process included some effort-reducing modifications to previous practices, including additional EAC triage to reduce reviewing load for papers with only common, remediable ethical issues.		For the ACL-IJCNLP ethics review process, the EAC successfully supported the processing of 247 papers, leading to a total of 28 Conditional Accept recommendations and no rejections from the EAC on ethical grounds. Our process included some effort-reducing modifications to previous practices, including additional EAC triage to reduce reviewing load for papers with only common, remediable ethical issues.

	An imposed compressed timeline, confusion about committee scope, and complexities introduced through the use of Softconf caused several processing issues that we believe could be remedied with restructuring and further discussion at the organizational level about the structure, role, and support needed for the EAC process.		An imposed compressed timeline, confusion about committee scope, and complexities introduced through the use of Softconf caused several processing issues that we believe could be remedied with restructuring and further discussion at the organizational level about the structure, role, and support needed for the EAC process.
	Statistics
			== Statistics ==

	Approximately 247 papers were referred to our committee for review. Of those, 50 were found to have been referred in error (either with no apparent justification or a justification suggesting a non-ethical reason for referral) and thus were not reviewed. 13 were withdrawn prior to the ethics review process being completed on the basis of the technical reviews.		Approximately 247 papers were referred to our committee for review. Of those, 50 were found to have been referred in error (either with no apparent justification or a justification suggesting a non-ethical reason for referral) and thus were not reviewed. 13 were withdrawn prior to the ethics review process being completed on the basis of the technical reviews.

Line 11:		Line 14:

	Importantly, this is not to say that all papers submitted to ACL would have been accepted by our review committee; instead, several papers that would have been rejected by our committee were already rejected in the technical review process due to methodological concerns (including those tied to ethics issues).		Importantly, this is not to say that all papers submitted to ACL would have been accepted by our review committee; instead, several papers that would have been rejected by our committee were already rejected in the technical review process due to methodological concerns (including those tied to ethics issues).
	Ethics Reviewing Timeline
	Pre-review and recruiting (~1 month):		== Ethics Reviewing Timeline ==
	February–March: Recruit reviewers		* Pre-review and recruiting (~1 month):
	March 20: Technical review deadline		** February–March: Recruit reviewers
	Reviewing (~1 month):		** March 20: Technical review deadline
	March 25: Received ethics review recommendations (start of author response period)		* Reviewing (~1 month):
	March 29: Send ethics review assignments to reviewers		** March 25: Received ethics review recommendations (start of author response period)
	March 31: Author response due		** March 29: Send ethics review assignments to reviewers
	April 14: AC meta reviews due		** March 31: Author response due
	April 15: Ethics reviews due		** April 14: AC meta reviews due
	Discussion (3 weeks):		** April 15: Ethics reviews due
	April 16-22: Ethics review discussion		* Discussion (3 weeks):
	April 23: Received list of tentative paper decisions from technical review		** April 16-22: Ethics review discussion
	April 26: Returned ethics decisions for papers slated by SAC to be accepted by technical review		** April 23: Received list of tentative paper decisions from technical review
	April 29-May 4: Ethics metareview entry and resolution for papers whose decision changed from SAC recommendation/papers with late SAC recommendations		** April 26: Returned ethics decisions for papers slated by SAC to be accepted by technical review
	Decisions:		** April 29-May 4: Ethics metareview entry and resolution for papers whose decision changed from SAC recommendation/papers with late SAC recommendations
	May 5: Decisions sent out by PCs, along with instructions for conditional acceptance from EAC		* Decisions:
	Conditional Accept Processing (~1 month)		** May 5: Decisions sent out by PCs, along with instructions for conditional acceptance from EAC
	May 5–June 1: Correspondence with conditional acceptance authors to help with revision		* Conditional Accept Processing (~1 month)
	June 1: Camera ready deadline		** May 5–June 1: Correspondence with conditional acceptance authors to help with revision
	June 7: Final conditional acceptance decision given		** June 1: Camera ready deadline
	Pain Points		** June 7: Final conditional acceptance decision given
	SoftConf. The current structure of SoftConf is poorly equipped to handle conducting ethics reviews on papers. In particular, our reviewers needed to be able to write reviews out of cycle with the other paper reviewers and with a different review form, which prohibited us from having our reviewer invitations overlap with the regular invitation period and complicated the review assignment and deadlines. Through the shared system, the only option to assemble a form for our reviewers was to “modify” the technical review form to repurpose some fields of the original review form to create the new form. As the progression for technical review continued, we would often also find information in our “area” incorrect, previously accessible information rendered no longer viewable after a technical review deadline passed, or attachments and other necessary information blocked from view while we conducted review. Our final system heavily leveraged additional manually-updated spreadsheets instead of direct SoftConf integration, which led to multiple cases of confusion when spreadsheets fell out of sync with SoftConf or when there was no SoftConf-amenable way to confirm a task was completed for all relevant papers.
			== Pain Points ==

			SoftConf. The current structure of SoftConf is poorly equipped to handle conducting ethics reviews on papers. In particular, our reviewers needed to be able to write reviews out of cycle with the other paper reviewers and with a different review form, which prohibited us from having our reviewer invitations overlap with the regular invitation period and complicated the review assignment and deadlines. Through the shared system, the only option to assemble a form for our reviewers was to “modify” the technical review form to repurpose some fields of the original review form to create the new form. As the progression for technical review continued, we would often also find information in our “area” incorrect, previously accessible information rendered no longer viewable after a technical review deadline passed, or attachments and other necessary information blocked from view while we conducted review. Our final system heavily leveraged additional manually-updated spreadsheets instead of direct SoftConf integration, which led to multiple cases of confusion when spreadsheets fell out of sync with SoftConf or when there was no SoftConf-amenable way to confirm a task was completed for all relevant papers.

	Timeline. The timeline for the Ethics Advisory Committee had to be built around a contracted timeline for the original conference review process, and suffered for not having been included in the thought process for the original schedule. Almost every step of the process was rushed as a result, from the early publication of the guidelines and structure for ethics review through to the final decisions. This included the modification of the original schedule, in which ACs would help identify some papers for ethics review before reviews were turned in, to only have a single round of review starting after technical reviews were due. Timing issues were further exacerbated by issues with Softconf’s incompatibility, which often required manual intervention and multiple day delays in order to access information needed for tight decision turnaround times. While we are confident in the committee’s work, we regret that so much had to be rushed.		Timeline. The timeline for the Ethics Advisory Committee had to be built around a contracted timeline for the original conference review process, and suffered for not having been included in the thought process for the original schedule. Almost every step of the process was rushed as a result, from the early publication of the guidelines and structure for ethics review through to the final decisions. This included the modification of the original schedule, in which ACs would help identify some papers for ethics review before reviews were turned in, to only have a single round of review starting after technical reviews were due. Timing issues were further exacerbated by issues with Softconf’s incompatibility, which often required manual intervention and multiple day delays in order to access information needed for tight decision turnaround times. While we are confident in the committee’s work, we regret that so much had to be rushed.

Knmnyn at 02:20, 11 July 2021

2021-07-11T02:20:38Z

Show changes

Knmnyn: Created page with "(This report is being drafted)"

2021-06-18T09:36:05Z

Created page with "(This report is being drafted)"

New page

(This report is being drafted)

2021Q3 Reports: Ethic Advisory Committee (EAC) - Revision history

Knmnyn: /* Recommendations for future conferences / committees */

Knmnyn at 03:21, 11 July 2021

Knmnyn at 03:17, 11 July 2021

Knmnyn at 02:20, 11 July 2021

Knmnyn: Created page with "(This report is being drafted)"