ACL Rolling Review Proposal - Revision history

ShiqiZhao at 16:06, 19 June 2020

2020-06-19T16:06:28Z

← Older revision		Revision as of 16:06, 19 June 2020
Line 109:		Line 109:
	=== Acknowledgements ===		=== Acknowledgements ===

	This document was drafted by Graham Neubig and revised by the ACL Reviewing Committee: Amanda Stent, Ani Nenkova, Anna Korhonen, Barbara Di Eugenio, Hinrich Schuetze, Iryna Gurevych, Joel Tetreault, Marti Hearst, Matt Gardner, Nitin Madnani, Tim ~~Baldiwn~~, Trevor Cohn, Yang Liu.		This document was drafted by Graham Neubig and revised by the ACL Reviewing Committee: Amanda Stent, Ani Nenkova, Anna Korhonen, Barbara Di Eugenio, Hinrich Schuetze, Iryna Gurevych, Joel Tetreault, Marti Hearst, Matt Gardner, Nitin Madnani, Tim Baldwin, Trevor Cohn, Yang Liu.

	Thank you to the ACL Executive Committee for additional feedback.		Thank you to the ACL Executive Committee for additional feedback.

HinrichSchuetze at 06:32, 19 June 2020

2020-06-19T06:32:10Z

← Older revision		Revision as of 06:32, 19 June 2020
Line 5:		Line 5:
	https://forms.office.com/Pages/ResponsePage.aspx?id=9028kaqAQ0OMdrEjlJf7WQiNRJRoOx9OlzQS6C5hck5URVc2MDZPRFBVNDRRRjBaMjBQVk41RVpMOC4u		https://forms.office.com/Pages/ResponsePage.aspx?id=9028kaqAQ0OMdrEjlJf7WQiNRJRoOx9OlzQS6C5hck5URVc2MDZPRFBVNDRRRjBaMjBQVk41RVpMOC4u

	The ACL reviewing committee has been working on improvements to the reviewing process, has recently introduced short term changes [https://www.aclweb.org/adminwiki/index.php?title=Short-Term_Reform_Proposals_for_ACL_Reviewing]a. What follows below is a proposal for a more long-term transition to a new system for review in *ACL conferences, where reviewing and acceptance of papers to publication venues is done in a two-step process:		The ACL reviewing committee has been working on improvements to the reviewing process and has recently introduced short term changes [https://www.aclweb.org/adminwiki/index.php?title=Short-Term_Reform_Proposals_for_ACL_Reviewing]. What follows below is a proposal for a more long-term transition to a new system for review in *ACL conferences, where reviewing and acceptance of papers to publication venues is done in a two-step process:

	*'''Step 1 -- Centralized Rolling Review:''' Authors submit papers to a unified review pool with monthly deadlines (similarly to TACL). Review is handled by an action editor, and revision and resubmission of papers is allowed.		*'''Step 1 -- Centralized Rolling Review:''' Authors submit papers to a unified review pool with monthly deadlines (similarly to TACL). Review is handled by an action editor, and revision and resubmission of papers is allowed.

HinrichSchuetze at 06:30, 19 June 2020

2020-06-19T06:30:49Z

← Older revision		Revision as of 06:30, 19 June 2020
Line 5:		Line 5:
	https://forms.office.com/Pages/ResponsePage.aspx?id=9028kaqAQ0OMdrEjlJf7WQiNRJRoOx9OlzQS6C5hck5URVc2MDZPRFBVNDRRRjBaMjBQVk41RVpMOC4u		https://forms.office.com/Pages/ResponsePage.aspx?id=9028kaqAQ0OMdrEjlJf7WQiNRJRoOx9OlzQS6C5hck5URVc2MDZPRFBVNDRRRjBaMjBQVk41RVpMOC4u

	The ACL reviewing committee has been working on improvements to the reviewing process, has recently introduced short term changes [~~add link here~~]. What follows below is a proposal for a more long-term transition to a new system for review in *ACL conferences, where reviewing and acceptance of papers to publication venues is done in a two-step process:		The ACL reviewing committee has been working on improvements to the reviewing process, has recently introduced short term changes [https://www.aclweb.org/adminwiki/index.php?title=Short-Term_Reform_Proposals_for_ACL_Reviewing]a. What follows below is a proposal for a more long-term transition to a new system for review in *ACL conferences, where reviewing and acceptance of papers to publication venues is done in a two-step process:

	*'''Step 1 -- Centralized Rolling Review:''' Authors submit papers to a unified review pool with monthly deadlines (similarly to TACL). Review is handled by an action editor, and revision and resubmission of papers is allowed.		*'''Step 1 -- Centralized Rolling Review:''' Authors submit papers to a unified review pool with monthly deadlines (similarly to TACL). Review is handled by an action editor, and revision and resubmission of papers is allowed.

HinrichSchuetze at 06:26, 19 June 2020

2020-06-19T06:26:44Z

← Older revision		Revision as of 06:26, 19 June 2020
Line 1:		Line 1:
			Published on June 19, 2020

			'''Note: We are soliciting feedback from the ACL community on the short- and long-term reviewing proposals. If you would like to provide feedback, please do so here:'''

			https://forms.office.com/Pages/ResponsePage.aspx?id=9028kaqAQ0OMdrEjlJf7WQiNRJRoOx9OlzQS6C5hck5URVc2MDZPRFBVNDRRRjBaMjBQVk41RVpMOC4u

	The ACL reviewing committee has been working on improvements to the reviewing process, has recently introduced short term changes [add link here]. What follows below is a proposal for a more long-term transition to a new system for review in *ACL conferences, where reviewing and acceptance of papers to publication venues is done in a two-step process:		The ACL reviewing committee has been working on improvements to the reviewing process, has recently introduced short term changes [add link here]. What follows below is a proposal for a more long-term transition to a new system for review in *ACL conferences, where reviewing and acceptance of papers to publication venues is done in a two-step process:

HinrichSchuetze at 06:21, 19 June 2020

2020-06-19T06:21:21Z

← Older revision		Revision as of 06:21, 19 June 2020
Line 1:		Line 1:
	~~These short-term proposals were adopted by the ACL Exec on June 8 as an initial step to improve~~
	~~reviewing. As we work towards a more comprehensive reform, we welcome feedback on all the aspects~~
	~~of our current reviewing system (including the short-term proposals). Feedback from our community will be~~
	~~valuable in informing further development of the system.~~

	The ACL reviewing committee has been working on improvements to the reviewing process, has recently introduced short term changes [add link here]. What follows below is a proposal for a more long-term transition to a new system for review in *ACL conferences, where reviewing and acceptance of papers to publication venues is done in a two-step process:		The ACL reviewing committee has been working on improvements to the reviewing process, has recently introduced short term changes [add link here]. What follows below is a proposal for a more long-term transition to a new system for review in *ACL conferences, where reviewing and acceptance of papers to publication venues is done in a two-step process:

HinrichSchuetze at 06:20, 19 June 2020

2020-06-19T06:20:55Z

← Older revision		Revision as of 06:20, 19 June 2020
Line 1:		Line 1:
	By the ACL ~~Reviewing Committee (~~June 12, ~~2020~~)		These short-term proposals were adopted by the ACL Exec on June 8 as an initial step to improve
			reviewing. As we work towards a more comprehensive reform, we welcome feedback on all the aspects
			of our current reviewing system (including the short-term proposals). Feedback from our community will be
			valuable in informing further development of the system.

	The ACL reviewing committee has been working on improvements to the reviewing process, has recently introduced short term changes [add link here]. What follows below is a proposal for a more long-term transition to a new system for review in *ACL conferences, where reviewing and acceptance of papers to publication venues is done in a two-step process:		The ACL reviewing committee has been working on improvements to the reviewing process, has recently introduced short term changes [add link here]. What follows below is a proposal for a more long-term transition to a new system for review in *ACL conferences, where reviewing and acceptance of papers to publication venues is done in a two-step process:

ShiqiZhao: /* Acknowledgements */

2020-06-18T01:38:54Z

Acknowledgements

← Older revision		Revision as of 01:38, 18 June 2020
Line 106:		Line 106:
	=== Acknowledgements ===		=== Acknowledgements ===

	This document was drafted by Graham Neubig and revised by the ACL Reviewing Committee: Amanda Stent, Ani Nenkova, Anna Korhonen, Barbara Di Eugenio, Hinrich ~~Schutze~~, Iryna Gurevych, Joel Tetreault, Marti Hearst, Matt Gardner, Nitin Madnani, Tim Baldiwn, Trevor Cohn, Yang Liu.		This document was drafted by Graham Neubig and revised by the ACL Reviewing Committee: Amanda Stent, Ani Nenkova, Anna Korhonen, Barbara Di Eugenio, Hinrich Schuetze, Iryna Gurevych, Joel Tetreault, Marti Hearst, Matt Gardner, Nitin Madnani, Tim Baldiwn, Trevor Cohn, Yang Liu.

	Thank you to the ACL Executive Committee for additional feedback.		Thank you to the ACL Executive Committee for additional feedback.

ShiqiZhao at 15:57, 17 June 2020

2020-06-17T15:57:32Z

@@ Line 95: / Line 95: @@
 *'''Papers being repeatedly rejected by conferences:''' What if in a "pick-from-the-pool" system, program chairs would have more leeway not picking that paper -- after all, they are not rejecting the paper, they are just making it available to other conferences/journals?
 **''Answer:'' This is also the case in the current system (often papers are rejected because they're "not ready yet"), but there is some chance that the system proposed here may encourage this further. One potential fix would be to encourage PCs to not pass over highly-rated papers too many times unless there is a really good reason. Because most PCs believe in the values of fairness, this could potentially favor highly-rated papers that have been passed over in the past.
-  For papers that are not so highly rated, the authors have several choices (analogous to the choices they have in the current system): (a) revise the paper in an attempt to improve the review ratings, (b) lower their standards for where they present their papers, such as submitting to a workshop instead of a conference, (c) if we implement a centralized public repository of under-review or deblinded papers such as the “Findings of the EMNLP” or “ACL Archive”, allow the paper to remain there without being presented at any conference.
+For papers that are not so highly rated, the authors have several choices (analogous to the choices they have in the current system): (a) revise the paper in an attempt to improve the review ratings, (b) lower their standards for where they present their papers, such as submitting to a workshop instead of a conference, (c) if we implement a centralized public repository of under-review or deblinded papers such as the “Findings of the EMNLP” or “ACL Archive”, allow the paper to remain there without being presented at any conference.

ShiqiZhao at 15:52, 17 June 2020

2020-06-17T15:52:59Z

← Older revision		Revision as of 15:52, 17 June 2020
Line 86:		Line 86:
	*'''Accepted Papers / Some Fraction of Papers in the Review System:''' We could take a middle-ground strategy between the two. For example, upon submission have authors select a preferred venue (ACL/EMNLP/NAACL), or preferred type of venue (Conference/Journal/Workshop). We could then use the fraction of the papers that selected a particular venue as preferred as an indication of that venue’s popularity, and calculate the denominator based on these statistics. Alternatively, we could use all papers in the system that selected a particular type of venue, e.g. “Conference”, in the denominator when calculating conference acceptance rates (and similarly for journals).		*'''Accepted Papers / Some Fraction of Papers in the Review System:''' We could take a middle-ground strategy between the two. For example, upon submission have authors select a preferred venue (ACL/EMNLP/NAACL), or preferred type of venue (Conference/Journal/Workshop). We could then use the fraction of the papers that selected a particular venue as preferred as an indication of that venue’s popularity, and calculate the denominator based on these statistics. Alternatively, we could use all papers in the system that selected a particular type of venue, e.g. “Conference”, in the denominator when calculating conference acceptance rates (and similarly for journals).
	*'''Compare Accepted Paper Scores to Full Submission Pool Scores:''' As another option, we could keep track of the overall distribution of numerical review scores in the system and the distribution for each conference. Then, we calculate an estimation of the review score necessary to be accepted at the conference, and find its percentile in the overall review pool. To take a made-up example, let's say that the median overall review score of papers accepted to ACL was 4.1, then and only 16% of papers in the overall review pool had scores 4.1 or higher. In this case, the acceptance rate for ACL would be reported as 16%. Instead of the median for each publication venue, we could also take the 10th or 20th percentile, representing the lower end of the venue’s accepted range.		*'''Compare Accepted Paper Scores to Full Submission Pool Scores:''' As another option, we could keep track of the overall distribution of numerical review scores in the system and the distribution for each conference. Then, we calculate an estimation of the review score necessary to be accepted at the conference, and find its percentile in the overall review pool. To take a made-up example, let's say that the median overall review score of papers accepted to ACL was 4.1, then and only 16% of papers in the overall review pool had scores 4.1 or higher. In this case, the acceptance rate for ACL would be reported as 16%. Instead of the median for each publication venue, we could also take the 10th or 20th percentile, representing the lower end of the venue’s accepted range.

			=== Other Fundamental Comments/Concerns ===

			*'''How to avoid reviewer burnout?:''' Some people don’t want to be reviewing constantly, all the time, how do we prevent this in a rolling review system?
			**''Answer:'' As noted above, we can have a centralized system that keeps track of “review balance” how much reviewing people have done and tries to distribute it equitably. In addition, some people prefer to review many papers at a time as opposed to a few every month. Thus, at the beginning of the monthly reviewing cycle, we could give people a chance to bid on papers that they would like to handle, and if they want to do a lot of reviewing/chairing that month so they don’t have to do much other months, they can bid to take several papers, increasing their review balance so they don’t have to do reviewing in succeeding months. In addition, we may add the ability of reviewers to indicate that they are on vacation or otherwise unavailable.
			*'''Would the “revise and resubmit” option favor boring work or lengthen the review process?:''' There are concerns that a “revise and resubmit” option favors more conservative work, particularly if the reviewers are the same through multiple rounds of review. In addition, review can take longer.
			**''Answer:'' Regarding lengthening the review process, the authors are given an option to submit their work for presentation as soon as they like, so they can conclude the review process at a point where they choose once they have received initial reviews and the paper has been judged compliant. In addition, the authors can appeal for a different set of reviewers if they feel that their reviews were unfair or of poor quality. A central review process would also allow us to collect statistics and adjust as we learn more about the trends.
			*'''Papers being repeatedly rejected by conferences:''' What if in a "pick-from-the-pool" system, program chairs would have more leeway not picking that paper -- after all, they are not rejecting the paper, they are just making it available to other conferences/journals?
			**''Answer:'' This is also the case in the current system (often papers are rejected because they're "not ready yet"), but there is some chance that the system proposed here may encourage this further. One potential fix would be to encourage PCs to not pass over highly-rated papers too many times unless there is a really good reason. Because most PCs believe in the values of fairness, this could potentially favor highly-rated papers that have been passed over in the past.
			For papers that are not so highly rated, the authors have several choices (analogous to the choices they have in the current system): (a) revise the paper in an attempt to improve the review ratings, (b) lower their standards for where they present their papers, such as submitting to a workshop instead of a conference, (c) if we implement a centralized public repository of under-review or deblinded papers such as the “Findings of the EMNLP” or “ACL Archive”, allow the paper to remain there without being presented at any conference.

ShiqiZhao at 15:49, 17 June 2020

2020-06-17T15:49:38Z

← Older revision		Revision as of 15:49, 17 June 2020
Line 71:		Line 71:
	'''Day-by-day capacity to have competent reviewers* handle a diverse set of papers:''' Once the action editor is chosen, they will presumably be an expert in the field. They can then use the automated system to find a list of suggested reviewers, and choose them based on their expertise (as well as reviewing load, see below). If there are no reviewers currently within the reviewing pool, they can invite external reviewers to the reviewing pool (possibly suggested by a similar mechanism to the one mentioned above for action editors).		'''Day-by-day capacity to have competent reviewers* handle a diverse set of papers:''' Once the action editor is chosen, they will presumably be an expert in the field. They can then use the automated system to find a list of suggested reviewers, and choose them based on their expertise (as well as reviewing load, see below). If there are no reviewers currently within the reviewing pool, they can invite external reviewers to the reviewing pool (possibly suggested by a similar mechanism to the one mentioned above for action editors).
	*'''Ability of program chairs to encourage participation of action editors, reviewers, or authors in certain thematic areas:''' Note that the program chairs will always have some discretion with respect to how they build the program of the conference, as they do now. Let's say a conference wants to have a theme session. The simplest way to achieve this is an announcement to the effect of "there will be a theme on XXX at ACL 2022, please submit papers to the reviewing pool on these topics!" If the reviewing pool already has the capacity to review these papers, then nothing needs to be changed. However, if the reviewing pool is lacking in editors or reviewers in these areas, then the above mechanisms could be used to add editors or reviewers accordingly.		*'''Ability of program chairs to encourage participation of action editors, reviewers, or authors in certain thematic areas:''' Note that the program chairs will always have some discretion with respect to how they build the program of the conference, as they do now. Let's say a conference wants to have a theme session. The simplest way to achieve this is an announcement to the effect of "there will be a theme on XXX at ACL 2022, please submit papers to the reviewing pool on these topics!" If the reviewing pool already has the capacity to review these papers, then nothing needs to be changed. However, if the reviewing pool is lacking in editors or reviewers in these areas, then the above mechanisms could be used to add editors or reviewers accordingly.

			=== How to Transition to the System? ===

			Related venues will need to agree to use the system and also transition to using it seamlessly, how do we ensure venues agree to use the system? For major conferences (e.g. ACL, EMNLP, NAACL, EACL, AACL) and potentially also major journals (TACL), there are two potential options: complete adoption and partial adoption. In complete adoption, all submissions would be handled through the two-step rolling review system. In a partial adoption, the venues could run their own review process, but additionally accept submissions from the centralized reviewing pool. Complete adoption would likely be more efficient/fair, but also require a decision from the ACL executive board to switch over to this new system and mandate this of the program chairs. One option would be to do a partial adoption for a few years, then move to a complete adoption.

			For more focused workshops or other non-major conferences in the ACL anthology sometimes the character of venues is very different, and it is necessary to ensure that people still have flexibility and don’t feel constrained. Focused workshops may prefer to partially adopt the system and still have their own reviewing pool, but other workshops may also prefer not to have to handle reviewing themselves, as it is a major burden.

			=== Measuring “Acceptance Rate” ===

			There are some organizations that measure the quality of conferences based on the acceptance rate of papers submitted there. This can be important for career advancement, so it is important to have an acceptance rate that appropriately represents the difficulty of publishing a paper in the conference. A few ideas for measuring acceptance rate for a particular venue are below:

			*'''Accepted Papers / Presentation Requests for Conference or Journal:''' This is simple, but may face problems if authors self-select, not requesting a presentation if their reviews are not good. Thus the acceptance rate could seem artificially high, which would make the venues seem less selective than they actually are.
			*'''Accepted Papers / Papers in the Review System:''' Another option would be to divide the number of accepted papers by the number of papers in the review system and eligible to make a presentation request at any point. This would give a low acceptance rate, because all papers submitted to ACL-related venues would be in the system making the denominator quite large.
			*'''Accepted Papers / Some Fraction of Papers in the Review System:''' We could take a middle-ground strategy between the two. For example, upon submission have authors select a preferred venue (ACL/EMNLP/NAACL), or preferred type of venue (Conference/Journal/Workshop). We could then use the fraction of the papers that selected a particular venue as preferred as an indication of that venue’s popularity, and calculate the denominator based on these statistics. Alternatively, we could use all papers in the system that selected a particular type of venue, e.g. “Conference”, in the denominator when calculating conference acceptance rates (and similarly for journals).
			*'''Compare Accepted Paper Scores to Full Submission Pool Scores:''' As another option, we could keep track of the overall distribution of numerical review scores in the system and the distribution for each conference. Then, we calculate an estimation of the review score necessary to be accepted at the conference, and find its percentile in the overall review pool. To take a made-up example, let's say that the median overall review score of papers accepted to ACL was 4.1, then and only 16% of papers in the overall review pool had scores 4.1 or higher. In this case, the acceptance rate for ACL would be reported as 16%. Instead of the median for each publication venue, we could also take the 10th or 20th percentile, representing the lower end of the venue’s accepted range.