Call for Participation: LangCode 2027 Shared Task 2, Adversarial Prompt Detection @ NAACL 2027

Abbreviated Title: 
LangCode 2027 Task 2
ACL-Sponsored Event: 
This is an ACL Sponsored Event
Event Notification Type: 
Call for Participation
Location: 
NAACL 2027
Event Dates: 
Tuesday, 1 June 2027 to Saturday, 5 June 2027
Submission Deadline: 
Sunday, 24 January 2027
Contact: 
Nishat Raihan
Contact Email: 
mraihan [at] nd.edu
City: 
San Francisco
State: 
California
Country: 
United States
Description/Details: 

Call for Participation

LangCode 2027 Shared Task 2: Adversarial Prompt Detection
LangCode 2027: The First Language and Code Workshop
NAACL 2027, San Francisco, USA, June 1-5, 2027
https://langcode2027.github.io/task2/

Code-model safety filters are calibrated almost entirely on English. In MALICE (Raihan, Meher, Dhingra and Zampieri, Findings of EMNLP 2026), the best classifier's accuracy fell from 0.93 on English to 0.21 on code-mixed malicious prompts. This task asks for guardrails that hold up on such prompts.

The task:
Given a prompt sent to a code model, label it safe (S) or unsafe (U). Every prompt is code-mixed (one of 10 languages mixed with English) or transliterated (English written in another script): Arabic, Bengali, Chinese, Hebrew, Hindi, Japanese, Korean, Russian, Tamil or Thai. Safe prompts cover the same security topics as unsafe ones, so a system that flags every security word fails them.

Scoring:
Teams are ranked by the Safety Score, R_U + R_S - 100 (Youden's J), where R_U and R_S are recall on unsafe and safe prompts. A missed attack and a flagged safe prompt cost the same. Labeling every prompt one way, or at random, scores 0; a perfect system scores 100.

Data and starter kit (on the task website):

* Train set: 20,000 labeled prompts
* Dev set: 10,000 labeled prompts
* Dev-test set: 10,000 prompts without labels, for Development-phase submissions
* Test set: 30,000 prompts without labels, released January 8, 2027
* The scorer that runs on CodaBench, with a format check for submission files
* A Colab notebook that fine-tunes BERT-base on a free T4 GPU and writes a checked submission file

Any public data, model or tool is allowed. The data contains prompts written to elicit malicious code and may be used only for research on detecting and defending against such prompts; the terms are on the Data page.

How to take part:
Register on CodaBench: https://www.codabench.org/competitions/18479/
Participation is free. Teams can have up to 5 members.

Important dates (AoE):

* Development phase: open now until January 7, 2027
* Test data release: January 8, 2027
* CodaBench registration deadline: January 17, 2027
* System submission deadline: January 24, 2027
* Official results: January 25, 2027
* System papers due: February 5, 2027
* Notification: March 26, 2027
* Camera-ready: April 16, 2027

System papers:
Up to 4 pages in the ACL template, submitted via OpenReview. Accepted papers appear in the ACL Anthology and are presented at the workshop.
Details: https://langcode2027.github.io/task2/paper/

Organizers:
Nishat Raihan, Beatrice Casey, David Chiang, Joanna C. S. Santos (University of Notre Dame);
Kevin Moran (University of Central Florida); Ziyu Yao, Marcos Zampieri (George Mason University)

Questions: the Forum tab of the CodaBench competition.