We are pleased to announce the Call for Participation for SemEval-2027 Task 11: VAKRA-Advanced — Evaluating Multi-Hop, Multi-Source Tool-Calling Capabilities in Executable Environments (https://semeval.github.io/SemEval2027/tasks).
VAKRA-Advanced investigates how AI agents plan and carry out tasks across APIs and documents. Participants will build systems that select tools, chain calls, use intermediate results, and ground their answers in information from structured and unstructured sources. The VAKRA benchmark provides an executable environment with more than 8,000 locally hosted APIs across 62 domains.
The task consists of four subtasks:
Subtask 1: API Chaining — plan and execute connected API calls.
Subtask 2: Tool Selection — identify the appropriate tools and APIs for a request from a list of 150+ APIs.
Subtask 3: Multi-hop API Reasoning — use outputs from earlier API calls to complete later steps.
Subtask 4: Multi-hop, Multi-source Reasoning — combine API results with retrieved documents to answer questions.
How to participate:
The dataset, sample data, code, and environment setup are available through the links below. Join the task mailing list for evaluation and submission announcements.
Important Dates:
Sample data and environment: https://github.com/IBM/vakra
Training data: https://huggingface.co/datasets/ibm-research/VAKRA
Evaluation start and end dates: January 2027
Paper submission: February 2027
Notification to authors: March 2027
Camera-ready submission: April 2027
Additional Information:
Task website: https://sites.google.com/view/vakrasemeval2027
Training Dataset: https://huggingface.co/datasets/ibm-research/VAKRA
Code and environment setup: https://github.com/IBM/vakra
VAKRA-Advanced mailing list: https://groups.google.com/g/vakra-advanced
Registration Form : https://forms.gle/2MLpxaU1kcD1SbCm7
Organizers:
Ankita Rajaram Naik, Anupama Murthi, Benjamin Elder, Siyu Huo, Praveen Venkateswaran, Hamid Adebayo, Sara Rosenthal, and Danish Contractor.
We look forward to your participation!