As part of a proposal to develop the VoxML modeling language into an ISO Standard for the representation of visual information, the 17th Joint ACL-ISO Workshop on Interoperable Semantic Annotation (ISA) is hosting a shared multimodal annotation. The goal of the task is to assess annotation techniques suitable for extracting habitats, affordances, and spatial relations of objects from image data.
We are supplying an annotation interface and a batch of curated images from the Visual Genome dataset showing objects in common configurations or activities. Annotators will be asked to view images, caption them, and annotate the objects, spatial relations, and activities (depicted or potential) in each image, according to a provided guideline. We ask participants in the shared task to annotate as many of the images as they can using the provided tool. Annotators and teams may then submit a project note (up to 4 pages), discussing, but not limited to:
• reflections on the adequacy of the annotation guideline;
• facility of the annotation tool;
• properties observed in the raw image data while annotating;
• confounding examples;
• potential problems pertaining to standardization, and their solutions;
• suggested deeper investigations into other things that would be interesting to annotate (other data, other properties we should have included, etc.).
Project notes will be reviewed for presentation (oral or poster) at ISA, space permitting.
Annotators should attempt to follow the guidelines, including vocabulary and format recommendations, but are permitted to go outside it if required. In that case we ask that they provide a summary justification in the project note submission.
The track organizers will provide feedback on submitted annotations, observations, and analyses during the ISA workshop, followed by a discussion of the annotation effort, lessons learned (e.g., problems, gaps, suitability of the results for machine learning, automated generation, etc.), and the adequacy and expressiveness of the VoxML representation language.
A background document on VoxML is available at: VoxML: A Visualization Modeling Language.
Participants may install the annotation tool by following the instructions at: https://github.com/csu-signal/VoxML-Track-Annotation-2021. Deadline for submissions of annotations and project notes is April 21, 2021.
For any questions please contact the track coordinators:
James Pustejovsky: jamesp [at] brandeis.edu
Nikhil Krishnaswamy: nkrishna [at] colostate.edu
Technical support: signallab.ai [at] gmail.com