SIMMC: A dataset for developing next-generation shopping assistants

What it is:

Situated and Interactive Multimodal Conversations (SIMMC) is a first-of-its-kind dataset that has been open-sourced to help researchers and engineers develop virtual assistants capable of handling complex, task-oriented conversations in a co-observed multimodal context.

Imagine you’re buying a product from an online store using AR/VR. One of the easiest ways to make this interaction happen smoothly would be to use a digital assistant that can act on your voice commands and cues (e.g., “I’d like to buy a brown leather couch”) — similar to how a salesperson might help you in a brick-and-mortar store.

However, the complexity of an AR/VR environment means the assistant must be able to function beyond simple call-and-response or question-and-answer actions. Successfully operating in a virtual environment means being able to handle and memorize the various multimodal inputs inherent to it, such as the visual aspects of objects, including color, size, shape, and orientation.

If the wider research and engineering community is going to develop digital assistants that can closely mimic their human counterparts, there are a range of novel and nontrivial research challenges to address.

SIMMC is a dataset specifically aimed at training agents that take multimodal actions grounded in a coevolving multimodal input context in addition to the dialog history. SIMMC tasks address task-oriented dialogs that encompass a rich, situated multimodal user context in the form of a co-observed image or a VR environment, which gets updated dynamically based on the dialog flow and the assistant actions. It enables AI assistants to understand the evolving context of the interaction in much the same way the user does.

Something Went Wrong

We're having trouble playing this video.

Learn more

What it does:

SIMMC contains about 13,000 human-to-human dialogs (totaling about 169,000 utterances). We chose shopping experiences — specifically furniture and fashion — as the domain for the SIMMC datasets because of the dynamic environment created by the shopping experience, where rich multimodal interactions happen around visually grounded items.

SIMMC offers four key advantages over previous multimodal dialog datasets:

Machine learning models trained on the furniture and fashion datasets were evaluated for their performance in API call prediction, response generation, and dialog state tracking.

Why it matters:

We’ve organized a challenge track at the Ninth Dialog System Technology Challenge (DSTC9) around SIMMC. This track invites the dialog research community to tackle efforts toward developing real-world assistant agents that can handle multimodal inputs and perform multimodal actions.

The SIMMC framework is a step toward building next-generation virtual assistants that can perform the sorts of multimodal reasoning needed to create dynamic experiences, such as AR/VR-based shopping. A dataset of this type also opens the door to further research into conversational AI, including multimodal entity disambiguation.

We’ve provided two datasets (furniture and fashion), as well as the contextual natural language understanding and coreference annotations on these datasets for further study. There are several strong baselines for some of the tasks enabled by the datasets that showcase their various uses in real-world applications. SIMMC is in the research phase right now, and we believe the collected annotations should facilitate further study into the tasks highlighted in this work and several other tasks.

Get it on GitHub:

Situated Interactive MultiModal Conversations (SIMMC) Challenge 2020

We would like to thank our collaborators:
Seungwhan Moon, Paul A. Crook, Ankita De, Shivani Poddar, Theodore Levin, David Whitney, Daniel Difranco, Eunjoon Cho, Rajen Subba, and Alborz Geramifard