Meta AI is pleased to invite university faculty to respond to this call for research proposals about the Dynabench platform. Launched in 2020, Dynabench is a research platform for dynamic data collection and benchmarking.
Static benchmarks have well-known issues. They saturate quickly, are susceptible to overfitting, contain exploitable annotator artifacts and have unclear or imperfect evaluation metrics. The Dynabench platform is a scientific experiment to answer the question: Can we make faster progress if we collect data dynamically, with humans and models in the loop, rather than in the old-fashioned static way? The aim is to continuously challenge existing benchmarking dogma and try to embrace dynamic solutions at all times.
The Dynabench platform initially comprised four core tasks in natural language processing. It has now reached the level of maturity where we hope it will be useful to the broader research community. Meta aims to open up the platform for anyone interested in human-and-model-in-the-loop data collection to run their own task, and is soliciting proposals for interesting tasks and experiments that Meta can help realize.
To foster further innovation in this area and deepen our collaboration with academia, Meta AI is pleased to invite university faculty to submit research proposals pertaining to Dynabench. A total of 4 to 8 awards are available, in the range of $15,000 to $55,000 each, including overhead in an amount up to 40% of project costs. Funding will be provided to RFP winners pursuant to a Sponsored Research Agreement (SRA) containing open science terms. Please note that the terms of the SRA will not be subject to negotiation. We strongly encourage researchers from diverse backgrounds and of diverse abilities to apply.
The deadline for submissions is March 23, 11:59pm PT.
Meta AI encourages creative submissions along the theme of “rethinking benchmarking” that leverage the Dynabench platform, with areas of interest that include, but are not limited to, the following:
New tasks or benchmarks that would benefit from having a dynamic (adversarial) component.
Experiments that examine concrete questions about the human-and-model-in-the-loop paradigm for data collection or evaluation.
Proposals for moving Dynabench beyond natural language processing and/or beyond the English language.
Prior to submitting a proposal, please confirm that your institution will agree to a contract for sponsored research from Meta. The agreement will operate under “Open Science” terms meaning that all research will be available for publication in the public sphere. Application materials include:
A summary of the project (1-2 pages), in English, explaining the area of focus, a description of techniques, any relevant prior work, and a timeline with milestones and expected outcomes;
A draft budget description (1 page) including an approximate cost of the award and explanation of how funds would be spent; .
Curriculum Vitae for all project participants; and
Organization details; this will include tax information and administrative contact details.
Proposals must comply with applicable U.S. and international laws, regulations, and policies;
Applicants must be current full-time faculty at an accredited academic institution that awards research degrees to PhD students;
Applicants must be the Principal Investigator on any resulting award;
Meta cannot consider proposals submitted, prepared or to be carried out by individuals residing in, or affiliated with an academic institution located in, a country or territory subject to comprehensive U.S. trade sanctions; and
Government officials (excluding faculty and staff of public universities, to the extent they may be considered government officials), political figures, and politically affiliated businesses (all as determined by Meta in its sole discretion) are not eligible.
Please read these terms carefully before proceeding.
Meta’s decisions will be final in all matters relating to Meta RFP solicitations, including whether or not to grant an award and the interpretation of Meta RFP Terms and Conditions. By submitting a proposal, applicants affirm that they have read and agree to these Terms and Conditions.
Meta is authorized to evaluate proposals submitted under its RFPs, to consult with outside experts, as needed, in evaluating proposals, and to grant or deny awards using criteria determined by Meta to be appropriate and at Meta’s sole discretion. Meta’s decisions will be final in all matters relating to its RFPs, and applicants agree not to challenge any such decisions.
Meta will not be required to treat any part of a proposal as confidential or protected by copyright, and may use, edit, modify, copy, reproduce and distribute all or a portion of the proposal in any manner for the sole purposes of administering the Meta RFP website and evaluating the contents of the proposal.
Neither Meta nor the applicant is obligated to enter into a business transaction as a result of the proposal submission. Meta is under no obligation to review or consider any proposal.
Feedback provided in a proposal regarding Meta products or services will not be treated as confidential or protected by copyright, and Meta is free to use such feedback on an unrestricted basis with no compensation to the applicant. The submission of a proposal will not result in the transfer of ownership of any IP rights.
Applicants represent and warrant that they have authority to submit a proposal in connection with a Meta RFP and to grant the rights set forth herein on behalf of their organization. All awards provided by Meta in connection with this RFP shall be used only in accordance with applicable laws and shall not be used in any way, directly or indirectly, to facilitate any act that would constitute bribery or an illegal kickback, an illegal campaign contribution, or would otherwise violate any applicable anti-corruption or political activities law.
Funding for winning RFP proposals will be provided to the academic institution with which the applicant is affiliated pursuant to a sponsored research agreement (“SRA”) between Meta and such academic institution. Applicants understand and acknowledge that their affiliated academic institution will need to agree to the non-negotiable terms and conditions of the SRA to receive funding.
Applicants must disclose any current or past collaborations with Meta researchers that are related to their RFP project proposal.