Dynabench Working Group
Mission
Accelerate machine innovation and increase scientific rigor in machine learning by providing a flexible machine learning benchmarking platform.
Purpose
Dynabench is a research platform for dynamic data collection and benchmarking. In particular, Dynabench challenges existing ML benchmarking dogma by embracing dynamic dataset generation. Benchmarks for machine learning solutions based on static datasets have well-known issues: they saturate quickly, are susceptible to overfitting, contain exploitable annotator artifacts and have unclear or imperfect evaluation metrics. In this sense, Dynabench enables a scientific experiment: is it possible to make faster progress if data is collected dynamically, with humans and models in the loop, rather than in the old-fashioned static way? Further, Dynabench enables an ecosystem of other ML benchmarks in areas such as algorithmic efficiency.
Deliverables
- Roadmap for Dynabench development
- Dynabench benchmarking platform
- Dynabench community support
Meeting Schedule
Weekly on Thursday from 10:35-11:30AM Pacific.
Join
Related Blog
-
-
-
Harnessing Human-AI Collaboration
Dynamic Adversarial Data Collection augments large scale datasets by adding diverse and high-quality data
Dynabech Working Group Projects
Dynabench
How to Join and Access Dynabench Working Group Resources
- To sign up for the group mailing list, receive the meeting invite, and access shared documents and meeting minutes:
- Fill out our subscription form and indicate that you’d like to join the Dynabench Working Group.
- Associate a Google account with your organizational email address.
- Once your request to join the Dynabench Working Group is approved, you’ll be able to access the Dynabench folder in the Public Google Drive.
- To engage in working group discussions, join the group’s channels on the MLCommons Discord server.
- To access the GitHub repository (public):
- If you want to contribute code, please submit your GitHub ID to our subscription form.
- Visit the GitHub repository.
Dynabench Website
Visit the Dynabench website to learn more about the communities and challenges hosted on the Dynabench platform.
Dynabench Working Group Chairs
To contact all Dynabench working group chairs email dynabench-chairs@mlcommons.org.
Adina Williams (adinawilliams@mlcommons.org) - LinkedIn
Adina Williams is an AI Research Scientist in the Facebook Artificial Intelligence Research (FAIR) Group in New York City. She received her PhD in Linguistics under the supervision of Liina Pylkkänen in the fall of 2018 from New York University, where she also contributed to the Machine Learning for Language Laboratory in the Center for Data Science with the support of Sam Bowman. Her research aims to bridge the gap between linguistics, cognitive science, and NLP. She is currently working on projects involving natural language inference, evaluating model biases, and information theoretic approaches to computational morphology.
Max Bartolo (max@mlcommons.org) - LinkedIn - Twitter
Max leads the Command modelling team at Cohere working on improving adversarial robustness and the overall capabilities of large language models. He is also one of the original contributors to the Dynabench working group, which he currently co-leads, and he also lectures at UCL.