March 11, 2021
Many technical approaches have been proposed for ensuring that decisions made by machine learning systems are fair, but few of these proposals have been stress-tested in real-world systems. This paper presents an example of one team’s approach to the challenge of applying algorithmic fairness approaches to complex production systems within the context of a large technology company. We discuss how we disentangle normative questions of product and policy design (like, “how should the system trade off between different stakeholders’ interests and needs?”) from empirical questions of system implementation (like, “is the system achieving the desired tradeoff in practice?”). We also present an approach for answering questions of the latter sort, which allows us to measure how machine learning systems and human labelers are making these tradeoffs across different relevant groups. We hope our experience integrating fairness tools and approaches into large-scale and complex production systems will be useful to other practitioners facing similar challenges, and illuminating to academics and researchers looking to better address the needs of practitioners.
Written by
Chloé Bakalar
Renata Barreto
Stevie Bergman
Miranda Bogen
Bobbie Chern
Sam Corbett-Davies
Melissa Hall
Isabel Kloumann
Michelle Lam
Joaquin Quiñonero CandelaManish Raghavan
Joshua Simons
Jonathan Tannen
Edmund Tong
Kate Vredenburgh
Jiejing Zhao
June 14, 2020
Many visual scenes contain text that carries crucial information, and it is thus essential to understand text in images for downstream reasoning tasks. For example, a deep water label on a warning sign warns people about the danger in the…
Ronghang Hu, Amanpreet Singh, Trevor Darrell, Marcus Rohrbach
June 14, 2020
April 25, 2020
The long-tail distribution of the visual world poses great challenges for deep learning based classification models on how to handle the class imbalance problem.…
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, Yannis Kalantidis
April 25, 2020