IBM Research releases ‘Diversity in Faces’ dataset to advance study of fairness in facial recognition systems

*Originally published January 29, 2019; updated February 15, 2019, to reflect important contributions from Joy Buolamwini and Timnit Gebru in Gender Shades (2018) cited in the Diversity in Faces arXiv paper.

Have you ever been treated unfairly? How did it make you feel? Probably not too good. Most people generally agree that a fairer world is a better world, and our AI researchers couldn’t agree more. That’s why we are harnessing the power of science to create AI systems that are more fair and accurate.

Many of our recent advances in AI have produced remarkable capabilities for computers to accomplish increasingly sophisticated and important tasks, like translating speech across languages to bridge communications across cultures, improving complex interactions between people and machines, and automatically recognizing contents of video to assist in safety applications.

Much of the power of AI today comes from the use of data-driven deep learning to train increasingly accurate models by using growing amounts of data. However, the strength of these techniques can also be a weakness. The AI systems learn what they’re taught, and if they are not taught with robust and diverse datasets, accuracy and fairness could be at risk. For that reason, IBM, along with AI developers and the research community, need to be thoughtful about what data we use for training. IBM remains committed to developing AI systems to make the world more fair.

The challenge in training AI is manifested in a very apparent and profound way with facial recognition technology. There can be difficulties in making facial recognition systems that meet fairness expectations. As shown by Joy Buolamwini and Timnit Gebru in Gender Shades in 2018, facial recognition systems in commercial use performed better for lighter individuals and males and worse for darker females ¹. The heart of the problem is not with the AI technology itself, per se, but with how the AI-powered facial recognition systems are trained. For the facial recognition systems to perform as desired – and the outcomes to become increasingly accurate –  training data must be diverse and offer a breadth of coverage, as shown in our prior work ². For example, the training data sets must be large enough and different enough that the technology learns all the ways in which faces differ to accurately recognize those differences in a variety of situations. The images must reflect the distribution of features in faces we see in the world.

How do we measure and ensure diversity for human faces? On one hand, we are familiar with how faces differ by age, gender, and skin tone, and how different faces can vary across some of these dimensions. Much of the focus on facial recognition technology has been on how well it performs within these attributes. But, as prior studies have shown, these attributes are just a piece of the puzzle and not entirely adequate for characterizing the full diversity of human faces. Dimensions like face symmetry, facial contrast, the pose the face is in, the length or width of the face’s attributes (eyes, nose, forehead, etc.) are also important.

Today, IBM Research is releasing a new large and diverse dataset called Diversity in Faces (DiF) to advance the study of fairness and accuracy in facial recognition technology. The first of its kind available to the global research community, DiF provides a dataset of annotations of 1 million human facial images. Using publicly available images from the YFCC-100M Creative Commons data set, we annotated the faces using 10 well-established and independent coding schemes from the scientific literature ^{3, 4, 5, 6, 7, 8, 9, 10, 11, 12}. The coding schemes principally include objective measures of human faces, such as craniofacial features, as well as more subjective annotations, such as human-labeled predictions of age and gender. We believe by extracting and releasing these facial coding scheme annotations on a large dataset of 1 million images of faces, we will accelerate the study of diversity and coverage of data for AI facial recognition systems to ensure more fair and accurate AI systems. Today’s release is simply the first step.

We believe the DiF dataset and its 10 coding schemes offer a jumping-off point for researchers around the globe studying the facial recognition technology. The 10 facial coding methods include craniofacial (e.g., head length, nose length, forehead height), facial ratios (symmetry), visual attributes (age, gender), and pose and resolution, among others. These schemes are some of the strongest identified by the scientific literature, building a solid foundation to our collective knowledge.

Our initial analysis has shown that the DiF dataset provides a more balanced distribution and broader coverage of facial images compared to previous datasets. Furthermore, the insights obtained from the statistical analysis of the 10 initial coding schemes on the DiF dataset has furthered our own understanding of what is important for characterizing human faces and enabled us to continue important research into ways to improve facial recognition technology.

References