Face recognition getting better at recognising masked faces

Facial recognition technology is getting better at recognising masked faces, according to a study by the National Institute of Standards and Technology (NIST). It is the Institute’s first study that measures the performance of face recognition algorithms developed following the onset of the COVID-19 pandemic.

A previous study in July explored the effect of masked faces on algorithms submitted before March 2020, indicating that software available before the pandemic often had more trouble with masked faces.

Mei Ngan, one of the study’s authors, said that some new algorithms from developers performed better than their predecessors, with error rates decreasing by as much as a factor of 10 between their pre- and post-COVID algorithms.

“In the best cases, software algorithms are making errors between 2.4 and 5% of the time on masked faces, comparable to where the technology was in 2017 on non-masked photos,” Ngan said.

The new study analysed 65 new submitted algorithms, along with those that were tested on masked faces in the previous round, providing cumulative results for 152 total algorithms. While the developers submitted the algorithms voluntarily, they did not indicate whether an algorithm was designed to handle face masks or whether it was used in commercial products.

Using the same set of 6.2 million images as it had previously, researchers tested the algorithms’ ability to perform ‘one-to-one’ matching, in which a photo is compared with a different photo of the same person — a function commonly used to unlock a smartphone. As with the July report, the images had mask shapes digitally applied, rather than showing people wearing actual masks.

The report found that when both the new image and the stored image are of masked faces, error rates run higher. With a couple of notable exceptions, when the face was occluded in both photos, false match rates ran 10 to 100 times higher than if the original saved image showed an uncovered face. Smartphones often use one-to-one matching for security, and it would be far more likely for a stranger to successfully unlock a phone if the saved image was of a masked person.

The more of a face a mask covers, the higher the algorithms’ error rate. Continuing a trend from the previous study, round mask shapes — which only cover the mouth and nose — generated fewer errors than wide shapes that stretch across the cheeks. Masks covering the nose generated more errors than those that did not.

Mask colours also affected the error rate; the new study explored the effects of red and white mask colours, as well as the black and light blue masks tested in July. The red and black masks tended to yield higher error rates than the other colours; however, the researchers did not investigate potential reasons for this effect.

Certain algorithms performed well with any combination of masked or unmasked faces. The ‘mask-agnostic’ software can handle images regardless of whether or not the faces are masked. The algorithms detect the difference automatically, without being told.

Researchers also found the individual algorithms tend to differ. End users need to get to know how their chosen software performs in their own specific situations, ideally using real physical masks rather than the digital simulations the team used in the study.

“It is incumbent upon the system owners to know their algorithm and their data. It will usually be informative to specifically measure accuracy of the particular algorithm on the operational image data collected with actual masks,” Ngan said.

Image credit: B. Hayes, M. Ngan/NIST