Australian police agencies are reportedly using a private, unaccountable facial recognition service that combines machine learning and wide-ranging data-gathering practices to identify members of the public from online photographs.

The service, Clearview AI, is like a reverse image search for faces. You upload an image of someone’s face and Clearview searches its database to find other images that contain the same face. It also tells you where the image was found, which might help you determine the name and other information about the person in the picture.

Clearview AI built this system by collecting several billion publicly available images from the web, including from social media sites such as Facebook and YouTube. Then they used machine learning to make a biometric template for each face and match those templates to the online sources of the images.

It was revealed in January that hundreds of US law enforcement agencies are using Clearview AI, starting a storm of discussion about the system’s privacy implications and the legality of the web-scraping used to build the database.

Australian police agencies initially denied they were using the service. The denial held until a list of Clearview AI’s customers was stolen and disseminated, revealing users from the Australian Federal Police as well as the state police in Queensland, Victoria and South Australia.

Lack of accountability

This development is particularly concerning as the Department of Home Affairs, which oversees the federal police, is seeking to increase the use of facial recognition and other biometric identity systems. (An attempt to introduce new legislation was knocked back last year for not being adequately transparent or privacy-protecting.)

Gaining trust in the proper use of biometric surveillance technology ought to be important for Home Affairs. And being deceptive about the use of these tools is a bad look.

Read more: Why the government's proposed facial recognition database is causing such alarm

But the lack of accountability may go beyond poor decisions at the top. It may be that management at law enforcement agencies did not know their employees were using Clearview AI. The company offers free trials to “active law enforcement personnel”, but it’s unclear how they verify this beyond requiring a government email address.

Why aren’t law enforcement agencies enforcing rules about which surveillance tools officers can use? Why aren’t their internal accountability mechanisms working?

There are also very real concerns around security when using Clearview AI. It monitors and logs every search, and we know it has already had one data breach. If police are going to use powerful surveillance technologies, there must be systems in place for ensuring those technological tools do what they say they do, and in a secure and accountable way.

Is it even accurate?

Relatively little is known about how the Clearview AI system actually works. To be accountable, a technology used by law enforcement should be tested by a standards body to ensure it is fit for purpose.

Clearview AI, on the other hand, has had its own testing done – and as a result its developers claim it is 100% accurate.

That report does not represent the type of testing that an entity seeking to produce an accountable system would undertake. In the US at least, there are agencies like the National Institute for Standards and Technology that do precisely that kind of accuracy testing. There are also many qualified researchers in universities and labs that could properly evaluate the system.

Instead, Clearview AI gave the task to a trio composed of a retired judge turned private attorney, an urban policy analyst who wrote some open source software in the 1990s, and a former computer science professor who is now a Silicon Valley entrepreneur. There is no discussion of why those individuals were chosen.

The method used to test the system also leaves a lot to be desired. Clearview AI based their testing on a test by the American Civil Liberties Union of Amazon’s Rekognition image analysis tool.

However, the ACLU test was a media stunt. The ACLU ran headshots of 28 members of congress against a mugshot database. None of the politicians were in the database, meaning any match returned would be an error. However, the test only required the system to be 80% certain of its results, making it quite likely to return a match.

Read more: Close up: the government's facial recognition plan could reveal more than just your identity

The Clearview AI test also used headshots of politicians taken from the web (front-on, nicely framed, well-lit images), but ran them across their database of several billion images, which did include those politicians.

The hits returned by the system were then confirmed visually by the three report authors as 100% accurate. But what does 100% mean here?

The report stipulates that the first two hits provided by the system were accurate. But we don’t know how many other hits there were, or at what point they stopped being accurate. Politicians have lots of smiling headshots online, so finding two images should not be complex.

What’s more, law enforcement agencies are unlikely to be working with nice clean headshots. Poor-quality images taken from strange angles – the kind you get from surveillance or CCTV cameras – would be more like what law enforcement agencies are actually using.

Despite these and other criticisms, Clearview AI CEO Hoan Ton-That stands by the testing, telling Buzzfeed News he believes it is diligent and thorough.

More understanding and accountability are needed

The Clearview AI case shows there is not enough understanding or accountability around how this and other software tools work in law enforcement. Nor do we know enough about the company selling it and their security measures, nor about who in law enforcement is using it or under what conditions.

Beyond the ethical arguments around facial recognition, Clearview AI reveals Australian law enforcement agencies have such limited technical and organisational accountability that we should be questioning their competency even to evaluate, let alone use, this kind of technology.

Jake Goldenfein is a board member of the Australian Privacy Foundation.