Substantiate AI detection claims

On April 28, 2025, the US Federal Trade Commission (FTC) made it clear that any claims about the ability of a tool to detect whether artificial intelligence (AI) generated content need to be substantiated, just like other advertising claims. The FTC announced a settlement containing a decision and order with Workado LLC, formerly known as Content at Scale AI.

Background

The increasing use of artificial intelligence has led to the rise in technologies claiming to be able to determine whether text was prepared by AI, especially in the context of education matters but in other areas as well. (See this 2024 article on AI and plagiarism, for example. Note that Content at Scale AI is one of the companies listed in the referenced survey.) 

According to the FTC’s complaint, Workado made advertising claims that its “AI Content Detector” would predict with 98.3% accuracy whether text was generated by a human or generative AI technology. The complaint states that the developers of the AI model used in “AI Content Detector” tested the model’s accuracy. The “best result for testing a mix of human- and AI-created text was 74.5%” and for non-academic content, “merely 53.2%.” The complaint concluded that the advertising was false or misleading, and violated Section 5 of the FTC Act as unfair or deceptive acts or practices.

Consent

Under the proposed 20-year consent agreement, which will be published for public comment in the Federal Register, Workado neither admits nor denies the allegations in the complaint. As part of the settlement, Workado is prohibited from making misleading or unsubstantiated representations about “any product that incorporates technology that detects or purports to detect content, including text and images, generated or altered in part or in whole by artificial intelligence.”

If Workado does make any claims regarding AI detection technology, those claims must be substantiated, and the settlement contains a list of what is required for that substantiation. All documents relating to:

• Testing and testing protocols, including the rationale for the methodologies and metrics used
• Data used for testing
• Data sources, and “distribution, evaluation classes, class distribution, and types and description of values included”
• Processing steps performed on testing data and how those steps differ from those performed on training data, including steps take to avoid overlap between the two datasets
• The “reasoning as to why the data used for evaluation is sufficient and appropriate given the expected use cases, including why the data distribution is appropriate”
• Statistical analysis of any test data

Our take

Although AI detection software has had its efficacy claims questioned for some time, this matter is another example of the problems these solutions face and the potential downstream impact the lack in accuracy may create. 

Given the rise in shareholder AI-washing lawsuits related to the continued exaggeration and misrepresentations of a company’s products and services, it would not be surprising to see the FTC and other regulators require similar levels of substantiation for any specific advertising claims made regarding AI.