Researchers use Ai2 to trace phrasing in AI books
Researchers are using Ai2’s infini-gram engine to trace distinctive language in AI-generated books, proving that machine-written bestsellers rely heavily on rare phrasing from existing works.
Researchers at Stony Brook University, led by assistant professor Tuhin Chakrabarty, are using the Allen Institute for AI's (Ai2) infini-gram engine to dissect machine-generated text and trace its origins. Unlike standard AI detectors that merely estimate whether a passage is synthetic, infini-gram indexes massive datasets to count phrase occurrences of any length. By pairing this engine with Google Books, Chakrabarty’s team can identify when AI-generated text lifts rare, distinctive expressions from existing literature, providing concrete evidence of source overlap.
In a recent study examining top-selling self-published genre books on Amazon, the researchers defined "rare expressions" as multi-word phrases appearing in five or fewer Google Books volumes and completely absent from infini-gram's web snapshot. Analyzing the 100 highest-revenue books with substantial AI text versus 100 with none, they found that rare expressions from existing books covered 43.2% of the AI-heavy texts compared to 37.6% of the human-written ones. When expanding the analysis to the top 200 books in each category, AI-heavy books still contained 4.4 percentage points more rare expressions. This gap widened to 22.5 points when compared to a reference set of award-winning or nominated literature. In that literary reference set, rare expressions covered an average of 19.1% of the text, compared with 37.2% in top-selling books with no AI text and 41.6% in those with substantial AI text.
For AI practitioners and researchers, this development shifts the paradigm of AI detection and copyright compliance. Tools like OlmoTrace—which connects outputs from Ai2's open Olmo models back to their Dolma training datasets—and the Creativity Index allow developers to audit model outputs for plagiarism. Instead of relying on a single probability score, practitioners can now pinpoint exact phrases, such as the "sour tang of fermenting" from the controversial GrantaGate story, to see where they originated. This capability is crucial for debugging training pipelines, verifying the originality of generated content, and investigating whether models have ingested pirated or copyrighted materials.
This is our own summary of reporting by AI2
