The value of a wild AI token changes sign: an 800-model scaling study says filter, don't mix
UMD and Pangram Labs pretrain 800 models on human + AI web text, fit a sign-changing scaling law, and find FineWeb and DCLM filters select for AI text.
A study from the University of Maryland and Pangram Labs pretrains 800 language models on mixtures of human and "wild" AI-generated web text. It finds the value of an AI token changes sign: it helps only data-starved models, and past roughly the Chinchilla-optimal 20 human tokens per parameter it raises loss on human text. The paper also measures that 27.5% of June 2026 quality-filtered web tokens are AI-generated, rising to 31.1% by August — and that the quality filters used to build pretraining corpora keep AI documents 2.3x (FineWeb) and 9.8x (DCLM) more often than human ones.
What shipped
The study (arXiv:2609.40295) targets what the authors call wild AI text: text LLMs wrote for human readers that arrives unlabeled in web crawls, as opposed to synthetic data or recursive model-collapse setups, which they argue no previous work directly addresses. They pretrained 800 models from 19.9M to 973M parameters, adding up to 64 wild AI tokens per human token to fixed human corpora, and evaluated on held-out human text (C4, FineWeb, Paloma) and on AI text.
The core result: AI text lowers loss on human text only below about 10 human tokens per parameter. At the Chinchilla-optimal 20 tokens per parameter the benefit is gone; larger AI ratios raise loss, while the same number of fresh human tokens keeps lowering it.
To capture this, the paper proposes a scaling law with separate benefit and harm terms — a saturating benefit plus a logarithmically growing harm penalty — that reduces exactly to Chinchilla when no AI tokens are present. The value of an AI token, expressed in human-token equivalents, changes sign: positive at 5 human tokens per parameter, below zero from 20 once the AI-to-human ratio r exceeds about 0.08. Fit on models up to 268M parameters, the law predicts models 3.6x larger with 41% lower error than the best existing law (paired error 0.83e-3 on C4, versus 1.41e-3 for Shukor et al. 2025 and 4.32e-3 for Chinchilla).
The practical consequence is a compute penalty: training on unfiltered web text at August 2026's 31.1% AI share requires 1.6x the compute of training on its human subset at 20 tokens per parameter — 2.1x at the 42% share forecast for end-2027, 3.0x at 51% for end-2028.
Two secondary findings are more actionable:
- Quality filters select for AI text. Auditing 10,000 raw Common Crawl documents, FineWeb's pipeline keeps AI documents 2.3x as often as human ones (29.3% vs 12.8%); DCLM's keeps them 9.8x as often (14.5% vs 1.5%).
- Mixed validation sets hide the harm. Because AI text is easier to predict, a validation set with a 22.3% AI share (their 2026 evaluation crawl) fails to show the harm in 95.5% of runs where human-text loss actually rose.
The authors release WildAI, an 83.31B-token corpus (96.04M documents) with per-document AI, topic, and format labels — 42.19B human tokens and 35.32B AI tokens — plus all 800 models and code. The corpus is a mixture pool, not a natural web sample: a first-pass labeler (EditLens, a Llama-3.2-3B) preselected likely-AI documents, so AI text is 42% of WildAI's tokens, far above its roughly 28–31% web share.
Why it matters
Every pretraining run now faces this choice, and real models are far past the regime where AI text helps. Qwen3-32B, for example, was trained on roughly 1,100 tokens per parameter — about 55x the Chinchilla-optimal 20. At those budgets the paper's law says wild AI tokens are net-harmful, so the question is not whether to include them but how aggressively to filter.
The filter-bias finding is the sharpest and most immediately checkable: the pipelines built to keep "good" text are systematically keeping AI text. If FineWeb keeps AI documents 2.3x as often, a corpus built with it is already more AI-heavy than the web it samples — before anyone makes a deliberate mixture decision. Anyone running a filter pipeline should re-audit it for AI-text survival bias.
The validation-set result is a trap most eval setups will hit: a mixed eval set can look flat or improving while human-text loss rises. Reporting human and AI validation loss separately is a cheap change with direct signal.
The paper's recommendations follow directly: filter AI text when the target is human text, repeat human text before expanding the dataset with AI web text, and report validation loss on human and AI text separately. The one place AI text stays valuable is when the target is AI text itself — for predicting Cosmopedia-style text, the optimal mix is over 90% AI.
The law itself is a planning tool: given a target human budget and an AI share, it yields the compute multiplier and the sign of the marginal AI token, so mixture decisions can be made before the training run starts.
Caveats
- The headline AI-share numbers come from Pangram 3.3.2, a detector from Pangram Labs — the company most of the authors are affiliated with (the first author's work was done during a Pangram internship). The detector's error rates (FPR 0.05%, FNR 1.99%) are self-reported; as an external sanity check, Pangram labels 0.062% of 60,000 pre-ChatGPT 2021 documents as AI, which the paper treats as an upper bound on the false-positive rate. We found no independent replication.
- The WildAI corpus, 800 models, and code are announced, but the GitHub repository (github.com/pangramlabs/WildAI) returned 404 as of writing — the release is not yet verifiable.
- The scaling law is fit on models up to 268M parameters and extrapolated 3.6x upward; the compute multipliers for 2027/2028 depend on forecast AI shares. All of it is vendor-reported, preprint, and unverified by third parties.
- The sign-change threshold (benefit below ~10 human tokens per parameter, harm from 20) is measured at small scale; whether it holds at frontier budgets is what the law predicts, not what it proves.
- WildAI's reported token counts don't reconcile — 42.19B human + 35.32B AI = 77.51B, not the stated 83.31B. The discrepancy is in the paper, not this post.