How AI Text Detectors Work: Methods, Accuracy, and Limitations

AI text detectors have become a flashpoint in newsrooms, classrooms, and content teams — tools that promise to tell you whether a human or a machine wrote something. But most explanations stop at "it uses AI to detect AI," which tells you nothing useful. The actual methods are measurable and specific, and so are the failure modes. Understanding how these detectors work makes you a sharper user of tools like our AI Text Detector — and helps you interpret results without over-trusting them.

What AI Detectors Are Actually Measuring

No detector reads a piece of text and asks "does this sound robotic?" That would be useless. Instead, detectors measure statistical properties of text — properties that differ, on average, between human writers and language models.

The two most commonly cited signals are perplexity and burstiness. There are others — n-gram overlap, watermark patterns, sentence length variance — but perplexity and burstiness are the core of virtually every publicly available detector today.

Here is the key insight: language models are trained to generate the most statistically predictable next token given the prior context. That optimization pressure leaves fingerprints. When a model writes a paragraph, it tends to choose words that were "expected" by another model trained on the same data distribution. Detectors exploit this.

A rough mental model:

Human writing  → high unpredictability, high variance in sentence rhythm
LLM writing    → lower unpredictability, lower variance in sentence rhythm
Detector       → score each document against both distributions → output probability

This is a probabilistic judgment, not a forensic fact. No detector produces ground truth.

Perplexity: The Probability Score Under the Hood

Perplexity is a standard NLP metric for how "surprised" a language model is by a sequence of tokens. Low perplexity means the model could have predicted this text easily. High perplexity means the text was unexpected.

Formally: perplexity = exp(-1/N * sum(log P(token_i | context)))

When a detector processes your text, it runs it through a reference model — often a GPT-2 or similar open-weights model — and computes the perplexity across the whole document. Text written by a language model tends to score lower perplexity because the generator and the evaluator share similar probability distributions.

A concrete example: take the phrase "The solution requires careful consideration of multiple variables." A language model writing about problem-solving will frequently produce something close to this — it is the kind of hedged, general phrasing that ranks highly in training data. A human author is more likely to write "You need to think about three things, and they pull in different directions." The second sentence is more surprising to a reference model, hence higher perplexity.

Typical perplexity thresholds used by detectors:

Perplexity < 30  → likely AI-generated (high confidence zone for most models)
Perplexity 30-60 → uncertain (mixed signal, needs secondary features)
Perplexity > 60  → likely human (but this is not guaranteed)

The problem is that these thresholds shift depending on the topic. Technical documentation naturally has low perplexity — it uses precise, expected vocabulary — which causes high false-positive rates on engineering writing.

Burstiness: Humans Are Inconsistent (In a Good Way)

Burstiness measures variance in sentence length and complexity across a passage. Humans write in bursts: a few short punchy sentences, then a long one with multiple clauses, then a medium one, then another short one. That rhythm is inconsistent in a way that feels natural.

Language models, when generating without constraints, tend toward a more even distribution. Each sentence comes out somewhere between 15 and 25 words. The rhythm is smooth — too smooth.

Here is a side-by-side to make this concrete:

Human passage (high burstiness):
  "Stop. That is the wrong instinct entirely.
   The reason most teams reach for a microservice architecture too early is that
   they have internalized the success stories — Netflix, Amazon, Uber — without
   reading the footnotes about when those companies actually made the switch.
   Scale first. Then split."
  Sentence lengths: 1, 1, 37, 2, 3 words → high variance

LLM passage (low burstiness):
  "Microservice architectures offer several advantages for large-scale systems.
   Teams can deploy individual services independently, which reduces risk.
   This approach also enables different services to be written in different languages.
   However, it introduces additional complexity in areas like networking and monitoring."
  Sentence lengths: 12, 11, 14, 13 words → low variance

Detectors compute a burstiness score — often the coefficient of variation of sentence lengths across a sliding window — and combine it with perplexity into a composite score. High burstiness pushes the score toward "human." Low burstiness pushes it toward "AI."

You can test this yourself in our AI Text Detector by pasting the same content written in two different styles.

Phrase Pattern Matching and N-gram Analysis

Beyond perplexity and burstiness, many commercial detectors maintain databases of high-frequency n-grams — word sequences that show up disproportionately in AI-generated text. These are not slurs or spam words; they are phrases that language models genuinely overuse because they appear frequently in high-quality training data.

Examples of flagged patterns (based on published research and open-source detector notes):

"It is important to note that..."
"In today's rapidly evolving..."
"Firstly... Secondly... Finally..."
"delve into"
"nuanced understanding"
"In conclusion, it is clear that..."
"multifaceted"

This approach is brittle — a human can write "delve into" and a well-prompted LLM can avoid it entirely — but it adds signal at low computational cost. Detectors typically combine it with perplexity and burstiness via a logistic regression or a small fine-tuned classifier.

The n-gram layer is also where detectors differ most across vendors. GPTZero, Originality.ai, and Copyleaks each train on different corpora and weight phrase patterns differently. That is one reason the same document can score 90% AI on one detector and 40% on another.

Why Accuracy Numbers Are Misleading

Detector vendors publish accuracy figures like "98% accurate" or "correctly identifies AI text 99% of the time." These numbers are almost always true in a narrow sense and misleading in practice.

The benchmark conditions that produce those numbers typically involve:

• Clean, unmodified ChatGPT output vs. clean human writing
• Balanced 50/50 datasets
• Academic or general-purpose prose
• No paraphrasing, editing, or human revision

Real-world conditions look nothing like this. In practice, most content that passes through a detector has been edited — sometimes lightly, sometimes heavily. A 2023 Stanford study found that GPT-4 output that had been minimally edited by a human writer dropped detection rates from 97% to under 50% for most open detectors.

The more honest framing is a confusion matrix:

                    Detector says AI    Detector says Human
Actual AI text      True Positive (TP)  False Negative (FN)
Actual Human text   False Positive (FP) True Negative (TN)

Typical TP rate (unedited AI):   85-98%
Typical FP rate (human text):    5-15% depending on domain
FP rate for ESL writers:         estimated 20-30% in some studies

The false positive rate on human text is the number that matters most for anyone using detection as a gatekeeping mechanism.

The False Positive Problem

The most consequential failure mode for AI detectors is flagging human writing as AI-generated. A student submits an essay; a detector calls it AI with 85% confidence; the student faces an academic integrity investigation. This has already happened hundreds of documented times.

The root cause is that the features detectors measure — low perplexity, low burstiness — are also features of certain human writing styles:

• Non-native English speakers tend to use simpler, more predictable sentence structures. This lowers burstiness and perplexity, which pushes scores toward "AI."
• Technical documentation uses precise vocabulary with low perplexity by design.
• Formulaic writing — legal boilerplate, financial disclosures, standard operating procedures — matches the statistical profile of LLM output.

The most-cited experiment: researchers submitted the U.S. Declaration of Independence through several major AI detectors. Multiple tools flagged it as likely AI-generated, with confidence scores above 70%. The document was written in 1776.

This is not a bug that will be patched. It is a fundamental consequence of the measurement approach. Any signal that distinguishes LLM output from average human writing will also fire on human writing that happens to share those statistical properties.

Our AI Text Detector surfaces a probability score, not a verdict — that framing matters. Use it to inform judgment, not replace it.

What Watermarking Changes (And What It Doesn't)

Watermarking is a newer approach that sidesteps the probabilistic problem entirely. Instead of measuring statistical properties after the fact, watermarking embeds a detectable signal into the generated text at inference time.

The most widely discussed scheme, published by researchers at the University of Maryland in 2023, works like this:

1. At each generation step, split the vocabulary into "green" and "red" token lists
   using a pseudorandom function keyed on the prior token.
2. During generation, softly bias the model toward selecting green tokens.
3. To detect: run the token sequence through the same keyed function.
   A high ratio of green tokens → watermark is present → AI-generated.

This scheme is theoretically sound: a document with a valid watermark was definitely generated by that model. No probabilistic threshold required.

The practical limitations are significant:

• Only the model owner can embed or detect the watermark. No third-party detector can use it without cooperation.
• The watermark is fragile to paraphrasing — rewriting more than roughly 30-40% of the tokens breaks it.
• It requires model-level changes; it does not retrofit to existing deployed models.
• OpenAI and Anthropic have not publicly committed to shipping watermarking in production models as of mid-2026.

For most real-world use cases, watermarking is not yet available. Statistical detection remains the operative method.

Conclusion

AI text detectors measure statistical signals — primarily perplexity, burstiness, and phrase patterns — that differ between language model output and typical human writing. Those signals are real but imperfect. Accuracy figures in marketing materials reflect clean-benchmark conditions that rarely apply to edited, domain-specific, or multilingual content. The false positive rate on human text is meaningful enough that any high-stakes decision made on detector output alone is unreliable.

The useful posture: treat detector output as one signal among several, not a verdict. If you are an editor or content manager, use tools like the AI Text Detector to flag documents for closer review — not to make final calls automatically. If you are a developer evaluating detectors for a pipeline, benchmark against your actual content domain, not the vendor's general-purpose benchmark. And if you are a writer who got flagged incorrectly, the most effective response is not to argue with the score but to provide provenance: drafts, notes, timestamps, sources. No detector score is evidence. It is a probability estimate made by a classifier trained on a particular dataset.