Why LLMs Are Unreliable Language Detectors
LLMs can translate, explain grammar, and write fluently in dozens of languages — so it seems obvious they should be great at simply identifying what language a text is written in. And for long, clean text in common languages, they usually are. But things fall apart at the edges that show up constantly in real-world use: short snippets, closely related languages, rare languages, mixed-language text, messy input, and situations where you actually need the system to say "I'm not sure."
The problem isn't that LLMs don't understand language. It's that being able to talk fluently about a task isn't the same as reliably performing that task the way a production system needs.
Generating text isn't the same as classifying it
A real classifier picks from a fixed list of languages by scoring each one and returning the highest score. An LLM works differently: it generates a response one word-piece at a time, predicting each piece based on everything before it. So when you ask "What language is this?", it's not running a classification routine — it's just producing a plausible-sounding answer. The label you want is buried inside a much bigger, open-ended generation process, and the answer can shift depending on prompt wording, prior examples, conversation history, and settings like randomness (temperature).
LLMs can become good classifiers — through fine-tuning, restricting outputs to fixed labels, or scoring token probabilities — but a bare, free-form answer doesn't come with the fixed labels, reliable confidence, or "I don't know" option a real system needs.
Short texts don't contain enough evidence
Take words like "no," "menu," "radio," "hotel," "gift," "die," or "chat." All are valid in multiple languages, sometimes with different meanings. No system can reliably guess the language from a genuinely ambiguous word — that's a fundamental information problem, not an LLM flaw.
The danger is what happens next: a real classifier can expose a score you can threshold, but a chat-tuned LLM tends to just confidently pick something. That's a risky combination — a nearly meaningless input can produce an answer that looks confident. The honest output for "radio" isn't "Spanish" — it's "unknown."
Closely related languages are genuinely hard
Some language pairs are notoriously easy to confuse: Serbian/Croatian/Bosnian/Montenegrin, Indonesian/Malay, Czech/Slovak, Norwegian/Danish, Russian/Ukrainian/Belarusian, Spanish/Galician/Portuguese, Afrikaans/Dutch, Persian/Dari. These share huge amounts of vocabulary and grammar, and some cases are more about how you define your labels than pure linguistics.
LLMs add another wrinkle: they can pick up on what a text is about — a place name, a person, a topic — and use that as a shortcut for guessing the language, even though topic isn't real linguistic evidence.
Script isn't language
Recognizing an alphabet is much easier than recognizing a language. Cyrillic could mean Russian, Ukrainian, Bulgarian, Belarusian, Serbian, or Kazakh. Arabic script covers Arabic, Persian, Urdu, Pashto, Sindhi, Uyghur, and more. A detector can look smart just by spotting the script and guessing whichever language is most common for it — without real evidence. When many languages share a script, the guess quietly falls back on which language is simply more frequent overall, which is especially unfair to lower-resource languages.
Tokenization can hide useful clues
Traditional language detectors lean heavily on letter patterns — spelling, suffixes, diacritics. LLMs process text in word-pieces (tokens) rather than raw letters. This isn't a total information loss, but it changes how easily those fine-grained clues are used, especially for informal spelling, missing accent marks, or low-resource languages. For instance, stripping Polish diacritics (ą, ę, ł, ś, ź, ż) removes some of its most distinctive markers, making it easier to confuse with other Slavic languages using the same alphabet.
Mixed-language text breaks the "one label" assumption
Real text is messy — a support ticket might mix two languages with URLs and error codes; a social post might switch languages mid-sentence. Forcing that into a single label is already a design decision. Should the output be one language per document, per sentence, per span, or a percentage breakdown? These are genuinely different tasks, and an LLM usually won't tell you which strategy it used to pick one language over another.
Rare languages get squeezed out
LLMs are trained on wildly uneven amounts of data — massive amounts of English, Spanish, French, Chinese, and comparatively little for many minority languages. When evidence is ambiguous, models tend to default toward the more common, higher-resource language nearby. Over time, this can systematically erase or misclassify minority-language content — a real problem if language detection feeds into search, moderation, or dataset-building pipelines.
"92% confidence" often isn't real confidence
If an LLM says "Slovak — 92% confidence," that number can just be generated text, not a real calibrated probability. Ask the same question with a slightly different prompt and the number may change. Genuine calibrated confidence requires deliberately building it — for example, restricting outputs to fixed labels and calibrating the resulting scores on real test data.
Production systems need to know when to say "I don't know"
Many demos are built to always answer something. Real systems often need the opposite — the ability to abstain. Returning "unknown, confidence 0.31" is often more useful than a falsely confident guess. Good abstention thresholds should be tuned on realistic data, since they behave differently for short queries vs. long documents, or for different scripts.
What actually works better
For most real applications, a purpose-built classifier — like fastText, CLD3, lingua, or GlotLID — is faster, cheaper, and more predictable than a general LLM. A solid pipeline: keep the raw text, normalize carefully without losing diacritics, separate real language content from URLs/code, decide your labeling unit up front, use script detection only as a narrowing step, run a validated classifier, apply tested confidence thresholds, allow "unknown" answers, route hard cases to humans or an LLM, and monitor errors by language — since overall accuracy can hide serious failures on rare languages and short text.
The core lesson: a model that can talk fluently about a task isn't automatically a model that can perform that task reliably.