Text-to-SQL in Production: From 90% to 25% — Why the Semantic Layer Is Non-Negotiable

The Cliff: From 90% to 25%

In 2024, GPT-4o scored 86.6% execution accuracy on Spider 1.0, the classic Text-to-SQL benchmark. That number was good enough to make the industry believe natural language would replace SQL "any day now."

Two years later, the same model dropped to 10.1% on Spider 2.0, which tests enterprise-grade workflows. The databases here span thousands of columns, multiple SQL dialects, and queries that routinely exceed 100 lines. The task is no longer "write one correct query" — it is "complete an engineering-grade data exploration inside a real business context."

Then, in August 2026, an even more sobering paper landed on arXiv. Bruno Santos Teixeira's SemPlan Benchmark (arXiv:2608.13612) tested four enterprise NL2SQL architectures against 1,800 synthetic cases. Every single architecture scored between 22.25% and 25.67%. None broke a quarter.

In plain English: plug the best Text-to-SQL system into a real enterprise database, and you get roughly one correct answer for every four questions. The three wrong answers often look indistinguishable from the right ones.

Why Enterprise Databases Break LLMs

The gap between academic benchmarks and production reality is guarded by three walls that larger models alone cannot climb.

1. Schema Explosion: Context Overload

Spider 1.0 averages 6.8 tables and 72.5 columns. The BEAVER project's real-world enterprise warehouses average 101.5 tables and 869.4 columns.

Stuff ten thousand field definitions into an LLM's context window, and its attention dilutes toward randomness. Research shows that even supplying models with oracle schema-linking annotations (human hints on which tables to use) only lifts accuracy from 11.4% to 18.9%. The bottleneck is not reasoning; it is information overload.

2. Implicit Relationships: LLMs Cannot Read "Industry Slang"

Real-world table names look like t_cust_ord_dtl_2024_q3. The field amt might mean "tax-inclusive revenue" in one table and "net refund" in another — separated by an entire tax-rate definition.

Without explicit documentation, LLMs guess from naming conventions. An April 2026 arXiv error-taxonomy study analyzing 4,602 incorrect queries found that 81% of failures were schema and semantic errors — wrong columns, wrong joins, misunderstood business meaning. Syntax errors were the minority.

3. Silent Failures: The Most Dangerous Output Is a Plausible Wrong Number

The scariest Text-to-SQL failure is not an error message. It is a quietly wrong result that looks perfectly reasonable.

One engineer testing against a 35-table production-shaped database discovered that a naive query missed a deduplication step and inflated revenue by 76%. The output rendered cleanly. Nothing flagged the issue until a domain expert reviewed it.

Then there is access control. If the AI queries through a service account with broad read access, it might answer "new customers this month" by dumping the entire contact list — including PII — with no role-based filter in the way.

The dbt Labs Experiment: Semantic Layers Pull Accuracy Back to 100%

The good news? There is a fix. And it does not live inside the model. It lives in the translation layer between the model and the data.

In April 2026, dbt Labs published a reproducible benchmark. They tested pure Text-to-SQL against a fully-modeled semantic layer on ACME Insurance's 15 tables and 11 business questions, running each question 20 times:

Same models. Same questions. Same database. The only variable was the semantic layer.

dbt Labs also ran a cross-model validation: four frontier models × multiple reasoning intensities (low to max). On the semantic-layer path, most combinations hit 100%. On raw Text-to-SQL, scores hovered between 50% and 65% — throwing more inference budget at the problem did nothing.

Their conclusion was blunt: spending budget on inference tokens yields near-zero returns; spending it on semantic modeling yields thirty to forty percentage points.

dbt Labs highlighted a data point most vendors skip: on multi-hop questions outside the semantic layer's coverage, the semantic layer scored 0% — because it refused to answer — while raw Text-to-SQL guessed correctly 70-100% of the time.

This is not a bug. It is the design intent: deliver trustworthy answers inside the boundary, and say "I don't know" outside it — rather than silently fabricating a number.

OntiCards: Data Cards as the Semantic Layer

The value of semantic layers is proven. But for most enterprises, building one is painful: who writes the documentation? How do you keep it current? What happens when teams disagree on definitions?

OntiCards takes a different path: AI generates the semantic-layer draft; domain experts calibrate it.

In our architecture, this semantic capability is packaged as Data Cards — one card per data source, containing six core elements:

• Source metadata: owning workspace, source system, refresh cadence

• Field semantics: business meaning, enumerations, metric definitions

• Data quality tags: completeness, accuracy, anomaly profiles

• Access and masking rules: who can read what, which fields require masking

• Business scenario tags: which analytical contexts this data serves

• Query and addressing rules: how to query, which sources to join

When a user asks a natural-language question, the system does not throw the prompt at an LLM to write raw SQL. It first consults the Data Cards to lock in the relevant sources, clarify field semantics, and check permission boundaries. The entire pipeline is driven by the Data Card system, not by a naked model guessing its way through your warehouse.

This aligns with dbt Labs' semantic-layer philosophy, but the implementation is lighter: instead of asking data teams to spend three months writing dbt models, AI generates the initial card set on the day a data source is connected. FDE engineers then iterate and refine semantics over time. Ship first, model second, grow continuously.

In our e-commerce data QA walkthrough, this card-driven NL2SQL already handles cross-table JOINs, time-window aggregations, and metric-definition transformations reliably. The system knows that "sales amount" is net_revenue in the orders table and must subtract refund_amount from the refunds table — it does not leave that guess to the LLM.

A Five-Question Checklist for Engineering Leaders

If you are evaluating an enterprise natural-language analytics solution, filter vendors with these five questions:

1. With 50+ tables, how does the system keep the query chain stable? Anyone relying on stuffing the full schema into the context window will break in production.

1. Does the model fail with an error, or does it fail silently? The latter is ten times more dangerous.

1. Who maintains the semantic layer? Pure manual documentation does not scale; pure AI generation is not trustworthy. Human-AI collaboration is the only viable path.

1. Ask the same business metric three times. Do you get the same number? Consistency is non-negotiable.

1. When a question exceeds the system's coverage, does it refuse or hallucinate? A system that knows its own boundaries is the only kind worth trusting.

Closing

Text-to-SQL is not a fake problem. Letting business users query data in natural language is the right direction for data democratization. But the path of "raw model writes SQL directly against production tables" has been falsified by multiple independent studies.

Model capabilities are improving fast. Enterprise data complexity is improving faster. What determines whether an answer is correct is not how many parameters the model has. It is whether the model has been given an accurate map of the enterprise data landscape.

The semantic layer — whether you call it a semantic layer, Data Cards, or a business metric registry — is that map. Without it, even the most powerful LLM is just a driver navigating an unfamiliar city with no GPS.

References

• Bruno Santos Teixeira, SemPlan: Benchmarking Structured Semantic Planning for LLM-Based Queries over Enterprise Data, arXiv:2608.13612, 2026-08-12. https://arxiv.org/abs/2608.13612

• dbt Labs, Semantic Layer vs. Text-to-SQL: 2026 Benchmark Update, 2026-04-07. See Atlan's analysis: https://atlan.com/know/ai-agent/data-for-ai/text-to-sql-for-enterprise/

• Text-to-SQL Explained: Why It Fails in Production and How to Fix It, Solution Gigs, 2026. https://solutiongigs.in/blog/text-to-sql

• The Semantic Layer: An Operator's Guide to AI Answers You Can Defend, Nufar Gaspar, 2026. https://nufargaspar.com/writing/semantic-layer-operators-guide

• Why Text-to-SQL Fails in the Enterprise, bits-bytes-nn, 2026-07. https://bits-bytes-nn.github.io/insights/data-architecture/2026/07/27/ai-ready-data-semantic-layer-knowledge-graph-en.html

• 语义层不是文档，是编译器 (Semantic Layers Are Compilers, Not Docs), 夜雨聆风, 2026. https://www.yeyulingfeng.com/a/921179.html