We asked 50 business questions: Sovarium vs a frontier model alone

What did the benchmark find?

Sovarium answered 49 of 50 business questions correctly, 98.0%. GPT-5.6 Sol, a frontier model answered 22 or 44.0%.

The questions are what you’d expect a stakeholder to ask a data analyst: revenue by region, refunds last year, active customers, ad spend and return on it. The underlying data is a two sided marketplace business that covers retail and ecommerce.

The gap was widest where a correct answer depends on a business rule the schema does not state.

What did we test?

We tested two approaches on the same 50 questions, warehouse and judge.

The only difference was the Context Layer: the written descriptions of what each table and column means, how tables relate, and the business context.

What data did the agents query?

Orion is a two sided online marketplace: buyers and sellers in the US, UK and EU, with orders, payments, refunds, shipments, ad campaigns, payouts and reviews spread across five schemas.

Like any business’ data Orion can be challenging for any AI to query because of things like:

• Refunds recorded two ways: in a refund table, and as negative order lines.
• Bare amount columns in three currencies, with the currency stored on a parent row.
• Guest checkouts linked to later accounts only by a hashed email.
• An order status that misses about 15% of cancellations because a nightly sync fails.

Which 50 questions did we ask?

We asked 50 questions that a typical business would ask from a data analyst.

How were the answers graded?

Each answer was graded against a ground truth computed from the warehouse, first by fixed rules and then by an LLM judge, whose verdicts we then checked by hand.

The judge, at temperature 0, sees the question, the expected answer, the agent’s full text, its charts and every SQL statement it ran. It compares the output to the expected answer.

Where does AI alone go wrong?

Business data is usually messy and the definitions and logic aren’t written down anywhere. AI only tools look at table and column names to try and work out how they link together and what they mean. The problem with that is schemas aren’t designed to be descriptive. A column named “amount” doesn’t tell you whether its including or excluding VAT, gross or net, or whether it includes cancellations.

That’s the kind of information that the Context Layer provides and its how our agent is so accurate. A few examples of failed questions are listed below:

1. Guest shoppers who later registered

Question: Of the 1,336 guest shoppers first seen through 2025, how many had registered an account by mid-2026?

GPT-5.6 Sol joined on registered_cust_id and answered 247.

Our Context Layer noted that the column is “Filled by an occasional manual reconciliation script, and only for about half of the guests who actually registered - do not rely on it; join email_hash directly to find guest-to-registered matches.”

Sovarium matched on the email hash and answered 476 correctly. The AI only answer was off by around half.

2. Refunds in 2025

GPT-5.6 Sol summed the refund table: $26,926.74, €11,379.71 and £8,729.01 across 571 refunds.

Our Context Layer noted for that table: “NOT the complete refund population: ~30% of refund events exist only as negative order lines in ord_ln with no row here”.

Sovarium added the negative order lines, counted refunds booked in both places once, and answered $67,513 in USD. The AI only answer was around 25% off.

3. Active customers

GPT-5.6 Sol counted accounts whose status code reads active: 6,690.

Our Context Layer’s note on that status column: “Set at signup and by a monthly batch that lags up to ~5 weeks; never recomputed from order history, so it is not a reliable activity signal.” The layer defines an active customer as one with a real, paid order in the last 365 days, after folding guests and merged accounts into one identity.

Sovarium applied that definition and got the correct answer: 1,785.

What should buyers take from it?

It’s important to not just point an AI at your warehouse. Models are extremely good at writing SQL queries but without a semantic layer they lack context and accuracy drops significantly.

The result agrees with what Anthropic’s data team reported in June 2026: Claude answered no more than 21% of its analytics evaluation questions correctly without skills, and consistently above 95% in aggregate once skills were added.