Genie vs custom agent · same questions, same golds, same strict matcher · back to assistant
Results
When to pick each
Databricks Genie
- Fast setup: tables, a page of instructions and two example queries gave a working space.
- Governance comes from Unity Catalog: row filters, masks, comments and tags apply to the data it reads.
- Trusted assets, benchmarks and agent mode are built in.
- Best when the question is a plain query over governed tables and the people asking already work in Databricks.
Custom agent
- Claim verification: numbers in an answer are checked against the query result, apart from round bounds ("above 70 %"), bare years and fractions written as words.
- Own evals: golds from independent pandas code, a strict matcher and repeated runs.
- Investigation with planted-truth tests, and refusals as part of the product.
- App-level roles (executive, hospital admin, educator, learner) that decide what a question may return.
- Best when answers must be defended and the rules are the product's, not the platform's.
Reading the table
- The agent was tuned on nine of these ten questions over several rounds; Genie got no iteration. On questions it had never seen, the agent scored 27 of 30 (eval/final_blind_results.json).
- One run per question per system. Ten questions say little about either system; this is a first look, not a ranking.
- Genie runs as one Free Edition identity. Hospital-scoped questions name the hospital in the question; that simulates the role and proves no separation.
- The Genie space has one page of instructions, two example queries and no benchmarks. The instructions reuse our own metric definitions. It was not tuned on these questions, but one example query is the generic form of the trend question.
- Genie has no routes. A check that expects the investigation route or a refusal also marks Genie down for being a different design; the answer-content reasons are listed in the details.
- Genie's answer is its text plus the result table. Our agent's answer is its text and key numbers.
- Single values count only as the headline: the first number of the answer text (labels and dates skipped). Tables are read in the metric column, and a "no value" question fails if the answer states a rate. Genie's answers were re-scored with this matcher from the saved text, with no new Genie call.
- "not run" means that side was not run for that question; it is never a pass or a fail.