Genie vs custom agent · same questions, same golds, same strict matcher · back to assistant

Results

When to pick each

Databricks Genie

  • Fast setup: tables, a page of instructions and two example queries gave a working space.
  • Governance comes from Unity Catalog: row filters, masks, comments and tags apply to the data it reads.
  • Trusted assets, benchmarks and agent mode are built in.
  • Best when the question is a plain query over governed tables and the people asking already work in Databricks.

Custom agent

  • Claim verification: numbers in an answer are checked against the query result, apart from round bounds ("above 70 %"), bare years and fractions written as words.
  • Own evals: golds from independent pandas code, a strict matcher and repeated runs.
  • Investigation with planted-truth tests, and refusals as part of the product.
  • App-level roles (executive, hospital admin, educator, learner) that decide what a question may return.
  • Best when answers must be defended and the rules are the product's, not the platform's.

Reading the table