​
Input

  • output (str): The generated SQL query.
  • expectedOutput (str): The reference (gold standard) SQL query.

​
Output

  • Result (float): A score between 0 and 1.

​
Interpretation

  • 1: The generated query is semantically equivalent to the reference query.
  • 0: The generated query is completely different or invalid.
  • The score reflects a holistic assessment of query correctness.

​
How It Works

The evaluator performs a multi-faceted analysis of the SQL queries, considering:
  • Syntax: Is the generated query valid SQL?
  • Structure: Does it use the same tables, columns, and clauses?
  • Semantics: Is it likely to produce the same result as the reference query? This may involve comparing execution plans.
This is a similarity metric designed specifically for evaluating generated SQL.

​
Use Cases

  • Evaluating natural-language-to-SQL models.
  • Assessing AI agents that generate SQL for data retrieval.