Evals

Evals turn a model into something you can regression-test. You define a suite of cases, each with assertions about the output, run the suite against a model, and get a pass rate back. Run it again later and Sable flags a regression when the pass rate dropped versus the previous run.

Runs are real inference: each case is a genuine metered call on your prepaid balance, so the pass rate reflects the model as it actually behaves. What is stored is only the score, the pass/fail of each case, and which assertions failed. The model output itself is never stored, in keeping with the privacy contract.

Evals are session-authed: they run behind your Sign-In With Ethereum dashboard session, and there is a portal page for them at Portal → Evals.

Assertions

Each case carries one or more assertions. An assertion has a type and a value:

Type	Passes when the output...
`contains`	Passes when the output...contains `value` as a substring (case-sensitive).
`icontains`	Passes when the output...contains `value`, case-insensitively.
`not_contains`	Passes when the output...does not contain `value`.
`equals`	Passes when the output...equals `value` exactly.
`regex`	Passes when the output...matches the regular expression `value`.
`min_length`	Passes when the output...is at least `value` characters long.

A case passes only when all of its assertions pass.

Quickstart

# 1. Create a suite
curl https://api.buildsable.com/v1/evals \
-H "authorization: Bearer $SABLE_SESSION_TOKEN" \
-H 'content-type: application/json' \
-d '{
  "name": "support-tone",
  "model": "sable",
  "cases": [
    {
      "input": "A customer asks for a refund. Reply in one sentence.",
      "assertions": [
        {"type": "icontains", "value": "refund"},
        {"type": "min_length", "value": 20}
      ]
    }
  ]
}'

# 2. Run it (real, metered inference)
curl -X POST https://api.buildsable.com/v1/evals/$EVAL_ID/run \
-H "authorization: Bearer $SABLE_SESSION_TOKEN"

A run reports the pass rate, whether it regressed, the cost, and the per-case outcome, including which assertions failed:

{
  "run_id": "run_5f2a…",
  "total": 1,
  "passed": 1,
  "pass_rate": 1.0,
  "regression": false,
  "cost_micro_usd": 240,
  "results": [
    { "index": 0, "passed": true, "failed_assertions": [] }
  ]
}

Tracking regressions over time

GET /v1/evals/{id}/runs returns the run history with each run's pass rate, so you can watch a suite over time and see exactly when a model change moved the number. The regression flag on a run is set whenever its pass rate is lower than the run before it, which is the signal to look at what changed.

Endpoints

All session-authed (Authorization: Bearer sess_…).

Method	Path	What it does
POST	Path`/v1/evals`	What it doesCreate a suite. Body `{name, model, cases}`.
GET	Path`/v1/evals`	What it doesList your suites.
DELETE	Path`/v1/evals/{id}`	What it doesDelete a suite.
POST	Path`/v1/evals/{id}/run`	What it doesRun the suite against its model. Returns pass rate, regression, cost, and per-case results.
GET	Path`/v1/evals/{id}/runs`	What it doesRun history with pass rate over time.

Privacy and limits

Output is never stored. A run keeps only the pass/fail per case and which assertions failed. The model's actual output is never persisted, exactly as with any other inference.
Runs cost real money. Every case is a genuine metered inference call on your balance. A large suite run repeatedly is real spend, so scope suites deliberately.
Regression is relative. regression: true means this run scored lower than the previous run of the same suite. The first run of a suite has nothing to compare against.