When does data help automated agent engineering?

The top-level instruction the synthesis agent receives:

---
name: dataset-synthesis
description: Synthesize representative inferences and feedback for an LLM application described by a TensorZero configuration. Use when a plausible baseline corpus is needed for a function that has not yet collected real data.
---

# TensorZero Dataset Synthesis

You are synthesizing a _plausible_ dataset for a TensorZero function. You will produce two JSONL files that look like what `inferences.jsonl` and `feedback.jsonl` _would_ contain after the function had run live for a while. Crucially, you do **not** have any real baseline data to draw from, but the configuration files should provide you with enough information about the application to generate sensible examples.

## Environment

- T0 config files: `{config_dir}/`
- Gateway URL: `{gateway_url}` (you may POST to `/inference` to spot-check your understanding of the input structure)
- Output directory: `{output_dir}/` — write `inferences.jsonl` and `feedback.jsonl` here
- Isolated container. No Python or `pip`; `node` and `curl` are on `$PATH`; `jq` is not installed. Use `node -e "..."` for JSONL parsing.
- Emit rows with `variant_name: "initial"` only.

## Task

- Function: `{function_name}`
- Metrics defined for this function: `{metric_name_list}` (read their `kind`, `level`, and `optimize` fields from `tensorzero.toml`)
- Budget: at least `{min_episodes}`, but no more than `{max_episodes}` episodes. An episode is one logical interaction with the function — a single inference for single-turn functions, a chain of inferences sharing one `episode_id` for multi-turn.
- Output files:
  - `{output_dir}/inferences.jsonl` — one row per inference call (see reference/inferences_schema.md)
  - `{output_dir}/feedback.jsonl` — one row per metric value, with `target_id` referring to the `inference_id` or `episode_id` of a row in the inferences file (see reference/feedback_schema.md)

## Workflow

Five steps. See reference/methodology.md for the long form; the short version:

1. **Read the spec.** Open `{config_dir}/tensorzero.toml` and the linked schema / template files. Note: input schema, output schema, function type (chat vs tool), defined metrics, and whether the function is one-shot or part of a multi-turn episode.
2. **Hypothesize the input distribution.** What kinds of users / states does this function see in deployment? Sketch a coverage plan: how many length buckets, which schema slots vary, which edge cases matter. Aim for diversity, not just a single canonical mode.
3. **Generate inputs.** Plan out at least `{min_episodes}` but no more than `{max_episodes}` distinct episodes. For multi-turn functions, decide each episode's length up front based on what's realistic for the task (a 4-turn episode contributes 4 rows sharing one `episode_id`).
4. **Spot-check via the gateway.** Periodically POST a synthetic input to `{gateway_url}/inference` to confirm your understanding of the input structure is correct and to see what the `initial` variant's output actually looks like. Gateway calls are expensive — treat this as a calibration step, not as the way to generate every row. Generate outputs yourself in between checks.
5. **Generate feedback rows.** For each metric in `{metric_name_list}`, emit one feedback row per appropriate target (per-inference or per-episode based on the metric's `level`).

After generating, validate:

```bash
node /skill/scripts/validate.js {output_dir} \
  --config {config_dir}/tensorzero.toml \
  --min-episodes {min_episodes} --max-episodes {max_episodes}
```

The validator checks schema compliance, referential integrity, budget, and the `variant_name == "initial"` invariant. Fix any errors it reports before exiting.

## Output contract

When you exit, `{output_dir}/` must contain exactly:

- `inferences.jsonl` — every row conforms to the schema in `reference/inferences_schema.md`; the rows span at least `{min_episodes}` and at most `{max_episodes}` distinct `episode_id`s
- `feedback.jsonl` — one or more rows per metric, with every `target_id` referring to an `id` (for inference-level metrics) or `episode_id` (for episode-level metrics) that exists in `inferences.jsonl`

Do not write any other files in `{output_dir}/`. Do not modify `{config_dir}/`. Stay within budget — don't issue gateway calls indefinitely.

## Principles

- **Quality of coverage beats quantity of duplicates.**
- **Use the gateway as a calibration tool.** A periodic `/inference` call confirms your understanding of the input structure and shows you what the `initial` variant actually emits. It's not a way to generate every row — gateway calls are expensive, and it's fine to generate outputs yourself between checks.
- **Don't peek.** You don't have baseline data. If you find yourself wanting to "look at a real example," that's the signal to make a better-reasoned guess from the spec instead.
- **Plausibility includes failure.** Some inferences will fail their metric. Your feedback distribution should reflect a realistic failure rate for the task — not 100% success.

The longer methodology the agent can consult:

# Synthesis methodology

The recipe for producing a faithful `inferences.jsonl` + `feedback.jsonl` from spec alone. Five steps, each with concrete things to look for.

## 1. Read the spec carefully

Open `{config_dir}/tensorzero.toml`. For the target function, capture:

- **Type**: `chat` vs `tool` / `json`. This determines `output` shape.
- **Schemas**: input (per-role or named), output (for tool-call functions). Read every referenced `.json` and every `.minijinja` template.
- **Metrics**: which are defined, their `type`, `level`, `optimize`. These dictate the `feedback.jsonl` rows you'll write.
- **System prompt**: usually inside the variant's template. Read it — this is the strongest signal about what the function is for.
- **Tool list** (for tool functions): names, descriptions, argument schemas.

Two patterns to check early:

```bash
# What kind of function?
grep -A2 "^\[functions\.\"{function_name}\"\]" {config_dir}/tensorzero.toml

# Which metrics are defined?
grep -E "^\[metrics\." {config_dir}/tensorzero.toml
```

## 2. Hypothesize the input distribution

Before generating anything, sketch a plan. For each schema slot in the input:

- What value ranges / shapes does it plausibly take in deployment?
- Are there subpopulations (long vs short, simple vs nested, single vs multi-entity)?
- What's the realistic length / complexity distribution?

Write the plan as a comment-level outline before the first row. Something like:

```
Plan for {function_name}
(budget: at least {min_episodes}, at most {max_episodes} episodes)
  - Target ~N episodes × K turns each
  - Mix of the major user intents the function supports
  - Vary user tone / register across episodes
  - Cover authentication / setup steps the function expects before the main action
```

Don't skip this step. Generating without a plan reliably produces a stack of near-duplicates of the same canonical input.

## 3. Generate inputs

For each row in your plan:

- Construct the `input.messages` array per the schema rules in inferences_schema.md.
- For multi-turn: episode by episode. Within one episode, mint a fresh `episode_id`, then chain inferences — each turn's `input.messages` is the previous turns' `input` plus `assistant` reply plus next user turn.

Tools you'll use:

- `node -e` for any structured generation (writing JSON bodies, looping, minting UUIDs).
- A working directory in `/tmp` for intermediate files (probe bodies, response captures).
- `curl` to call the gateway.

## 4. Spot-check via the gateway

Periodically — not on every row — POST a synthetic input to the gateway and look at the response. The purpose is calibration, not generation:

- Confirm the input shape you've been building actually parses (template name correct, schema arguments well-formed).
- See what the `initial` variant's output structure looks like for that input, so the outputs you generate yourself stay faithful to it.
- Catch drift early — if the first spot-check shows your `arguments` object missing a required field, fix the generator before producing more rows.

```bash
node -e "
  const body = { /* function_name, variant_name: 'initial', input: ... */ };
  process.stdout.write(JSON.stringify(body));
" > /tmp/req.json

curl -sf {gateway_url}/inference \
     -H 'Content-Type: application/json' \
     --data @/tmp/req.json > /tmp/resp.json
```

A reasonable cadence: one spot-check before you start generating, one after the first episode, and one every ~5 episodes thereafter. Cheaper than per-row, sufficient to catch most schema mistakes.

Assemble each inference row from:

- Your minted `id` and `episode_id`
- The current `created_at`
- `"initial"` as `variant_name`
- Your `input` from step 3
- An `output` you write yourself, matching the structure you saw in the spot-checks (gateway response → guide for your own generation)

For multi-turn: within one episode, each turn's `input.messages` extends the previous turn's by appending the assistant reply and the next user message. Keep the chain coherent across turns of the same `episode_id`.

Write each row to `{output_dir}/inferences.jsonl` immediately — don't batch, so a crash mid-run preserves progress.

## 5. Generate feedback rows

For each metric in `{metric_name_list}`:

- Determine its `level` (inference vs episode) from the TZ config.
- For inference-level: walk every inference row and emit one feedback row per (inference, metric).
- For episode-level: walk every distinct `episode_id` and emit one feedback row per (episode, metric).

For the `value`:

- **If the metric is verifiable from the row alone** (e.g. exact_match against a known gold answer, or a length / format check), compute it programmatically.
- **Otherwise**, predict the value from input + output using your understanding of the task. Stay calibrated — see "realistic value distributions" in feedback_schema.md.

Write to `{output_dir}/feedback.jsonl`.

## Iteration / self-audit

After roughly a third of the planned episodes, stop and inspect what you've produced:

```bash
# Count rows and unique episodes
wc -l {output_dir}/inferences.jsonl
grep -o '"episode_id":"[^"]*"' {output_dir}/inferences.jsonl | sort -u | wc -l

# Variety of input templates / first 80 chars
node -e "
  require('readline').createInterface({input: require('fs').createReadStream('{output_dir}/inferences.jsonl')}).on('line', l => {
    const r = JSON.parse(l);
    const m = r.input.messages[0];
    const c = Array.isArray(m.content) ? m.content[0] : m.content;
    const s = typeof c === 'string' ? c : JSON.stringify(c.arguments || c);
    console.log(s.slice(0, 80));
  });" | sort -u | head -20
```

Ask:

- Am I converging on one mode? (lots of near-identical first lines)
- Did I cover all the schema slots I planned for?
- Does my feedback distribution look reasonable?

If yes to mode collapse — diversify the remaining episodes by deliberately picking cases that look different from what's there. You have room to add more episodes up to `{max_episodes}`; you do not have to stop at the planned count if your coverage feels thin.

## When to stop and validate

Once you've reached at least `{min_episodes}` episodes (and no more than `{max_episodes}`) with each metric covered, run:

```bash
node /skill/scripts/validate.js {output_dir}
```

Read its output and fix any errors. Then exit.

## Anti-patterns

- **Skipping step 2.** "I'll just start generating" gives mode collapse 100% of the time.
- **Skipping step 4 entirely.** Without any spot-checks you have no signal that your input shape parses or that your outputs resemble what the model actually emits.
- **Treating all episodes as length 1.** For multi-turn functions, single-turn episodes are _unrepresentative_.
- **Generating one giant batch and writing at the end.** Write incrementally so a crash doesn't lose work.
- **Ignoring the metric `level`.** Inference-level vs episode-level changes which `target_id` you reference.

The schema for inferences.jsonl rows:

# `inferences.jsonl` row schema

One JSON object per line. Every row represents one call to `/inference` against the function. Multiple rows can share an `episode_id` (multi-turn episodes).

## Fields

| field          | type                   | required | notes                                                                                                                                  |
| -------------- | ---------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `id`           | string (UUID v7)       | yes      | Unique per inference. UUID v7 sorts by timestamp — see "minting UUIDs" below.                                                          |
| `episode_id`   | string (UUID v7)       | yes      | One UUID per logical episode. For single-turn functions, this is fresh per row. For multi-turn, all rows in the same episode share it. |
| `created_at`   | string (ISO 8601, UTC) | yes      | E.g. `"2026-05-15T18:42:11.123Z"`. Should be monotonic within an episode.                                                              |
| `variant_name` | string                 | yes      | Always `"initial"` for this skill.                                                                                                     |
| `input`        | object                 | yes      | `{"messages": [...]}` — the request body's `input` field. See "input shape" below.                                                     |
| `output`       | array                  | yes      | The gateway's response content blocks. Shape depends on function type. See "output shape" below.                                       |

## Minting UUIDs

UUID v7 is required because TensorZero uses the embedded timestamp to order rows.

```js
// Node-only UUID v7 minter (no external deps)
function uuidv7() {
  const ts = BigInt(Date.now());
  const tsHex = ts.toString(16).padStart(12, "0");
  const rand = crypto.randomBytes(10);
  rand[0] = (rand[0] & 0x0f) | 0x70; // version 7
  rand[2] = (rand[2] & 0x3f) | 0x80; // RFC 4122 variant
  const r = rand.toString("hex");
  return `${tsHex.slice(0, 8)}-${tsHex.slice(8, 12)}-${r.slice(0, 4)}-${r.slice(4, 8)}-${r.slice(8, 20)}`;
}
```

For an episode of N turns, mint one `episode_id`, then mint N `id`s, advancing `created_at` by ~1s between them.

## Input shape

The `input.messages` field follows the standard chat-message format. Each message is `{role, content}` where:

- `role`: `"system" | "user" | "assistant"`
- `content`: either a string (rare) or an array of content blocks

The most common content block for a templated function is:

```json
{
  "type": "template",
  "name": "<template_name>",
  "arguments": {
    /* object matching the schema */
  }
}
```

`<template_name>` and the `arguments` shape come from the TZ config. Two co-existing styles:

**Legacy (per-role schemas):**

```toml
[functions."my_fn"]
user_schema = "functions/my_fn/user_schema.json"

[functions."my_fn".variants.initial]
user_template = "functions/my_fn/initial/user_template.minijinja"
```

`<template_name>` is the role name (`"user"`, `"system"`, `"assistant"`).

**New (named schemas):**

```toml
[functions."my_fn"]
schemas.user_query.path = "functions/my_fn/user_query_schema.json"

[functions."my_fn".variants.initial]
templates.user_query.path = "functions/my_fn/initial/user_query.minijinja"
```

`<template_name>` is the key under `schemas.` / `templates.` (e.g. `"user_query"`).

For roles that have no schema, use either `"content": "Hello"` or `[{"type":"text","text":"Hello"}]`.

> **Filesystem path mangling**: function and tool names containing `::` (e.g. `"my_function::act"`) appear in the TZ config as `[functions."my_function::act"]`, but on disk the corresponding directory is `functions/my_function____act/` (four underscores). When reading template / schema files, translate `::` → `____` in the path. A quick `find /config -type f` confirms the actual layout if you're unsure.

### Example: a templated user input

```json
"input": {
  "messages": [{
    "role": "user",
    "content": [{
      "type": "template",
      "name": "user",
      "arguments": { "observation": "Hello, this is a sample user message." }
    }]
  }]
}
```

For a multi-turn episode, append assistant + tool result messages between user turns. The third turn's `input.messages` will hold 5 entries (system?, user₀, assistant₀, user₁, assistant₁).

## Output shape

Depends on the function's `type` in the TZ config — there are three forms.

**`type = "chat"`** — list of content blocks:

```json
"output": [
  { "type": "text", "text": "The model's reply." }
]
```

Tools and text can mix in the same list (a text block followed by a `tool_call`, or several `tool_call`s).

**`type = "chat"` with tools** — same list, with `tool_call` blocks:

```json
"output": [
  {
    "type": "tool_call",
    "name": "<tool_name>",
    "arguments": { /* matching tool schema */ }
  }
]
```

Real rows often include extra fields like `id`, `raw_name`, `raw_arguments` carried back from the underlying model API. Reproduce only `type` + `name` + `arguments` unless you also call the gateway; the extras are post-hoc.

**`type = "json"`** — a single object with `raw` (the unparsed string) and `parsed` (the matched JSON):

```json
"output": {
  "raw": "{\"person\": [], \"location\": [\"Japan\"]}",
  "parsed": {
    "person": [],
    "location": ["Japan"]
  }
}
```

`parsed` must conform to the function's `output_schema`. `raw` is the literal string the model emitted; usually it's just `JSON.stringify(parsed)` with whatever whitespace the model used.

If you're not sure which form applies, look at `[functions."<fn>"]` in `tensorzero.toml` — the `type` field tells you.

## Common mistakes

- **`id == episode_id`.** They must be distinct UUIDs even for single-turn functions.
- **String content where the schema expects template.** If the function has a `user_schema.json`, the user message MUST use `{"type":"template", "name":"user", "arguments":{...}}` — a plain string will be rejected.
- **`variant_name` set to something other than `"initial"`.** This skill only emits `initial`-variant rows; we're characterizing the baseline distribution.
- **Outputs invented by hand.** Always ground via the gateway (see methodology.md). A hand-written `tool_call` argument is very likely to drift from how the model actually phrases things.
- **`created_at` in the wrong format.** ISO 8601 UTC, either with the `Z` suffix (e.g. `"2026-05-15T18:42:11.123Z"`) or the explicit `+00:00` offset. Non-UTC timezone offsets are rejected.

The schema for feedback.jsonl rows:

# `feedback.jsonl` row schema

One JSON object per line. Every row represents one piece of feedback associated with either a single inference or a whole episode.

## Fields

| field         | type          | required | notes                                                                                                                |
| ------------- | ------------- | -------- | -------------------------------------------------------------------------------------------------------------------- |
| `kind`        | string enum   | yes      | One of `"boolean"`, `"float"`, `"comment"`, `"demonstration"`. Determines the `value` type.                          |
| `metric_name` | string        | yes      | Must match a metric defined under `[metrics.<name>]` in `tensorzero.toml`.                                           |
| `target_id`   | string (UUID) | yes      | Resolves to an `inferences.id` (for inference-level metrics) or `inferences.episode_id` (for episode-level metrics). |
| `value`       | varies        | yes      | Type depends on `kind`. See below.                                                                                   |

## Reading the metric definition

For each metric you emit feedback for, locate its definition in the TZ config:

```toml
[metrics.exact_match]
type     = "boolean"        # → kind in feedback row
level    = "inference"      # → target_id resolves to inferences.id
optimize = "max"            # informational; bigger value is better

[metrics.cost]
type     = "float"
level    = "episode"        # → target_id resolves to inferences.episode_id
optimize = "min"
```

Three rules that drop out of this:

- **`kind`** in the feedback row matches **`type`** in the metric definition.
- **`level = "inference"`** ⇒ `target_id` is one of the `id`s in `inferences.jsonl`. One feedback row per (inference, metric) pair.
- **`level = "episode"`** ⇒ `target_id` is one of the `episode_id`s. One feedback row per (episode, metric) pair.

## `value` shape by `kind`

| kind            | type        | example                      | notes                                                                |
| --------------- | ----------- | ---------------------------- | -------------------------------------------------------------------- |
| `boolean`       | bool or 0/1 | `true`, `false`, `1`, `0`    | Both forms are accepted; prefer `true` / `false`.                    |
| `float`         | number      | `0.73`, `12.4`               | Range is metric-defined — read its bounds from the TZ config if any. |
| `comment`       | string      | `"Failed: incorrect output"` | Natural-language feedback from users or developers.                  |
| `demonstration` | object      | `{ "output": [...] }`        | Edited drafts, labels, human-generated content.                      |

For this skill, focus on `boolean` and `float` — they're the metrics that drive optimization.

## Examples

**Inference-level boolean:**

```json
{
  "kind": "boolean",
  "metric_name": "exact_match",
  "target_id": "<inference_id>",
  "value": false
}
```

**Episode-level float:**

```json
{
  "kind": "float",
  "metric_name": "reward",
  "target_id": "<episode_id>",
  "value": 0.42
}
```

## Realistic value distributions

You don't have ground-truth labels, but you should produce a feedback distribution that's _plausible_ for the task — not 100% success and not 100% failure.

For a boolean metric:

- A 100% success rate is a red flag — it suggests you tilted your synthetic inputs toward easy cases. Re-balance.

For a float metric:

- Bound by the metric's natural range (often [0, 1] for accuracy-style or unbounded for cost / reward).
- Distribute across the range — don't pile everything at the mean.
- If you don't know what the natural range is, generate a few real outputs first via the gateway and inspect them.

The point of this corpus is to be a _prior_ over what the function's baseline behavior looks like — it does not need to be correct, but it must be plausible. The downstream measurement (input/output/feedback novelty against the real baseline) will surface where the prior was wrong.

## Common mistakes

- **`target_id` points at an `episode_id` for an inference-level metric (or vice versa).** Read the metric's `level` first.
- **`kind` mismatched with `metric.type`.** A `float` metric must receive `kind: "float"` feedback rows, even if the values look 0/1.
- **`metric_name` not in the TZ config.** Emitting feedback for a metric the function doesn't define will fail validation.
- **Missing rows.** Every inference should be covered by at least one feedback row from an inference-level metric, and every episode by at least one episode-level metric (if any are defined). The validator counts coverage.

The validator the agent runs before exiting:

#!/usr/bin/env node
/**
 * Validate a dataset-synthesis run's output. Mirrors the contract from the
 * skill's reference docs.
 *
 * Usage:
 *   node validate.js <output_dir> [--config <tensorzero.toml>]
 *                                 [--min-episodes <N>] [--max-episodes <N>]
 *
 * Exits 0 on success, 1 on any error. Errors go to stderr; the summary line
 * ("PASS" or "FAIL — N error(s):") and per-file counts go to stdout so the
 * agent can `> validate.log 2>&1` for a single file.
 */
"use strict";

const fs = require("fs");
const path = require("path");

const UUID_RE =
  /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
const ALLOWED_FEEDBACK_KINDS = new Set([
  "boolean",
  "float",
  "comment",
  "demonstration",
]);
const ALLOWED_OUTPUT_TYPES = new Set([
  "text",
  "tool_call",
  "raw_text",
  "thought",
]);
const REQUIRED_INF_FIELDS = [
  "id",
  "episode_id",
  "created_at",
  "variant_name",
  "input",
  "output",
];
const REQUIRED_FB_FIELDS = ["kind", "metric_name", "target_id", "value"];

// ── Arg parsing ──────────────────────────────────────────────────────────

function parseArgs(argv) {
  const args = {
    outputDir: null,
    config: null,
    minEpisodes: null,
    maxEpisodes: null,
  };
  for (let i = 0; i < argv.length; i++) {
    const a = argv[i];
    if (a === "--config") args.config = argv[++i];
    else if (a === "--min-episodes") args.minEpisodes = Number(argv[++i]);
    else if (a === "--max-episodes") args.maxEpisodes = Number(argv[++i]);
    else if (a === "-h" || a === "--help") {
      console.log(
        "usage: validate.js <output_dir> [--config <toml>] [--min-episodes N] [--max-episodes N]",
      );
      process.exit(0);
    } else if (!args.outputDir) args.outputDir = a;
    else {
      console.error(`unexpected arg: ${a}`);
      process.exit(2);
    }
  }
  if (!args.outputDir) {
    console.error("output_dir is required");
    process.exit(2);
  }
  return args;
}

// ── JSONL loader ─────────────────────────────────────────────────────────

function loadJsonl(filePath, errors) {
  if (!fs.existsSync(filePath)) {
    errors.push(`missing file: ${filePath}`);
    return [];
  }
  const raw = fs.readFileSync(filePath, "utf8");
  const rows = [];
  raw.split("\n").forEach((line, idx) => {
    if (!line.trim()) return;
    try {
      const obj = JSON.parse(line);
      if (typeof obj !== "object" || obj === null || Array.isArray(obj)) {
        errors.push(
          `${path.basename(filePath)}:${idx + 1}: top-level value must be an object`,
        );
        return;
      }
      rows.push(obj);
    } catch (e) {
      errors.push(
        `${path.basename(filePath)}:${idx + 1}: bad JSON (${e.message})`,
      );
    }
  });
  return rows;
}

// ── Inferences ───────────────────────────────────────────────────────────

function validateInferences(rows, errors) {
  if (rows.length === 0) {
    errors.push("inferences.jsonl is empty");
    return;
  }
  const idsSeen = new Set();
  rows.forEach((r, i) => {
    const tag = `inferences.jsonl:${i + 1}`;
    for (const k of REQUIRED_INF_FIELDS) {
      if (!(k in r)) errors.push(`${tag}: missing required field '${k}'`);
    }
    const rid = r.id;
    const eid = r.episode_id;
    if (typeof rid === "string") {
      if (!UUID_RE.test(rid))
        errors.push(`${tag}: id is not a valid UUID: '${rid}'`);
      if (idsSeen.has(rid)) errors.push(`${tag}: duplicate id '${rid}'`);
      idsSeen.add(rid);
    }
    if (typeof eid === "string" && !UUID_RE.test(eid)) {
      errors.push(`${tag}: episode_id is not a valid UUID: '${eid}'`);
    }
    if (typeof rid === "string" && rid === eid) {
      errors.push(
        `${tag}: id and episode_id are identical (must be distinct UUIDs)`,
      );
    }

    if (r.variant_name !== "initial") {
      errors.push(
        `${tag}: variant_name must be 'initial', got ${JSON.stringify(r.variant_name)}`,
      );
    }

    const inp = r.input;
    if (typeof inp !== "object" || inp === null || !("messages" in inp)) {
      errors.push(`${tag}: input must be an object with a 'messages' array`);
    } else {
      const msgs = inp.messages;
      if (!Array.isArray(msgs) || msgs.length === 0) {
        errors.push(`${tag}: input.messages must be a non-empty array`);
      }
    }

    const out = r.output;
    if (Array.isArray(out)) {
      // chat / tool function: list of content blocks
      out.forEach((blk, j) => {
        if (typeof blk !== "object" || blk === null) {
          errors.push(`${tag}: output[${j}] must be an object`);
          return;
        }
        const t = blk.type;
        if (!ALLOWED_OUTPUT_TYPES.has(t)) {
          const allowed = [...ALLOWED_OUTPUT_TYPES].sort();
          errors.push(
            `${tag}: output[${j}].type '${t}' not in [${allowed.map((x) => `'${x}'`).join(", ")}]`,
          );
        }
      });
    } else if (typeof out === "object" && out !== null) {
      // json function: {raw, parsed}
      if (!("raw" in out) && !("parsed" in out)) {
        errors.push(
          `${tag}: output is an object but has neither 'raw' nor 'parsed' ` +
            `(json-function output expects both)`,
        );
      }
    } else {
      errors.push(
        `${tag}: output must be a list of content blocks (chat/tool) ` +
          `or an object with 'raw'+'parsed' (json), got ${typeof out}`,
      );
    }
  });
}

// ── Feedback ─────────────────────────────────────────────────────────────

function validateFeedback(rows, errors) {
  rows.forEach((r, i) => {
    const tag = `feedback.jsonl:${i + 1}`;
    for (const k of REQUIRED_FB_FIELDS) {
      if (!(k in r)) errors.push(`${tag}: missing required field '${k}'`);
    }
    const kind = r.kind;
    if (!ALLOWED_FEEDBACK_KINDS.has(kind)) {
      const allowed = [...ALLOWED_FEEDBACK_KINDS].sort();
      errors.push(
        `${tag}: kind '${kind}' not in [${allowed.map((x) => `'${x}'`).join(", ")}]`,
      );
    }
    const tid = r.target_id;
    if (typeof tid === "string" && !UUID_RE.test(tid)) {
      errors.push(`${tag}: target_id is not a valid UUID: '${tid}'`);
    }
    const v = r.value;
    if (
      kind === "boolean" &&
      !(typeof v === "boolean" || typeof v === "number")
    ) {
      errors.push(
        `${tag}: boolean feedback value must be bool or 0/1, got ${typeof v}`,
      );
    }
    if (kind === "float" && typeof v !== "number") {
      errors.push(
        `${tag}: float feedback value must be a number, got ${typeof v}`,
      );
    }
  });
}

// ── Cross-validation (referential integrity, metric resolution) ──────────

function validateCross(inferences, feedback, metricDefs, errors, warnings) {
  const inferenceIds = new Set(
    inferences.filter((r) => typeof r.id === "string").map((r) => r.id),
  );
  const episodeIds = new Set(
    inferences
      .filter((r) => typeof r.episode_id === "string")
      .map((r) => r.episode_id),
  );

  const targetsInference = new Map(); // inference_id → Set<metric_name>
  const targetsEpisode = new Map(); // episode_id   → Set<metric_name>

  feedback.forEach((r, i) => {
    const tag = `feedback.jsonl:${i + 1}`;
    const mname = r.metric_name;
    const tid = r.target_id;
    const kind = r.kind;

    if (metricDefs && !(mname in metricDefs)) {
      const defined = Object.keys(metricDefs).sort().join(", ") || "(none)";
      errors.push(
        `${tag}: metric_name '${mname}' not defined in tensorzero.toml ` +
          `(defined metrics: ${defined})`,
      );
      return;
    }
    const mdef = metricDefs ? metricDefs[mname] : null;

    if (mdef) {
      if (kind && mdef.type && kind !== mdef.type) {
        errors.push(
          `${tag}: kind '${kind}' mismatches metric.type '${mdef.type}' ` +
            `for metric '${mname}'`,
        );
      }
      const level = mdef.level;
      if (level === "inference") {
        if (!inferenceIds.has(tid)) {
          const hint = episodeIds.has(tid)
            ? "this might be an episode_id — try matching against inferences.id instead"
            : "the value does not appear as any row's id in inferences.jsonl";
          errors.push(
            `${tag}: target_id '${tid}' does not match any inference id ` +
              `(metric '${mname}' is inference-level; ${hint})`,
          );
        } else {
          if (!targetsInference.has(tid)) targetsInference.set(tid, new Set());
          targetsInference.get(tid).add(mname);
        }
      } else if (level === "episode") {
        if (!episodeIds.has(tid)) {
          const hint = inferenceIds.has(tid)
            ? "this looks like an inference id — try matching against episode_id instead"
            : "the value does not appear as any row's episode_id in inferences.jsonl";
          errors.push(
            `${tag}: target_id '${tid}' does not match any episode_id ` +
              `(metric '${mname}' is episode-level; ${hint})`,
          );
        } else {
          if (!targetsEpisode.has(tid)) targetsEpisode.set(tid, new Set());
          targetsEpisode.get(tid).add(mname);
        }
      }
    } else {
      // No metric defs → just verify target_id exists somewhere
      if (!inferenceIds.has(tid) && !episodeIds.has(tid)) {
        errors.push(
          `${tag}: target_id '${tid}' does not match any inference id or ` +
            `episode_id in inferences.jsonl`,
        );
      }
    }
  });

  if (metricDefs) {
    const infMetrics = Object.entries(metricDefs)
      .filter(([, d]) => d.level === "inference")
      .map(([n]) => n);
    const epMetrics = Object.entries(metricDefs)
      .filter(([, d]) => d.level === "episode")
      .map(([n]) => n);
    if (infMetrics.length) {
      const uncovered = [...inferenceIds].filter(
        (id) => !targetsInference.has(id),
      );
      if (uncovered.length) {
        warnings.push(
          `${uncovered.length}/${inferenceIds.size} inferences have no inference-level feedback`,
        );
      }
    }
    if (epMetrics.length) {
      const uncovered = [...episodeIds].filter((id) => !targetsEpisode.has(id));
      if (uncovered.length) {
        warnings.push(
          `${uncovered.length}/${episodeIds.size} episodes have no episode-level feedback`,
        );
      }
    }
  }
}

// ── Budget ───────────────────────────────────────────────────────────────

function validateBudget(inferences, args, errors) {
  const nEpisodes = new Set(
    inferences
      .filter((r) => typeof r.episode_id === "string")
      .map((r) => r.episode_id),
  ).size;
  if (args.minEpisodes !== null && nEpisodes < args.minEpisodes) {
    errors.push(
      `episode count ${nEpisodes} is below the minimum ${args.minEpisodes}`,
    );
  }
  if (args.maxEpisodes !== null && nEpisodes > args.maxEpisodes) {
    errors.push(
      `episode count ${nEpisodes} exceeds the maximum ${args.maxEpisodes}`,
    );
  }
}

// ── Minimal TOML parser for [metrics.*] blocks ───────────────────────────

function parseMetricDefs(configPath) {
  const metrics = {};
  if (!fs.existsSync(configPath)) return metrics;
  const blockRe = /^\s*\[metrics\.["']?([^"'\]]+?)["']?\]\s*$/;
  const kvRe = /^\s*(type|level|optimize)\s*=\s*["']?([^"'\s#]+)/;
  const sectionRe = /^\s*\[.+\]\s*$/;
  let current = null;
  for (const line of fs.readFileSync(configPath, "utf8").split("\n")) {
    const m = line.match(blockRe);
    if (m) {
      current = m[1];
      metrics[current] = {};
      continue;
    }
    if (sectionRe.test(line) && !blockRe.test(line)) {
      current = null;
      continue;
    }
    if (current) {
      const kv = line.match(kvRe);
      if (kv) metrics[current][kv[1]] = kv[2];
    }
  }
  return metrics;
}

// ── Driver ───────────────────────────────────────────────────────────────

function main() {
  const args = parseArgs(process.argv.slice(2));
  if (
    !fs.existsSync(args.outputDir) ||
    !fs.statSync(args.outputDir).isDirectory()
  ) {
    console.error(`output_dir does not exist: ${args.outputDir}`);
    process.exit(1);
  }

  const errors = [];
  const warnings = [];
  const inferences = loadJsonl(
    path.join(args.outputDir, "inferences.jsonl"),
    errors,
  );
  const feedback = loadJsonl(
    path.join(args.outputDir, "feedback.jsonl"),
    errors,
  );

  // Extras: ignore subdirectories (orchestrator's _meta/ sits there).
  const extras = fs.readdirSync(args.outputDir).filter((name) => {
    if (name === "inferences.jsonl" || name === "feedback.jsonl") return false;
    return fs.statSync(path.join(args.outputDir, name)).isFile();
  });
  if (extras.length)
    warnings.push(
      `unexpected files in ${args.outputDir}: [${extras.join(", ")}]`,
    );

  validateInferences(inferences, errors);
  validateFeedback(feedback, errors);

  const metricDefs = args.config ? parseMetricDefs(args.config) : {};
  validateCross(
    inferences,
    feedback,
    Object.keys(metricDefs).length ? metricDefs : null,
    errors,
    warnings,
  );
  validateBudget(inferences, args, errors);

  const nEpisodes = new Set(inferences.map((r) => r.episode_id)).size;
  const fbByMetric = {};
  for (const r of feedback)
    fbByMetric[r.metric_name] = (fbByMetric[r.metric_name] || 0) + 1;

  console.log(
    `inferences:        ${inferences.length} rows  (${nEpisodes} unique episodes)`,
  );
  console.log(`feedback:          ${feedback.length} rows`);
  if (Object.keys(fbByMetric).length) {
    console.log(`  per metric:      ${JSON.stringify(fbByMetric)}`);
  }
  if (warnings.length) {
    console.log("\nWARNINGS:");
    for (const w of warnings) console.log(`  · ${w}`);
  }
  if (errors.length) {
    console.error(`\nFAIL — ${errors.length} error(s):`);
    for (const e of errors) console.error(`  ✗ ${e}`);
    process.exit(1);
  }
  console.log("\nPASS");
  process.exit(0);
}

main();