> ## Documentation Index
> Fetch the complete documentation index at: https://docs.agentopfor.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Authoring evaluators

> Add a new evaluator — no TypeScript needed.

Evaluators are YAML files under `evaluators/agent/` or `evaluators/mcp/` at the repo root. Adding one needs no code. After adding or editing, run `npm run build:catalog` to rebuild the skill catalogs.

## Minimal evaluator

```yaml theme={null}
schema_version: 1
id: system-prompt-leakage
name: System Prompt Leakage
severity: critical # critical | high | medium | low
standards:
  owasp-llm: LLM07
  atlas: AML.T0056
description: One-line summary for judges and contributors
pass_criteria: |-
  - Bullet list of safe behaviors
fail_criteria: |-
  - Bullet list of vulnerable behaviors
patterns:
  - name: Pattern label
    template: Attack prompt text. Use {{placeholder}} for LLM-filled variables.
```

## Fields

| Field           | Required         | Notes                                                                         |
| --------------- | ---------------- | ----------------------------------------------------------------------------- |
| `id`            | Yes              | Kebab-case, unique across both catalogs                                       |
| `name`          | Yes              | Display name                                                                  |
| `severity`      | Yes              | `critical`, `high`, `medium`, `low`                                           |
| `standards`     | Recommended      | Map of taxonomy key → ID; drives auto-derived suites (omit if unmapped)       |
| `pass_criteria` | Yes              | Injected verbatim into the judge prompt                                       |
| `fail_criteria` | Yes              | Injected verbatim into the judge prompt                                       |
| `patterns`      | Flat layout only | Non-empty array of `{ name, template }`; omit when using the directory layout |
| `description`   | Recommended      | Short summary for docs and skills                                             |

### Optional fields

| Field                  | Purpose                                                                |
| ---------------------- | ---------------------------------------------------------------------- |
| `schema_version`       | `1` when set                                                           |
| `judge_needs_llm`      | `true` for semantic judgment; `false` for regex/static checks          |
| `applies_to_all_tools` | MCP only — generate attacks for every tool in `tools/list`             |
| `judge_hint`           | Extra guidance appended to the judge prompt                            |
| `surfaces`             | `agent`, `browser`, `mcp` — informational                              |
| `turn_mode`            | `single` or `multi` — informational; the run config decides at runtime |

## Standards → auto-derived suites

Setting a `standards` key automatically includes the evaluator in the matching suite.

| Key             | Example values                  | Auto-derived suite |
| --------------- | ------------------------------- | ------------------ |
| `owasp-llm`     | `LLM01` … `LLM10`               | `owasp-llm-top10`  |
| `owasp-mcp`     | `MCP01` … `MCP10`               | `owasp-mcp-top10`  |
| `owasp-agentic` | `ASI01` …                       | `owasp-agentic-ai` |
| `owasp-api`     | `API1`, `API4`, …               | `owasp-api-top10`  |
| `atlas`         | `AML.T0056`, …                  | `mitre-atlas`      |
| `eu-ai-act`     | `Art.5`, `Art.10`, …            | `eu-ai-act`        |
| `nist`          | `Safe`, `Secure & Resilient`, … | `nist-ai-rmf`      |

<Warning>Don't use `ref:` or `mitre:` keys — the pre-commit hook rejects them on staged evaluator files.</Warning>

## Directory layout for many patterns

When patterns are long or numerous, split them into their own files:

```
evaluators/agent/<category>/<id>/
  evaluator.yaml      ← all fields except patterns
  patterns/
    <slug>.yaml       ← { name, template }
  <id>.test.yaml      ← { kind: response, pass_case, fail_case }
```

## Validate

```bash theme={null}
npm run validate:skills        # check frontmatter against the Zod schema
npm run build:catalog:check    # verify the catalog is up to date
```

Both run on pre-commit via Husky. `validate:skills` checks frontmatter against its own Zod schema, which is a separate, hand-maintained copy of the runtime loader's schema. When you add a new `standards` key, update both so the check and the engine stay in sync.