Authoring evaluators

Evaluators are YAML files under evaluators/agent/ or evaluators/mcp/ at the repo root. Adding one needs no code. After adding or editing, run npm run build:catalog to rebuild the skill catalogs.

Minimal evaluator

schema_version: 1
id: system-prompt-leakage
name: System Prompt Leakage
severity: critical # critical | high | medium | low
standards:
  owasp-llm: LLM07
  atlas: AML.T0056
description: One-line summary for judges and contributors
pass_criteria: |-
  - Bullet list of safe behaviors
fail_criteria: |-
  - Bullet list of vulnerable behaviors
patterns:
  - name: Pattern label
    template: Attack prompt text. Use {{placeholder}} for LLM-filled variables.

Fields

Field	Required	Notes
`id`	Yes	Kebab-case, unique across both catalogs
`name`	Yes	Display name
`severity`	Yes	`critical`, `high`, `medium`, `low`
`standards`	Recommended	Map of taxonomy key → ID; drives auto-derived suites (omit if unmapped)
`pass_criteria`	Yes	Injected verbatim into the judge prompt
`fail_criteria`	Yes	Injected verbatim into the judge prompt
`patterns`	Flat layout only	Non-empty array of `{ name, template }`; omit when using the directory layout
`description`	Recommended	Short summary for docs and skills

Optional fields

Field	Purpose
`schema_version`	`1` when set
`judge_needs_llm`	`true` for semantic judgment; `false` for regex/static checks
`applies_to_all_tools`	MCP only — generate attacks for every tool in `tools/list`
`judge_hint`	Extra guidance appended to the judge prompt
`surfaces`	`agent`, `browser`, `mcp` — informational
`turn_mode`	`single` or `multi` — informational; the run config decides at runtime

Standards → auto-derived suites

Setting a standards key automatically includes the evaluator in the matching suite.

Key	Example values	Auto-derived suite
`owasp-llm`	`LLM01` … `LLM10`	`owasp-llm-top10`
`owasp-mcp`	`MCP01` … `MCP10`	`owasp-mcp-top10`
`owasp-agentic`	`ASI01` …	`owasp-agentic-ai`
`owasp-api`	`API1`, `API4`, …	`owasp-api-top10`
`atlas`	`AML.T0056`, …	`mitre-atlas`
`eu-ai-act`	`Art.5`, `Art.10`, …	`eu-ai-act`
`nist`	`Safe`, `Secure & Resilient`, …	`nist-ai-rmf`

Don’t use ref: or mitre: keys — the pre-commit hook rejects them on staged evaluator files.

Directory layout for many patterns

When patterns are long or numerous, split them into their own files:

evaluators/agent/<category>/<id>/
  evaluator.yaml      ← all fields except patterns
  patterns/
    <slug>.yaml       ← { name, template }
  <id>.test.yaml      ← { kind: response, pass_case, fail_case }

Validate

npm run validate:skills        # check frontmatter against the Zod schema
npm run build:catalog:check    # verify the catalog is up to date

Both run on pre-commit via Husky. validate:skills checks frontmatter against its own Zod schema, which is a separate, hand-maintained copy of the runtime loader’s schema. When you add a new standards key, update both so the check and the engine stay in sync.

Get Started

CLI

Browser Extension

MCP Server

Skills

SDK

Evaluators & Suites

Trace-aware Testing

Reference

Authoring evaluators

Minimal evaluator

Fields

Optional fields

Standards → auto-derived suites

Directory layout for many patterns

Validate

​Minimal evaluator

​Fields

​Optional fields

​Standards → auto-derived suites

​Directory layout for many patterns

​Validate

Minimal evaluator

Fields

Optional fields

Standards → auto-derived suites

Directory layout for many patterns

Validate