Measurement Profile-relative · External adapters

Evals & benchmarks

Without measurement, a taste registry is a folder of opinions. With only universal beauty scores, it is a category error. We measure agreement with a declared standard first—and connect to research benchmarks where they inform judge quality and safety.

Principle

A score is always relative to a package. The question “is this beautiful?” is replaced by “does this fit Editorial technē?” or “would this curator reject it?” External benchmarks then answer a different family of questions: is our judge competent at creative-writing preference, aesthetic perception, or bias-sensitive assessment?

Profile-relative first. Universal metrics second. Reasons always.

Built-in evals

EvalWhat it measuresData
Pairwise agreement Judge ranking vs curator on held-out pairs Golden pairs per package
Threshold agreement Pass / fail / borderline labels Labeled candidates
Calibration Whether high scores are rare and meaningful Score histograms on open sets
Anti-exemplar recall Known failures still fail Living slop / anti sets
Transfer fidelity Work under transfer guidance still matches Human recognition studies
Retrieve quality Exemplar relevance for a concept query Query → exemplar judgments
Package hygiene Schema, license, citation, anti-exemplars present Static checks in CI

Scaffold path: evals/ and packages/*/evals/. Harness code lands as the golden sets grow.

External benchmarks to integrate

These are not “the product metric.” They are adapters: optional report cards for judge backends and multimodal packages, grounded in active research.

Prose

LitBench

Large-scale creative-writing preference pairs; strong OTS judges still only ~mid-70% human agreement—showing why specialized verifiers matter. Use as a baseline when packaging literary tastes.

Prose / style

EQ-Bench Creative Writing & Slop Score

LLM-judged creative writing with axes for style, slop, repetition, length. Useful optional dimensions for anti-slop packages and report cards.

Vision

AesBench

Expert-oriented benchmark for multimodal models on image aesthetics perception. Relevant when packages include screenshots and moodboard images.

Vision / safety

AesBiasBench

Separates stereotype bias in aesthetic evaluation from alignment with genuine human preference—critical for any public multimodal judge.

Voice

Stylometric & style fidelity

Embedding cosine / stylometric distance between author corpora and generations. Natural fit for “in the style of this curator” packages (with license care).

Preference ML

Bradley–Terry / reward models

Pairwise preference training is the standard path from golden sets to stronger profile-relative scorers—same schema, better backend.

Research directions that inform packages

Personalized image aesthetics (PIAA)

Individual taste ≠ population mean—validates per-package scoring over global beauty nets.

Mode collapse & “safe” style

LLMs skew to low-originality high-fluency prose; anti-exemplar packs should target that basin explicitly.

Persona / style sensitivity

Benchmarks show model quality shifts under writing-style variation—agents need named standards, not silent defaults.

Human–MLLM gaps

Multimodal judges disagree with humans on which attributes drive quality; reasons and golden sets remain mandatory.

Leaderboards (planned)

Two boards, never confused:

  • A

    Judge leaderboard

    Heuristic vs embedding vs LLM vs trained RM—on shared golden packs and optional external adapters.

  • B

    Package hygiene & difficulty

    Completeness, inter-annotator difficulty, stability over time—not “which taste is best.”

How to contribute evals

Add golden pairs under a package; document labeling guidelines; open a PR with inter-annotator notes. External adapter PRs should emit a common report shape (schema evolving in evals/README.md). Disagreement with a judge is data—file it as a golden, not only as a vibe.

Registry design Laboratory GitHub
Indicative references.
  1. LitBench (creative writing preference; EACL 2026).
  2. EQ-Bench Creative Writing v3; Slop Score (eqbench.com).
  3. AesBench (arXiv:2401.08276); AesBiasBench (EMNLP 2025).
  4. MCP specification and registry (modelcontextprotocol.io).
  5. Agent Skills as open standard (agentskills.io / Anthropic skills announcement).