AI Evaluation Methodology
Evaluation
that learns.
Recursive Self-Improvement (RSI), as measured by SysWisdom, tracks whether a model improves across hint-guided generations — not a single benchmark snapshot, but a convergence arc toward H(p) = 0.
wisdom_log.jsonl · Conductor / Agent / Advisor · Shannon entropy H(p) · Ollama CPU
The Wisdom Formula
Three governing principles. Each one measurable.
The AI Quality Manifesto defines wisdom as the product of three properties. Every dimension of the fluffyRCI methodology maps directly to one of them.
Completeness+Consistency+Validity=Wisdom
Completeness
Ceiling layer revelation
Standard evals test only what a model attempts. fluffyRCI retires mastered topics to expose harder edge cases hidden beneath a fixed question bank. A model can score 90% on a static benchmark by avoiding what it doesn't know.
Consistency
Entropy convergence arc
A trustworthy model converges. Shannon entropy H(p) measured across generations reveals whether capability is compounding or oscillating. An oscillating H(p) after four generations is a reliability flag, not a passing score.
Validity
Cross-family judgment
A judge from the same model family as the agent shares its training biases. Valid measurement requires an independent judge. Cross-family evaluation is a known gap in the current implementation, with a documented fix path.
What fluffyRCI measures
Four capabilities absent from the standard eval stack
HELM, lm-eval, and MT-Bench each measure quality at a moment in time. None of them measure whether quality improves. These four capabilities exist only in fluffyRCI.
Generational arc
Does the model improve across hint-guided generations?
Each generation allows the agent to receive a corrective hint on failed questions. Improvement across generations — tracked as a binary V-stream — is the temporal signal no benchmark captures. A model that learns from the hint is different from one that doesn't, even when their Gen 1 scores are identical.
Unique to fluffyRCICeiling layer revelation
Are mastered topics retired to expose harder ones?
Topics answered correctly across consecutive generations are retired from the active question pool. The denominator is fixed to count this honestly. Topics the model never faces cannot be said to be known — ceiling revelation surfaces unknown unknowns that a fixed question bank permanently hides.
Unique to fluffyRCICache-as-speed-signal
Is latency reduction evidence of knowledge consolidation?
A model that has genuinely learned answers faster on repeated questions. Response latency is tracked across generations on retested topics. Latency reduction on a correct retested answer is a knowledge consolidation signal — distinct from model-level improvement, which would conflate caching with learning.
Unique to fluffyRCIEntropy convergence arc
Is H(p) moving toward zero, or flattening at a ceiling?
H(p) = −∑p·log₂(p) computed over the per-question correctness distribution. H(p) = 0 means the model is deterministically correct on every tested topic. H(p) flat after Gen 3 is a capability ceiling signal. A single benchmark score hides both cases behind the same number.
Unique to fluffyRCITool comparison
What each approach measures — and what it misses
| Capability | HELM | lm-eval | MT-Bench | fluffyRCI |
|---|---|---|---|---|
| Static benchmark scores | ✓ | ✓ | ✓ | ✓ |
| Multi-turn conversation quality | — | — | ✓ | ✓ |
| Generational improvement tracking | — | — | — | ✓ Unique |
| Ceiling layer revelation | — | — | — | ✓ Unique |
| Cache-as-speed-signal | — | — | — | ✓ Unique |
| Entropy convergence arc H(p) | — | — | — | ✓ Unique |
| Fully offline / air-gapped | — | ✓ | — | ✓ |
| CPU-only, no GPU required | — | — | — | ✓ |
| Hint-driven learning loop | — | — | — | ✓ |
| Human governance embedded in loop | — | — | — | ✓ |
HELM: Stanford CRFM · lm-eval: EleutherAI LM Evaluation Harness · MT-Bench: LMSYS/UC Berkeley · fluffyRCI: SysWisdom
Methodological transparency
Known limitations and documented fix paths
Every evaluation system has known weaknesses. Publishing them — alongside the specific fix for each — is how a methodology earns credibility. These five limitations were identified during initial runs.
Single run — no error bar+
Each topic is evaluated once per generation. A single run cannot distinguish a model that answers consistently from one that got lucky. Without error bars, H(p) values cannot be meaningfully compared across models.
Fix → Run ≥ 5 repeats per topic per generation. Report confidence intervals alongside H(p). Flag any run count < 5 as "preliminary."
Non-fixed question bank across model sizes+
The initial 1B model evaluation used 24 topics; the 3B run used 28. Cross-model comparison on different topic banks is invalid — a higher H(p) on 28 topics may reflect harder questions, not lower capability.
Fix → Fix the question bank at N topics before any comparative evaluation. Treat bank additions as a new evaluation version, not a continuation of a prior run.
Same-family judge model+
The advisor/judge model is drawn from the same family as the agent. Models in the same family share training distribution biases, which can cause the judge to confirm the agent's errors rather than detecting them.
Fix → Use a cross-family judge: if the agent is Llama, judge with an OpenAI or Anthropic model. Document the judge family in every published report.
H = 0 ambiguity+
H(p) = 0 has two distinct causes: the model answered every question correctly, or the question bank was exhausted. The current implementation cannot distinguish these cases from H(p) alone.
Fix → Report (V_avg, H) as a pair. V_avg > 0.95 with H = 0 means mastery. V_avg ≤ 0.80 with H = 0 means bank exhaustion — flag as "ceiling not reached, bank exhausted."
Denominator inflation from retired topics+
When topics are retired after mastery, they are excluded from the H(p) denominator. This inflates apparent entropy reduction — the model appears to converge faster than it is, because hard questions are being removed from the pool.
Fix → Retired topics remain in the denominator, counted as correct. This separates "questions we no longer need to ask" from "questions we are confident the model knows."
RSI Architecture
Conductor · Agent · Advisor
The three-layer system runs fully offline using Ollama on CPU. Every question, answer, judgment, and entropy snapshot is appended to wisdom_log.jsonl — the institutional memory that makes each run reproducible.
Conductor
Session orchestrator
Selects the next question from the active bank. Tracks the V-stream and generation state. Decides when a topic has been mastered and retires it. Supplies corrective hints after a failed generation.
Agent
Knowledge responder
Answers each question using only its training and the accumulated wisdom_log.jsonl. On a failed question, receives a hint from the conductor before the next generation attempt.
Advisor
Binary judge
Evaluates each answer and returns V ∈ {0, 1} with an explanation. Currently same-family as the agent (documented limitation above). The binary V-stream feeds directly into H(p) calculation.
wisdom_log.jsonl
Every run appends to a single JSONL file: question, generation number, answer, V value, judge explanation, latency, and entropy snapshot. A model returning to the same question bank in a future session can resume from a documented prior state — institutional memory that outlasts any single session.
SysWisdom · AI Quality Research
The methodology is documented.
The conversation is open.
This framework is how SysWisdom approaches AI quality evaluation — measurable, human-governed, entropy-convergent. If you’re designing an evaluation process for AI systems in governed contexts, the methodology is available and the approach is discussable.
aj@syswisdom.ai
No form. No funnel. Email directly.