tools i use for evals and agent testing

things that showed up in my own repos or in how i talk about the work. logos link out; chips link to project pages on this site.

12 tools, 5 projects

tracing

Langfuse
the trace store openissue reads from. read-only on purpose. a seeded trace proves the api path works, not that anyone had the incident.
used inopenissue
OpenTelemetry + OpenInference
if your spans are otel, openissue reads them without an adapter. the reference lab has to pass on fixtures and on otel, or it doesn't count. openinference is the span shape i used in the lab fixtures.
used inopenissue
LangSmith
simtest can turn a langsmith trace into a seed file, so a bad run becomes a test case instead of a screenshot.
used insimtest

eve evals
scaffolded the local research brain with it and fought the node version first. the smoke evals run on a deterministic shim so ci doesn't need a key. the live ones need a real model and a judge.
used inpocket / probable

Vitest
where the boring checks live. the brain has a test that it refuses a live trade, which matters more than anything in the ui.
used inopenissue pocket / probable solana agent safety
pytest
default for the python side: simtest, gauntlet, and the sim engine.
used insimtest solana agent safety
GitHub Actions
simtest runs on every pr as an eval gate. i added a branch that's supposed to fail, so i know the red actually shows up.
used insimtest

Surfpool
local solana validator i run exploit transactions against. gauntlet has a mock mode, but the real runs go through surfpool.
used insolana agent safety

OpenAI API
in openissue the model only explains what the detector found. structured output, store off, and it can't touch the counts. also showed up in gauntlet and sim-engine model runs.
used inopenissue solana agent safety
Anthropic API
second provider in openissue, same contract as the first. also the anthropic path in the local research brain.
used inopenissue pocket / probable
Vercel AI SDK
wired into the local research brain via the anthropic provider. i'm not claiming live gateway judge evals here.
used inpocket / probable
OpenRouter
how we ran twelve models for beacon without twelve accounts. lossfunk gave us the credits.
used inbeacon

W&B Weave
my first time wrapping a function so every call got logged. small 2024 notebook, but that's where traces started for me.
Giskard
ran a hallucination scan on a rag model in late 2023. tutorial-level, be honest about that.
Guardrails AI
early output validation experiment in 2023. a custom validator and a rail spec, nothing i ship on now.