# Sohom Pal

> i make agents prove what they did. traces in, failing tasks and a regression eval out.

sohom pal. AI engineer. Based in Bangalore, India.

ai engineer. building something new right now, more on that here soon. until the end of july 2026 i was at pocket, working on probable, a prediction-market agent, after spending april at network school with the pocket team. before that: lossfunk research on beacon, and safety evals for agents that touch solana.

now: heads down on something new.

- [the map of my internet](https://sxohom.xyz/internet/)
- [tools i use for evals](https://sxohom.xyz/tools/)
- [things you could buy](https://sxohom.xyz/buy/)
- [colophon](https://sxohom.xyz/colophon/)

## Contact

if you have an agent you can't prove works, [send me a trace](mailto:hi@sxohom.xyz).

- email: hi@sxohom.xyz
- X: https://x.com/sxohom
- GitHub: https://github.com/sohomx
- site: https://sxohom.xyz

## Receipts

- [paper(beacon): arXiv 2510.16727 · 420 pairs · 12 models](https://arxiv.org/abs/2510.16727)
- [ship(openissue): npm @sxohom/openissue](https://www.npmjs.com/package/@sxohom/openissue)
- [bench(gauntlet): 96 scenarios · safety weighted .40](https://github.com/light-research/gauntlet)
- [tool(simtest): fails the PR when the agent gets worse](https://github.com/sohomx/simtest)

## Projects

- [pocket / probable](https://sxohom.xyz/projects/pocket-probable/) (apr 2026 to july 2026): the proof layer. [markdown](https://sxohom.xyz/projects/pocket-probable.md)
- [beacon](https://sxohom.xyz/projects/beacon/) (2025): sycophancy benchmark. [markdown](https://sxohom.xyz/projects/beacon.md)
- [solana agent safety](https://sxohom.xyz/projects/solana-agent-safety/) (2025 to 2026): gauntlet · sim engine · idl agent. [markdown](https://sxohom.xyz/projects/solana-agent-safety.md)
- [openissue](https://sxohom.xyz/projects/openissue/) (2026): deterministic trace to incident. [markdown](https://sxohom.xyz/projects/openissue.md)
- [simtest](https://sxohom.xyz/projects/simtest/) (2026): fuzz agents in ci. [markdown](https://sxohom.xyz/projects/simtest.md)

## Pages

- [home](https://sxohom.xyz/)
- [home markdown](https://sxohom.xyz/index.md)
- [colophon](https://sxohom.xyz/colophon/)
- [tools](https://sxohom.xyz/tools/) · [markdown](https://sxohom.xyz/tools.md)
- [internet](https://sxohom.xyz/internet/) · [markdown](https://sxohom.xyz/internet.md)
- [things you could buy](https://sxohom.xyz/buy/)
- [llms.txt](https://sxohom.xyz/llms.txt)
- [llms-full.txt](https://sxohom.xyz/llms-full.txt)


---

# colophon

site for Sohom Pal (sohom pal).

## type

- system ui sans for body, headings, nav
- ibm plex mono for receipts and code

## colour

- paper: #f7f7f8
- ink: #1a1a1a
- muted: #666 / #999
- hairline: #e5e5e5

## stack

next.js static export, react, tailwind css v4, github pages. no analytics, no cookies, no tracking.

## built with

cursor agents, from a brief sohom wrote. quiet personal-site layout.

[home](https://sxohom.xyz/) · [llms.txt](https://sxohom.xyz/llms.txt)


---

# tools i use for evals and agent testing

> things that showed up in my own repos or in how i talk about the work. logos link out; chips link to project pages on this site.

12 tools, 5 projects.

Author: [Sohom Pal](https://sxohom.xyz/)

## tracing

- [Langfuse](https://langfuse.com): the trace store openissue reads from. read-only on purpose. a seeded trace proves the api path works, not that anyone had the incident. used in: [openissue](https://sxohom.xyz/projects/openissue/).
- [OpenTelemetry + OpenInference](https://opentelemetry.io): if your spans are otel, openissue reads them without an adapter. the reference lab has to pass on fixtures and on otel, or it doesn't count. openinference is the span shape i used in the lab fixtures. used in: [openissue](https://sxohom.xyz/projects/openissue/).
- [LangSmith](https://www.langchain.com/langsmith): simtest can turn a langsmith trace into a seed file, so a bad run becomes a test case instead of a screenshot. used in: [simtest](https://sxohom.xyz/projects/simtest/).

## evals

- [eve evals](https://github.com/vercel/eve): scaffolded the local research brain with it and fought the node version first. the smoke evals run on a deterministic shim so ci doesn't need a key. the live ones need a real model and a judge. used in: [pocket / probable](https://sxohom.xyz/projects/pocket-probable/).

## testing

- [Vitest](https://vitest.dev): where the boring checks live. the brain has a test that it refuses a live trade, which matters more than anything in the ui. used in: [openissue](https://sxohom.xyz/projects/openissue/), [pocket / probable](https://sxohom.xyz/projects/pocket-probable/), [solana agent safety](https://sxohom.xyz/projects/solana-agent-safety/).
- [pytest](https://docs.pytest.org): default for the python side: simtest, gauntlet, and the sim engine. used in: [simtest](https://sxohom.xyz/projects/simtest/), [solana agent safety](https://sxohom.xyz/projects/solana-agent-safety/).
- [GitHub Actions](https://github.com/features/actions): simtest runs on every pr as an eval gate. i added a branch that's supposed to fail, so i know the red actually shows up. used in: [simtest](https://sxohom.xyz/projects/simtest/).

## sims

- [Surfpool](https://www.surfpool.run): local solana validator i run exploit transactions against. gauntlet has a mock mode, but the real runs go through surfpool. used in: [solana agent safety](https://sxohom.xyz/projects/solana-agent-safety/).

## models / infra

- [OpenAI API](https://platform.openai.com/docs): in openissue the model only explains what the detector found. structured output, store off, and it can't touch the counts. also showed up in gauntlet and sim-engine model runs. used in: [openissue](https://sxohom.xyz/projects/openissue/), [solana agent safety](https://sxohom.xyz/projects/solana-agent-safety/).
- [Anthropic API](https://docs.anthropic.com): second provider in openissue, same contract as the first. also the anthropic path in the local research brain. used in: [openissue](https://sxohom.xyz/projects/openissue/), [pocket / probable](https://sxohom.xyz/projects/pocket-probable/).
- [Vercel AI SDK](https://ai-sdk.dev): wired into the local research brain via the anthropic provider. i'm not claiming live gateway judge evals here. used in: [pocket / probable](https://sxohom.xyz/projects/pocket-probable/).
- [OpenRouter](https://openrouter.ai): how we ran twelve models for beacon without twelve accounts. lossfunk gave us the credits. used in: [beacon](https://sxohom.xyz/projects/beacon/).

## older stuff

- [W&B Weave](https://weave-docs.wandb.ai): my first time wrapping a function so every call got logged. small 2024 notebook, but that's where traces started for me.
- [Giskard](https://www.giskard.ai): ran a hallucination scan on a rag model in late 2023. tutorial-level, be honest about that.
- [Guardrails AI](https://www.guardrailsai.com): early output validation experiment in 2023. a custom validator and a rail spec, nothing i ship on now.

## what i code with

[Cursor](https://cursor.com), [Claude Code](https://claude.com/claude-code), [Codex](https://openai.com/codex), [Conductor](https://conductor.build), [CodeRabbit](https://www.coderabbit.ai)

[html page](https://sxohom.xyz/tools/) · [home](https://sxohom.xyz/) · [llms.txt](https://sxohom.xyz/llms.txt)


---

# map of my internet

> people whose sites i keep going back to.

60 stations, 11 lines.

Author: [Sohom Pal](https://sxohom.xyz/)

## eval nerds

- [jason liu](https://jxnl.co) ([jxnl.co](https://jxnl.co), [@jxnlco](https://x.com/jxnlco)): dx engineer on the codex team at openai, made instructor. his site mixes rag, agents and evals with posts on ambition and self-worth, on purpose.
- [Nirant Kasliwal](https://nirantk.com) ([nirantk.com](https://nirantk.com), [@NirantK](https://x.com/NirantK)): wrote an nlp book for software engineers, built fastembed, contributed to ragas. also made the starter guide to bengaluru.
- [Minh Nhat Nguyen](https://linktr.ee/menhguin) ([linktr.ee](https://linktr.ee/menhguin), [@menhguin](https://x.com/menhguin)): trains programmable-weights llms, previously at hud evals, and built ai hub central to a million users before it sold. hard sci-fi fan.
- [Andrew Ng](https://www.andrewng.org) ([andrewng.org](https://www.andrewng.org), [@AndrewYNg](https://x.com/AndrewYNg)): coursera cofounder, teaches at stanford. recently put out the ai engineering skills map, a guide to what to learn when you build with models.
- [prathosh A P](https://prathosh.in) ([prathosh.in](https://prathosh.in), [@prathoshap](https://x.com/prathoshap)): professor at iisc working on trustworthy ml, machine unlearning and ai for medicine. cofounded latentforce, which launched a shared workspace for teams of agents today.

## people shipping agents

- [Teknium](https://github.com/teknium1) ([github.com](https://github.com/teknium1), [@Teknium](https://x.com/Teknium)): cofounder and lead engineer of hermes agent at nous research, previously at stability. hermes started as agent building blocks for data generation and rl.
- [pili](https://0xpili.xyz) ([0xpili.xyz](https://0xpili.xyz), [@0xpili_](https://x.com/0xpili_)): writes "thoughtful technology" essays on crypto, ai and being human, like the agentic middle layer and requests for agentfi primitives. got her agents writing in simplified technical english to cut the slop.
- [jason liu](https://jxnl.co) ([jxnl.co](https://jxnl.co), [@jxnlco](https://x.com/jxnlco)): dx engineer on the codex team at openai, made instructor. his site mixes rag, agents and evals with posts on ambition and self-worth, on purpose.
- [Phil](https://practicecents.com) ([practicecents.com](https://practicecents.com), [@Phil_Holland](https://x.com/Phil_Holland)): post-training lead at spacexai, air force veteran, trained at eastman school of music. made a mac skill that talks to reminders, calendar, notes, contacts and messages. his profile links a free tuner and metronome for band kids.
- [sphinx](https://sphinxstack.com) ([sphinxstack.com](https://sphinxstack.com), [@protosphinx](https://x.com/protosphinx)): founder of deskera, now building erp.ai and a pile of open things: partmode (cad), bomwiki and sphinxstack, a free library of agent skills. writes short posts about agency.
- [Simon](https://x.com/tokumin) ([x.com](https://x.com/tokumin), [@tokumin](https://x.com/tokumin)): builds products at google labs, currently thinking about family assistants.
- [@ghuubear](https://snakeoilsalesman.in) ([snakeoilsalesman.in](https://snakeoilsalesman.in), [@ghuubear](https://x.com/ghuubear)): lead yapper at metaforms. his blog is about truth and deception, with posts like "the real bottleneck for autonomous agents is usable context" and "why llm writing feels dead".

## coding agent gang

- [Theo](https://t3.gg) ([t3.gg](https://t3.gg), [@theo](https://x.com/theo)): runs t3 code and t3.chat, made create-t3-app, and makes youtube videos about all of it.
- [dax](https://thdxr.com) ([thdxr.com](https://thdxr.com), [@thdxr](https://x.com/thdxr)): building opencode at anomaly. "i build things then try to remember to write about them." just shipped opentunnel, end-to-end encrypted public urls for local apps.
- [Charlie Holtz](https://charlieholtz.com) ([charlieholtz.com](https://charlieholtz.com), [@charlieholtz](https://x.com/charlieholtz)): makes conductor, a mac app for running a team of coding agents in parallel. before that, hacker in residence at replicate, and quant finance before that.
- [Thorsten Ball](https://thorstenball.com) ([thorstenball.com](https://thorstenball.com), [@thorstenball](https://x.com/thorstenball)): works on amp, writes the register spill newsletter, wrote the interpreter and compiler books.
- [sunil pai](https://sunilpai.dev) ([sunilpai.dev](https://sunilpai.dev), [@threepointone](https://x.com/threepointone)): worked on react, cloudflare workers and partykit, now on durable infra. wrote "the senior engineer death spiral" and is giving a talk called "your ai agent is a distributed system".
- [Emanuele Di Pietro](https://emanueledipietro.com) ([emanueledipietro.com](https://emanueledipietro.com), [@emanueledpt](https://x.com/emanueledpt)): indie hacker in italy shipping synara, an open source desktop workspace for coding agents (90 releases in 6 months), and remodex, codex on your iphone.

## indie shippers

- [Pranav Hari](https://pranavhari.com) ([pranavhari.com](https://pranavhari.com), [@pHequals7](https://x.com/pHequals7)): makes muesli, local-first dictation and meeting transcription. his pinned post is the dune litany rewritten for fomo.
- [Tanmay Sonawane](https://tanmay.me) ([tanmay.me](https://tanmay.me), [@tanmays](https://x.com/tanmays)): ios developer and ux designer in mumbai. made finma, soor, vero, wdgts and write, and just shipped halo, a personal assistant that sorts out what matters right now.
- [kyzo](https://kyzo.io) ([kyzo.io](https://kyzo.io), [@ky__zo](https://x.com/ky__zo)): twice-exited founder, "you can just hack things". building one, a mac inbox where you hear and answer questions from your agents by voice.
- [Virgile Rietsch](https://go.blitzreels.com/x) ([go.blitzreels.com](https://go.blitzreels.com/x), [@virgilerietsch](https://x.com/virgilerietsch)): building blitzreels, a video clipping agent. also open sourced blitzclean, a ram monitor and disk cleaner for mac.
- [mageframe](https://github.com/mageframe) ([github.com](https://github.com/mageframe), [@mageframeEXE](https://x.com/mageframeEXE)): built zero, which watches your screen and writes a timeline of your day. also wired pokémon red's memory into a video model to make an endless anime.
- [jitesh](https://jiteshcodes.com) ([jiteshcodes.com](https://jiteshcodes.com), [@Jitesh_117](https://x.com/Jitesh_117)): lives in vim. built and open sourced vim royale, multiplayer vim duels with elo. engineer at consuma, keeps a learning blog.
- [sphinx](https://sphinxstack.com) ([sphinxstack.com](https://sphinxstack.com), [@protosphinx](https://x.com/protosphinx)): founder of deskera, now building erp.ai and a pile of open things: partmode (cad), bomwiki and sphinxstack, a free library of agent skills. writes short posts about agency.
- [@levelsio](https://levels.io) ([levels.io](https://levels.io), [@levelsio](https://x.com/levelsio)): builds startups alone and bootstrapped: nomads.com, remote ok, photo ai, interior ai. wrote a book called make.
- [guru](https://hackyguru.com) ([hackyguru.com](https://hackyguru.com), [@hackyguru](https://x.com/hackyguru)): devrel lead at logos, formerly walletconnect and protocol labs. kept his whoop strap, dropped the subscription, and runs it fully local on his own fork.
- [Jamon Holmgren](https://jamon.dev) ([jamon.dev](https://jamon.dev), [@jamonholmgren](https://x.com/jamonholmgren)): coding since 1992, cofounded infinite red, makes games on the side. wrote about how his agent workflow got faster and fun again.
- [DHH](https://dhh.dk) ([dhh.dk](https://dhh.dk), [@dhh](https://x.com/dhh)): made ruby on rails and omarchy, runs 37signals, races at le mans. recently wrote "pencils down" about the end of writing code by hand.

## gpu poor and gpu rich

- [tokenbender](https://tokenbender.com) ([tokenbender.com](https://tokenbender.com), [@tokenbender](https://x.com/tokenbender)): abhishek mishra, founder of deus experiments, works on post-training and autoresearch harnesses. went from intel server firmware to rl and sparsity.
- [elie bakouch](https://huggingface.co/spaces/HuggingFaceTB/smol-training-playbook) ([huggingface.co](https://huggingface.co/spaces/HuggingFaceTB/smol-training-playbook), [@eliebakouch](https://x.com/eliebakouch)): research at prime intellect, previously hugging face. co-wrote the smol training playbook, 200+ pages on training llms end to end. recently ran a big open experiment with agents iterating on a research environment and shared the traces.
- [Archie Sengupta](https://archiesengupta.com) ([archiesengupta.com](https://archiesengupta.com), [@archiexzzz](https://x.com/archiexzzz)): does pre- and post-training at sarvam and robotics and simulation research at midcentury. wants to work on hard problems in biology, and his family were doctors.
- [maharshi](https://maharshi.bearblog.dev) ([maharshi.bearblog.dev](https://maharshi.bearblog.dev), [@maharshii](https://x.com/maharshii)): ml performance at fal. wrote "vibecoding gpu kernels", arguing kernels are easy to check so models can write them, and says inference work has moved up the abstraction ladder.
- [vixhal](https://github.com/vixhal-baraiya) ([github.com](https://github.com/vixhal-baraiya), [@TheVixhal](https://x.com/TheVixhal)): physics and ml double major building a graph database at hydra db. wrote up how they turned qwen3-4b into a decision model on a macbook with mlx.
- [Sanskar Pandey](https://scholar.google.com/citations?user=04PtMiIAAAAJ) ([scholar.google.com](https://scholar.google.com/citations?user=04PtMiIAAAAJ), [@sanskxr02](https://x.com/sanskxr02)): building fidelity dynamics, post-training and evals for robot learning, previously at sarvam. ✔ beacon co-author.

## research lists

- [Andrew Ng](https://www.andrewng.org) ([andrewng.org](https://www.andrewng.org), [@AndrewYNg](https://x.com/AndrewYNg)): coursera cofounder, teaches at stanford. recently put out the ai engineering skills map, a guide to what to learn when you build with models.
- [Ankit Jxa](https://hireankit.bearblog.dev) ([hireankit.bearblog.dev](https://hireankit.bearblog.dev), [@kingofknowwhere](https://x.com/kingofknowwhere)): eight years of math and llms, now contact center engineering and audio llms, building agiathome. keeps a list of research problems that are fun rather than sensible.
- [Chris Barber](https://chrisbarber.co) ([chrisbarber.co](https://chrisbarber.co), [@chrisbarber](https://x.com/chrisbarber)): makes lists and surveys about data, compute and ai: the breakout list of startups, the frontier list, and an rl environment faq written with epoch ai.
- [Sanyam Jain](https://www.sanyam-ai.in) ([sanyam-ai.in](https://www.sanyam-ai.in), [@Sanyam0605](https://x.com/Sanyam0605)): ml engineer and researcher at iit delhi who likes reading papers and writing code in the same week. wrote up 40 days and 20 companies of ai hiring in india.
- [Suhail](https://x.com/Suhail) ([x.com](https://x.com/Suhail), [@Suhail](https://x.com/Suhail)): founded mixpanel, now starting something new that "started w 2 8xB200s".
- [Nick](https://x.com/nickcammarata) ([x.com](https://x.com/nickcammarata), [@nickcammarata](https://x.com/nickcammarata)): mechanistic interpretability and meditation, per his bio.

## systems people

- [Arpit Bhayani](https://arpitbhayani.me) ([arpitbhayani.me](https://arpitbhayani.me), [@arpit_bhayani](https://x.com/arpit_bhayani)): principal engineer at razorpay, made dicedb, building a lightweight ide called px0. curates tdd.cat, an engineering newspaper on databases, system design and applied ai.
- [Daniel Lockyer](https://daniellockyer.com) ([daniellockyer.com](https://daniellockyer.com), [@DanielLockyer](https://x.com/DanielLockyer)): performance consultant who fixes high cpu, memory leaks, slow apis and cloud bills. once found and patched a scroll lag bug on x's web app. runs a 2:43 marathon.
- [sunil pai](https://sunilpai.dev) ([sunilpai.dev](https://sunilpai.dev), [@threepointone](https://x.com/threepointone)): worked on react, cloudflare workers and partykit, now on durable infra. wrote "the senior engineer death spiral" and is giving a talk called "your ai agent is a distributed system".
- [Devanshu Sharma](https://devanshusharma.com) ([devanshusharma.com](https://devanshusharma.com), [@DevanshuXi](https://x.com/DevanshuXi)): distributed systems and databases person who cares about correctness. now on infra at mastra, got into cal.com with a cold email.
- [Can Duruk](https://justoffbyone.com) ([justoffbyone.com](https://justoffbyone.com), [@can](https://x.com/can)): leads product engineering at modal, former cto of felt, early at uber. writes off by one about engineering and managing engineers.
- [shrinath](https://github.com/ShrinathNR) ([github.com](https://github.com/ShrinathNR), [@shrinathx](https://x.com/shrinathx)): working on light research, teaches and builds at solana turbine. ✔ built gauntlet with me.
- [Karan Shingde](https://kmeanskaran.com) ([kmeanskaran.com](https://kmeanskaran.com), [@kmeanskaran](https://x.com/kmeanskaran)): mlops and inference engineer in pune who consults on llm infra and rag. writes long posts on what interviews actually ask.
- [Akshay](https://join.dailydoseofds.com) ([join.dailydoseofds.com](https://join.dailydoseofds.com), [@akshay_pachaar](https://x.com/akshay_pachaar)): cofounder of daily dose of data science, formerly at lightning ai. writes plain-language explainers on llms, agents and rag, like an llm engineer's handbook for inference serving.

## personal site enjoyers

- [judah](https://joodaloop.com) ([joodaloop.com](https://joodaloop.com), [@joodalooped](https://x.com/joodalooped)): "ordinary genius" who makes very good websites through webcraft. joodaloop.com is his eternal notepad of riffs and evergreen notes.
- [Ankit](https://ankitkr0.com) ([ankitkr0.com](https://ankitkr0.com), [@ankitkr0](https://x.com/ankitkr0)): makes his own products, like gone, an indiranagar café atlas and polymarket times, a newspaper run on prediction odds. has worked on gigabrain, buy me a coffee and spenny.
- [andrew alimbuyuguen](https://alimbuyuguen.com) ([alimbuyuguen.com](https://alimbuyuguen.com), [@aalimbuyuguen](https://x.com/aalimbuyuguen)): vc and design partner at internet studio. his site is a design and illustration portfolio.
- [srijan](https://srijan.is) ([srijan.is](https://srijan.is), [@srijancse](https://x.com/srijancse)): lives in bangalore, grew up all over india, learning catalan to talk to fellow barça fans. the site is just writing and listening.
- [Henrik Karlsson](https://www.henrikkarlsson.xyz) ([henrikkarlsson.xyz](https://www.henrikkarlsson.xyz), [@phokarlsson](https://x.com/phokarlsson)): writes escaping flatland, essays on writing as thinking, relationships, self-cultivation and llms. pinned: treat people like they are too complex to fit in your head.
- [siddharth](https://itsiddharth.design) ([itsiddharth.design](https://itsiddharth.design), [@itsiddharth_](https://x.com/itsiddharth_)): designer, engineer, artist in new york. did gold card work at robinhood and writes at timecapsule.co.in.
- [sarv](https://kami.so) ([kami.so](https://kami.so), [@SarvasvKulpati](https://x.com/SarvasvKulpati)): "computers for human flourishing". building kami, a small colorful talking pocket computer that gives you an excuse to leave your phone behind.

## design nerds

- [andrew alimbuyuguen](https://alimbuyuguen.com) ([alimbuyuguen.com](https://alimbuyuguen.com), [@aalimbuyuguen](https://x.com/aalimbuyuguen)): vc and design partner at internet studio. his site is a design and illustration portfolio.
- [Felipe](https://felipe.design) ([felipe.design](https://felipe.design), [@felipedotdesign](https://x.com/felipedotdesign)): designer at aave, previously x. seven years of 0 to 1 products and interfaces built for ai.
- [Lenard Flören](https://lenardfloeren.com) ([lenardfloeren.com](https://lenardfloeren.com), [@LenardFloeren](https://x.com/LenardFloeren)): art director and product designer in berlin who built himself a workout and yoga app with no coding background.
- [M.A. Baytaş](https://baytas.net) ([baytas.net](https://baytas.net), [@doctorbaytas](https://x.com/doctorbaytas)): designer and engineer with a phd in interaction design. creative director at moment, previously design engineer at attio, hosts design discipline.
- [Simon Sarris](https://simonsarris.com) ([simonsarris.com](https://simonsarris.com), [@simonsarris](https://x.com/simonsarris)): essays, projects and recipes from new hampshire. "the map is mostly water." latest: some notes on pleasure.

## long essays

- [Henrik Karlsson](https://www.henrikkarlsson.xyz) ([henrikkarlsson.xyz](https://www.henrikkarlsson.xyz), [@phokarlsson](https://x.com/phokarlsson)): writes escaping flatland, essays on writing as thinking, relationships, self-cultivation and llms. pinned: treat people like they are too complex to fit in your head.
- [Paras Chopra](https://invertedpassion.com) ([invertedpassion.com](https://invertedpassion.com), [@paraschopra](https://x.com/paraschopra)): building lossfunk, an ai lab. his site's motto is "know what's true and do what's right", plus a book on mental models and hundreds of threads. ✔ lossfunk gave us the openrouter credits for beacon.
- [Zara Zhang](https://zarazhang.com) ([zarazhang.com](https://zarazhang.com), [@zarazhangrui](https://x.com/zarazhangrui)): builder who made a claude skill that makes slides on the web, and writes essays like "to learn anything, first unlearn school".
- [Simon Sarris](https://simonsarris.com) ([simonsarris.com](https://simonsarris.com), [@simonsarris](https://x.com/simonsarris)): essays, projects and recipes from new hampshire. "the map is mostly water." latest: some notes on pleasure.
- [Chris Lakin](https://chrislakin.com) ([chrislakin.com](https://chrislakin.com), [@chrislakin](https://x.com/chrislakin)): researches the inner bottlenecks of people working to help superintelligence go well. writes locally optimal.
- [christian](https://cxgonzalez.substack.com) ([cxgonzalez.substack.com](https://cxgonzalez.substack.com), [@cxgonzalez](https://x.com/cxgonzalez)): runs seeing clearly, a podcast and newsletter on predictive processing, identity and creative practice. his latest is an argument about where eden was.

## people i build with

- [Sanskar Pandey](https://scholar.google.com/citations?user=04PtMiIAAAAJ) ([scholar.google.com](https://scholar.google.com/citations?user=04PtMiIAAAAJ), [@sanskxr02](https://x.com/sanskxr02)): building fidelity dynamics, post-training and evals for robot learning, previously at sarvam. ✔ beacon co-author.
- [shrinath](https://github.com/ShrinathNR) ([github.com](https://github.com/ShrinathNR), [@shrinathx](https://x.com/shrinathx)): working on light research, teaches and builds at solana turbine. ✔ built gauntlet with me.

[html page](https://sxohom.xyz/internet/) · [home](https://sxohom.xyz/) · [llms.txt](https://sxohom.xyz/llms.txt)


---

# things you could buy

> stuff i actually use. no affiliate links. prices in usd, approx.

Author: [Sohom Pal](https://sxohom.xyz/)

## Items

- [IKEA BEHÖVD vacuum flask](https://www.ikea.com/us/en/p/behoevd-vacuum-flask-light-green-beige-70353890/) · $7: keeps chai hot through a long Cursor session. looks nicer than it has any right to.
- [amazon basics foldable laptop stand](https://www.amazon.in/dp/B0BSXDGL49) · $3: raises the screen to eye level for a few dollars. neck pain is not a personality trait.
- [Solimo blackout curtains](https://www.amazon.in/Amazon-Brand-Polyester-Blackout-Curtains/dp/B0C656ZR1P) · $6: bangalore streetlights are not a sleep schedule. these make naps actually work.
- [amazon basics mini UPS for wifi router](https://www.amazon.in/Devices-Supports-Routers-Discharge-Protection/dp/B0CTXCX22D) · $9: when BESCOM blinks, the call stays up. plug it between the wall and the router.
- [Loop Quiet 2 earplugs](https://us.loopearplugs.com/products/quiet) · $25: dulls traffic and neighbors without ANC batteries. not silence. quiet enough.
- [Decathlon wooden cross-training rings](https://www.decathlon.in/p/8751180/cross-training-rings-adjustable-with-ergonomic-grip-wooden-ring-black) · $21: a full gym that hangs from a door frame. rings beat memberships i never use.
- [Designing Data-Intensive Applications, 2nd ed](https://www.amazon.com/Designing-Data-Intensive-Applications-Reliable-Maintainable/dp/1098119061) · $57: the one systems book i keep lending and buying again. tradeoffs, not buzzwords.
- [IKEA KRUX LED work lamp](https://www.ikea.com/in/en/p/krux-led-work-lamp-white-90325472/) · $26: a desk lamp shaped like a dog. warm light, and strangers always ask about it.
- [Spigen 70W GaN dual USB-C charger](https://www.amazon.com/Spigen-ArcStation-Pro-GaN-Charger/dp/B08PS5B2H8) · $19: one brick for the mac and the phone. leave the stock chargers in the drawer.
- [ORACURA OC150 water flosser](https://www.amazon.in/ORACURA-Smart-Water-Flosser-OC001/dp/B013GU4EHU) · $25: flossing i will actually do. dentists keep recommending this brand for a reason.
- [Raycast](https://www.raycast.com/) · free: spotlight, but it ships macros and window stuff. free tier is enough to start.
- [Cursor Pro](https://cursor.com/pricing) · $20/mo: how i ship. the editor that stopped feeling like an editor.
- [Fitbit Air](https://store.google.com/product/google_fitbit_air?hl=en-US) · $100: sleep and steps without a watch the size of a phone. i already wear one.
- [Kindle Paperwhite 16 GB](https://www.amazon.com/All-new-Amazon-Kindle-Paperwhite-glare-free/dp/B0CFPJYX7P) · $200: no tabs, no replies. just the book. weeks of battery if you leave wifi off.
- [14-inch MacBook Pro](https://www.apple.com/macbook-pro/) · from $1,999: the fans stay quiet and the battery outlasts the meeting.

## Keep a couple backups

It also does not hurt to keep at least a couple backups of:

- usb-c chargers and cables (forever)
- a charged 20,000 mAh power bank before monsoon week (charge weekly)
- distilled water for the RO / inverter battery if you have one (buy as needed)
- soft foam earplugs, one unopened pack (2-3 years)
- blackout curtain clips / spare curtain rings (forever)
- toothbrush heads or a spare brush (forever)
- a second pair of cheap spectacles if you wear them (forever)
- umbrellas. bangalore rain laughs at one (1-2 seasons each)
- a secondary UPI / bank account with a small float (forever)
- offline copies of aadhaar, pan, and passport scans on an encrypted drive (forever)

[html page](https://sxohom.xyz/buy/) · [home](https://sxohom.xyz/) · [llms.txt](https://sxohom.xyz/llms.txt)


---

# pocket / probable

> the proof layer · apr 2026 to july 2026

Author: [Sohom Pal](https://sxohom.xyz/)

i spent april at network school with the pocket team, close to the product, and then i stayed on probable until the end of july. what i kept running into was not whether the agent could talk about a market. it was what you have left after the run. a final message is easy. knowing which prompt caused it, which tools ran, what evidence came back, and whether anything actually happened is the part that usually disappears. that is the part i care about. i started calling it the proof layer, mostly so i would stop accepting the last bubble in the chat as the result.

the benchmark came out of that. i wrote, in the pr, that we needed something that tells us whether probable is actually making good prediction-market decisions, instead of only producing good-looking answers. so it runs the real agent on frozen cases. it scores the action and the bankroll impact, and it keeps a bad provider out of the product score. a timeout is not the model being wrong. if you mix those, you end up tuning the agent to survive your infrastructure.

research had the same hole. you could count sources after the fact and still not know what the answer was allowed to claim. i wanted the ledger first. trust decisions come from what was actually retrieved, not from a paragraph that sounds sourced. if the evidence is thin or the sources disagree, the answer is supposed to get shorter and less sure, not smoother.

i also got tired of evals that could look green while the prompt, the tools, and the runtime were not really in the test. that is the sentence i wrote on the sign-off draft. a fixture can pass forever if the thing you changed never ran. the telegram harness was my way of checking the other direction. a live proof only counted if the workflow id, the tool trace, the database row, and the ops row joined up. i had a case where the snippet we stored was null and telegram still had the answer. that is the kind of thing that makes me not trust a green check.

the local research brain is separate, and i labeled it that way on purpose. it is not production pocket code. i scaffolded it with eve on my laptop, fought the node version because eve wanted something newer than the one homebrew had given me, and kept wallets and signing out of it. the demo i actually care about is simple. ask it to buy $100 of yes right now. it should refuse and send you back to paper-only. if it cannot do that, the rest of the research ui does not matter.

## what i did

- i named and built the proof layer for probable so a run leaves prompt, tools, evidence, and side effects, not only a final message.
- i wrote the frozen-case benchmark that scores action and bankroll impact without letting infra timeouts poison the product score.
- i joined telegram live-proofs across workflow id, tool trace, database row, and ops row.

## how

- ledger-first research: trust from what was retrieved. thin or disagreeing sources shorten the answer.
- local research brain stays paper-only. wallets and signing stay out. it refuses a live $100 yes buy.
- pocket repos stay private. no production metrics claimed here.

## artifacts

- [Pocket](https://getpocketapp.com/)
- [Network School](https://ns.com/about)

## links

- [html page](https://sxohom.xyz/projects/pocket-probable/)
- [home](https://sxohom.xyz/)
- [llms.txt](https://sxohom.xyz/llms.txt)


---

# beacon

> sycophancy benchmark · 2025

Author: [Sohom Pal](https://sxohom.xyz/)

at lossfunk i was working on sycophancy. getting a model to push back on a weak claim instead of agreeing with you. you can hear it in a chat. then you try to write it down and it slips, because a free-form answer can be agreeable and careful in the same breath. i wanted it to be a choice. two responses. one more principled. one more agreeable. pick one. once it is a pick, you can score it, and you can argue about the score instead of about the vibe.

we built 420 of those pairs, by hand, and people scored them for critical thinking and fluency. a lot of the prompts came from the kinds of arguments people already have, change my view, am i the asshole, and some synthetic ones so the set was not only internet fights. we ran twelve models. the leaderboard is on a smaller subset, because the full set is for the failure analysis, not for a single number you can tweet.

the failures were more specific than "it was nice." hedged agreement. a penalty for sounding direct. folding when the prompt got emotional. picking the fluent answer over the right one. the one that annoyed me was the prompt fix. we tried preambles that tell the model to be principled, and for most models the score got worse. llama and gemma dropped hard. mixtral barely moved up. the paper calls the pattern whack-a-mole. you push one failure down and another one shows up. that is why i don't trust a paragraph in the system prompt as an alignment result.

the mitigation i take seriously is the one that had to move a number on held-out items. on llama 3.1 8b, activation steering beat the baseline, and the share of errors that were just emotional framing went down. it is one model and a small set. i would rather say that than imply we fixed sycophancy. lossfunk gave us the openrouter credits to run it. the dataset is public. the point of the project, for me, is the method. define the failure, build the probe, run the models, look at the errors, and don't call it a fix until the probe moves.

## what i did

- i co-authored beacon and built the forced-choice probe so sycophancy is a pick, not a vibe.
- i helped assemble 420 hand-built pairs and score them for critical thinking and fluency.
- i ran twelve models and kept the leaderboard on a smaller subset so the full set stays for failure analysis.

## how

- prompt preambles mostly hurt (whack-a-mole). activation steering on llama 3.1 8b moved held-out numbers.
- public dataset and arxiv paper. method over claiming a fix.

## artifacts

- [paper (arXiv)](https://arxiv.org/abs/2510.16727)
- [dataset](https://huggingface.co/datasets/sanskxr02/Beacon)
- [Lossfunk](https://lossfunk.com/)

## links

- [html page](https://sxohom.xyz/projects/beacon/)
- [home](https://sxohom.xyz/)
- [llms.txt](https://sxohom.xyz/llms.txt)


---

# solana agent safety

> gauntlet · sim engine · idl agent · 2025 to 2026

Author: [Sohom Pal](https://sxohom.xyz/)

this started because agents were being pointed at solana with tests that ask whether the task finished. swap, add liquidity, place the trade. finishing is the wrong reward if the pool is a honeypot or the mint can inflate under you. i wrote the contrast in the gauntlet readme as plainly as i could. solana gym measures what agents can do. gauntlet measures what they should not do. built to answer whether this agent is safe to ship.

there are 96 scenarios, four difficulty levels. the score is not "did it act." task is 30 percent, safety is 40, efficiency 20, capital 10. a correct refusal helps you. an unsafe execution hurts a lot. an invalid refusal hurts a little, and a silent failure hurts more. i put the anti-gaming line in the readme because i knew what people would do. you cannot get a high safety score by refusing everything. the always-execute agent is the control in my own results. it completes the task and scores terribly on safety, which is the point.

the sim engine is the other half. gauntlet tests the agent. the sim engine tests the program the agent is about to touch. you give it an anchor idl, a model proposes exploits, it builds real transactions and runs them on surfpool, and you get a report with cwe classifications. fund extraction, privilege, reentrancy, state corruption, denial of service. if it finds a high or critical issue, the process exits non-zero. i want that in a pipeline, not in a doc someone reads after the exploit.

the idl agent is what i built because i was tired of throwaway scripts. one place to load an idl, queue instructions, simulate, and send. it defaults to devnet. mainnet is behind a consent step, on purpose. if the idl has no program address, it does not pretend it can send a real instruction. a lot of this work is just refusing to blur those lines. devnet and mainnet. a completed task and a safe one. a demo and a run you can inspect later.

## what i did

- i built gauntlet: 96 scenarios, safety-weighted score, anti-gaming so refuse-everything loses.
- i built the sim engine: anchor idl in, exploit txs on surfpool, cwe report, non-zero exit on high or critical.
- i built the idl agent: load, queue, simulate, send. devnet by default, mainnet behind consent.

## how

- score = task .30 + safety .40 + efficiency .20 + capital .10.
- always-execute control completes tasks and fails safety. that is the point of the bench.
- keep demo vs inspectable run, and devnet vs mainnet, as hard lines.

## artifacts

- [Gauntlet](https://github.com/light-research/gauntlet)
- [sim engine](https://github.com/light-research/solana-sim-engine)
- [IDL agent](https://github.com/light-research/solana-idl-agent)

## links

- [html page](https://sxohom.xyz/projects/solana-agent-safety/)
- [home](https://sxohom.xyz/)
- [llms.txt](https://sxohom.xyz/llms.txt)


---

# openissue

> deterministic trace to incident · 2026

Author: [Sohom Pal](https://sxohom.xyz/)

i kept ending up with traces that were red and still not an incident. someone had to read them, decide what broke, and point at the evidence. if you hand that to a model, it will write a cleaner story than the detector. i didn't want that. the detection engine is deterministic. the model is not allowed to add, suppress, or edit the issues, the counts, the severity, or the evidence. if the reports don't contain enough to say what happened, the limitation stays in the packet. it does not invent an answer.

the demo fixture is not abstract. it is a probable execution-worker-unavailable trace. that is the connection, for me. the same class of failure i was debugging on pocket, where the worker is down and the agent still has to say something, is the example openissue ships with. the detectors are specific. worker unavailable. a cluster of tool failures. a delegated execution that never comes back with a result. i didn't want a generic "anomaly" label.

the reference lab is how i checked that i wasn't fooling myself. seven days, twenty task attempts a day, with the ground truth written down separately from the traces, never stuffed into the metadata. some days should be quiet. one day is an upstream 503 on the same tool across users. another is a missing poller. a later day repeats an earlier fingerprint, because a regression that comes back under a new name is the whole problem. fingerprints had to stay stable across a thousand reshuffles of the same events.

i also wrote down what this does not prove. a seeded langfuse trace does not mean pocket, or anyone else, had that incident in production. the lab is not customer validation. i don't think you need ten customer teams to know whether the detector is lying. you need a ground truth you didn't let the model see, and a rule that the model cannot edit the finding. the cli is the source of truth. a new detector only gets added when a real failure cannot be said with the ones that already exist.

## what i did

- i shipped a cli that turns traces into an incident packet with cited evidence.
- i kept detection deterministic. the model explains findings. it never edits them.
- i bundled a probable execution-worker-unavailable fixture (labelled demo, not a production incident).

## how

- specific detectors: worker unavailable, tool-failure cluster, delegated execution with no result.
- reference lab: 7 days x 20 attempts, ground truth outside metadata, stable fingerprints.
- published on npm as @sxohom/openissue (apache-2.0).

## artifacts

- [npm @sxohom/openissue](https://www.npmjs.com/package/@sxohom/openissue)

## links

- [html page](https://sxohom.xyz/projects/openissue/)
- [home](https://sxohom.xyz/)
- [llms.txt](https://sxohom.xyz/llms.txt)


---

# simtest

> fuzz agents in ci · 2026

Author: [Sohom Pal](https://sxohom.xyz/)

i wrote this because agents don't get tested the way apis do. nobody fuzzes them. you get a few prompts that look like the demo, and the weird task shows up later, in front of a user, or as a bill. simtest throws multi-step tasks at the agent and fails the pr if the schema breaks, the policy breaks, or the cost budget breaks. red or green. i would rather see it on the pull request than in a postmortem.

you don't need an openai key for the basic schema and exception tests. that mattered to me. if the cheap path needs a model, people won't run it. the seeds are supposed to look like a real user, an attacker, or an audit, not like a template with the nouns swapped. the seed builder i ran kept a few dozen high-signal prompts out of a few hundred generated ones. i don't want a thousand near-duplicates making the suite look serious.

the ci rules are boring on purpose. a quick fuzz is a hundred seeds, under a minute, under a dollar. the default cap is a few dollars so a loop can't wander off. if the noise between runs gets too high, or the cost jumps, the check fails. an agent test that isn't deterministic is just a vibe with extra steps. it can take traces from the frameworks people already use, langgraph, crewai, autogen, langsmith. the point is not another harness you have to adopt. the point is that the pr goes red when the agent gets worse in a way you can name.

## what i did

- i built ci fuzzing for multi-step agents: schema, policy, or cost budget breaks fail the pr.
- i kept a no-key path for basic schema and exception tests so the suite actually runs.
- i wrote a seed builder that keeps a few dozen high-signal prompts, not a thousand near-duplicates.

## how

- quick fuzz: about 100 seeds, under a minute, under a dollar. default cost cap so loops can't wander.
- fails on noise spikes and cost jumps. if it isn't deterministic, it isn't a test.
- plugs into traces from langgraph, crewai, autogen, langsmith.

## artifacts

- [GitHub](https://github.com/sohomx/simtest)

## links

- [html page](https://sxohom.xyz/projects/simtest/)
- [home](https://sxohom.xyz/)
- [llms.txt](https://sxohom.xyz/llms.txt)