AI Experiment Lead

A rigorous experimentalist who architects evaluation infrastructure for AI features — designing experiments that separate genuine model improvements from noise, regression from progress, and user delight from statistical flukes.

- CALIBER methodology: clarify decision, architect evaluation stack, lock design, instrument, build analysis, evaluate, register learnings

- Hypothesis specification for AI with falsifiable claims distinguishing model quality from UX or prompt changes

- Offline evaluation suite design with test sets, edge cases, regression benchmarks, and LLM-as-judge calibration

- Human evaluation protocol creation with inter-annotator agreement targets and annotator fatigue management

- Sample size and power analysis for stochastic outputs accounting for high variance in generative models

- Online experimentation guardrails: gradual rollout ramps, automatic kill switches, segment degradation detection

- Pre-registration discipline preventing p-hacking and post-hoc metric selection before results arrive

- Experiment registry and artifact reuse: test sets, rubrics, scoring pipelines propagated across teams

Drop into Claude, ChatGPT, Cursor, or any AI tool. Bring your real experiment problem — a model improvement you need to validate, evaluation metrics that don't match business outcomes, a rollout that needs guardrails, pressure to ship without measurement. It thinks like a data scientist who's shipped AI features through organizational chaos and learned to make it harder to be wrong.

Bundles or prompts related to AI quality, experimentation infrastructure, and product metrics.