AI Benchmarking Analyst

An evaluation engineer who transforms subjective perceptions of AI model performance into rigorous, reproducible measurement systems — designed to ensure deployment decisions rest on evidence, not vendor demos or cherry-picked examples.

- The BASELINE methodology — 7-pillar benchmarking framework from scoping through maintenance and escalation

- Task taxonomy development mapping abstract qualities to concrete, measurable behaviors with capability decomposition

- Test set curation with stratified sampling, contamination auditing, and canary string analysis

- Rubric engineering achieving Cohen's kappa above 0.75 through iterative inter-annotator calibration

- Statistical rigor: bootstrap confidence intervals, paired significance testing, effect size estimation, power analysis

- Multi-dimensional evaluation frameworks covering latency-quality tradeoffs, human-AI alignment, safety, regression detection

- Layered reporting — executive summaries, product dashboards, technical appendices with explicit limitation disclosure

- Benchmark maintenance playbooks with version control, contamination monitoring, production outcome correlation tracking

Drop into Claude, ChatGPT, Cursor, or any AI tool. Bring your real benchmarking problem — a model selection decision, a regression you need to catch, a safety evaluation you need to operationalize, a vendor claim you need to verify independently. It thinks like an evaluation engineer who has built benchmarks that survive adversarial scrutiny and hold up under Goodhart's Law.

Bundles or prompts related to AI quality assurance, model evaluation methodology, and product analytics.