Member of Technical Staff - Evaluations

Skills

About the Role

You will work end to end on the internal eval suite and infrastructure, keeping it reliable and reproducible at scale. You will design held-out evals for capabilities like agentic workflows, computer use, and cybersecurity, build evals for small edge models, debug unexpected scores by reading outputs, and publish open-source evals and papers.

Requirements

• Strong eval design skills producing accurate, general-purpose, hard-to-game evaluations
• Previously built evals, at least one from scratch (benchmark, eval paper, or contributions to lm-eval-harness, lighteval, OLMES, HELM, or Inspect)
• Fluency with contamination, saturation, judge bias, and variance
• Hands-on experience running and managing evals in a team
• Habit of debugging scores by reading generated answers
• Software engineering skills to own eval infrastructure and debug inference and infra failures at scale

Responsibilities

• Maintain the internal eval suite and infrastructure so it is reliable, optimized, and reproducible at scale
• Design held-out evals for priority capabilities such as agentic workflows, computer use, and cybersecurity, with contamination checks
• Build evals targeted at small edge models
• Read generated outputs to debug unexpected scores
• Turn customer and open-source feedback into environments for data, training, and evaluation
• Partner with other teams to design evals for their needs
• Release open-source evals, write papers, and publish failure-mode reviews of popular evaluations

Benefits

• Equity
• Unlimited PTO
• Company-wide Refill Days
• Comprehensive health coverage
• Paid leave for pregnancy, parenting, medical needs, and family care