Product Evals Engineer, ChatGPT Learning
AI research and deployment company building and deploying frontier AI products, including ChatGPT, Codex, and its API platform.
Projects
About OpenAI
OpenAI’s mission is to ensure artificial general intelligence benefits all of humanity. It consists of the nonprofit OpenAI Foundation and OpenAI Group, a public benefit corporation governed by the Foundation.
Skills
Candidate Availability
Required and preferred rules are kept separate and reflect the wording in the original posting.
About the Role
You will lead development of systems that evaluate response quality in ChatGPT's learning experiences. You'll define quality metrics, build scalable evaluation pipelines combining human judgment and model-based grading, create datasets and benchmarks, design experiments, analyze failure modes and build tools that make results actionable. This role is hybrid in San Francisco.
Requirements
- Strong software engineering skills including data pipelines and evaluation infrastructure
- Understanding of information retrieval concepts and metrics
- Experience with human-labeled data, relevance judgments, annotation or model-based evaluation
- Comfort with qualitative and quantitative data analysis
- Ability to turn ambiguous goals into testable quality definitions
- Clear communication of technical findings
Responsibilities
- Build and evolve evaluation frameworks for ChatGPT Learning
- Define quality metrics and evaluation criteria
- Develop scalable evaluation pipelines combining human judgment, behavioral signals, model-based grading and deterministic checks
- Create representative datasets and benchmarks
- Design experiments to identify quality gaps
- Analyze evaluation results and user behavior
- Improve reliability of automated evaluators through calibration and validation
- Partner with engineering and research teams
- Build tools for accessible evaluation results
- Help establish best practices for measuring AI search quality
Benefits
- Equity
- Performance-related bonus
- Medical, dental and vision insurance with HSA contributions
- Health FSA, Dependent Care FSA and commuter pre-tax accounts
- 401(k) with employer match
- Paid parental leave
- Paid medical and caregiver leave
- Flexible PTO
- 13+ paid company holidays and office closures
- Mental health and wellness support
- Employer-paid life and disability coverage
- Learning and development stipend
- Daily meals and meal delivery credits
- Relocation support
- Charitable donation matching and wellness stipends
