Applied Research Engineer Evaluation

Skills

About the Role

You will build and maintain evaluation pipelines for model checkpoints, finetuning runs, and competitor benchmarks. You will design automated and human evaluations, validate metric reliability, identify model weaknesses, define target metrics, and use API and MCP signals to improve evaluation coverage.

Requirements

• Meaningful hands-on experience evaluating generative models
• Advanced degree in a relevant field or equivalent practical experience
• Strong Python and software engineering skills
• Working knowledge of PyTorch and modern machine learning infrastructure
• Strong knowledge of statistics, sample sizes, inter-annotator agreement, and confidence levels
• Experience building and operating LLM-as-judge systems
• Knowledge of modern benchmarking practices, datasets, and benchmarks

Responsibilities

• Build and maintain automated benchmarking, LLM-as-judge, and human-evaluation pipelines
• Design evaluation coverage for prompt adherence, editing fidelity, aesthetics, text rendering, consistency, and safety
• Validate agreement and stability between human and automated evaluations
• Benchmark models against competitors and interpret performance differences
• Identify output-quality weaknesses not captured by aggregate metrics
• Define target metrics for reinforcement learning and finetuning work and verify improvements
• Design use of API and MCP signals and identify new signals to capture
• Maintain metrics and anticipate gaming and overfitting risks

Benefits

• Equity
• Coverage of reasonable travel costs