Member of Technical Staff - Pre-Training Evals Infra
Reflection is an AI research lab building open frontier models and a full AI stack for developers, enterprises, and public-sector users.
Funding history
About Reflection
Reflection develops open-weight AI models, open-source software for customizing and running agents, AI-factory infrastructure, and related solutions. Its current research emphasizes large language models, reinforcement learning, and agentic reasoning.
Skills
Candidate Availability
Required and preferred rules are kept separate and reflect the wording in the original posting.
About the Role
You will build evaluation infrastructure for researchers to understand model capabilities, guide training decisions, and assess release readiness. You will design evaluation abstractions, develop distributed execution systems, optimize inference performance, ensure correctness and reproducibility, and create tools for inspecting and debugging results. You will monitor evaluation workloads, investigate failures, and turn research needs into reusable self-service systems.
Requirements
- Proficiency in Python, C++, or Rust.
- Ability to reason about concurrency, memory behavior, and performance.
- Experience designing, building, and operating reliable distributed systems.
- Knowledge of model inference and evaluation.
- Ability to investigate complex problems across system boundaries and identify root causes.
- Experience translating technical-user needs into APIs and usable tools.
- Ability to own ambiguous problems, prioritize competing needs, and balance delivery with reliability and maintainability.
- Attention to detail when investigating discrepancies in model outputs, evaluation results, and system behavior.
Responsibilities
- Design and evolve evaluation abstractions, APIs, and data models.
- Build scalable and reliable distributed infrastructure for evaluating model checkpoints.
- Optimize inference and distributed-execution performance, throughput, GPU utilization, and time to results.
- Establish evaluation correctness and reproducibility through validation, regression testing, versioning, and provenance.
- Build researcher-facing interfaces and dashboards for model-output inspection, experiment comparison, and debugging.
- Monitor evaluation workloads and results, triage failures, and develop alerts and diagnostics.
- Partner with researchers to turn bespoke workflows into reusable self-service systems.
Benefits
- Stock options.
- Comprehensive medical, dental, vision, and life insurance.
- Annual wellness allowance.
- Daily in-office lunch and dinner.
- 22 weeks of paid parental leave.
- Unlimited paid time off in the U.S.
- 30 days of vacation in the U.K.
- Visa sponsorship support.
- Regular off-sites, happy hours, and team celebrations.