Member of Technical Staff, Evals
Magic is an AI research and engineering company building frontier code models to automate software engineering and research.
Funding history
About Magic
Magic AI, Inc. develops long-context foundation models and agentic systems for software engineering, combining pre-training, reinforcement learning, ultra-long context, and inference-time compute in pursuit of safe AGI.
Skills
Candidate Availability
Required and preferred rules are kept separate and reflect the wording in the original posting.
About the Role
You will develop new benchmarks and evaluation methodologies, run data ablations, assess dataset quality, and investigate whether measurements accurately reflect model performance. You will sweat the details, auditing public benchmarks, papers, and open-source implementations for subtle bugs or flawed assumptions, and build tools that make evaluation results useful across the company.
Requirements
- Experience developing model evaluations, benchmarks, or ML experiments
- Strong programming skills and experience working with machine learning systems
- Track record of owning technical projects end-to-end
- Skepticism toward results that cannot be reproduced, validated, or explained
- Experience analyzing model performance and interpreting experimental results
- Comfortable navigating ambiguity and determining whether a measurement captures the behavior it claims to measure
- Commitment to raising the standard of rigor in AI evaluations
Responsibilities
- Design and validate new evaluations and benchmarks across model training and deployment
- Investigate model behavior and identify gaps in existing evaluations
- Measure dataset quality and identify opportunities to improve training data
- Improve evaluation correctness, reproducibility, and reliability
- Audit and improve upon public benchmarks, research papers, and open-source implementations
- Define metrics that accurately reflect model quality
- Build tools that make evaluation results accessible and useful
Benefits
- Significant equity as part of total compensation
- 401(k) plan with 6% salary matching
- Health, dental, and vision insurance for you and your dependents
- Unlimited paid time off
- Visa sponsorship and relocation support for candidates moving to San Francisco