Research Engineer Data and Evaluation

2 hours agoSalary: 150K - 350KSan Francisco, USA Remote Full Time Ai Jobs by Prime Intellect

Skills

Agent Data Pipeline Dataset Curation Distributed Processing Evaluation Language Model Machine Learning Python Reinforcement Learning Reward Design Statistical Testing Synthetic Data Verifier

Candidate Availability

Remote · San Francisco · Required

Required and preferred rules are kept separate and reflect the wording in the original posting.

About the Role

You will design evaluations for coding and long-horizon agents, build executable environments and verifiers, investigate model failures and reward hacks, and scale RL data pipelines. You will work with researchers to validate whether task and training-data changes improve performance on held-out tasks.

Requirements

Python software engineering skills
Experience building data pipelines, evaluation systems, or ML workflows
Experience with language models or agents
Understanding of training-data and evaluation effects on model behavior
Ability to identify failure modes and create testable measurements
Experience owning technical projects through experimentation, implementation, and validation
Experience building technical evaluations or agent environments

Responsibilities

Design and build evaluations for coding and long-horizon agents
Build executable environments with realistic task setup and success criteria
Develop and test scoring logic and verifiers
Investigate reward hacks and misleading progress measures
Trace failures through model outputs and harness code
Build and scale RL data pipelines for coding and agentic tasks
Test task and training-data changes on held-out tasks

Benefits