Research Engineer Benchmarks and Evaluation
Ramp is an AI-powered financial operations platform for businesses, combining corporate cards, expense management, bill payments, accounting automation, procurement, travel, treasury, and related finance workflows.
About Ramp
Ramp is a privately held New York financial-services company founded in 2019. Its platform combines corporate cards, spend and expense management, bill payments, accounting automation, procurement, travel, treasury, and other financial-operations services. Ramp reports serving more than 70,000 companies.
Skills
Candidate Availability
Required and preferred rules are kept separate and reflect the wording in the original posting.
About the Role
You will turn real finance workflows, agent traces, edge cases, and expert labels into benchmark suites and RL environments. You will decide what to measure, run benchmark suites across models and agents, build calibrated graders, and convert evaluations into reward functions. You will also publish benchmarks and write-ups on model performance for finance work.
Requirements
- Experience building LLM benchmarks or evaluations used for real decisions.
- Python and software engineering skills.
- Statistical judgment covering variance, confidence intervals, sample size, and inter-rater agreement.
- Knowledge of post-training, SFT, and RL with verifiable rewards.
- SQL and large-dataset experience.
- Ability to identify meaningful measurements.
Responsibilities
- Build versioned benchmarks from finance workflows, agent traces, edge cases, and expert labels.
- Own benchmarking strategy, execute benchmark suites, and report results.
- Design rubrics, programmatic checks, and LLM-as-judge graders.
- Calibrate graders against human agreement and identify their limitations.
- Turn evaluations into reward functions and reinforcement-learning environments.
- Detect reward hacking and contamination before release.
- Publish external benchmarks and model-performance write-ups.
Benefits
- Flexible PTO
- Centralized home-office equipment ordering
- Health and wellness stipend
- Budget for intra-office travel
- Weekly coffee stipend
- 100% medical, dental, and vision insurance coverage for employees, with partial dependent coverage
- One Medical annual membership
- 401(k) with employer match
- Fertility HRA up to $10,000 per year
- Paid parental leave
- Pet insurance
- In-office meals and refreshments
- Relocation expense coverage to New York City or San Francisco
- Canadian group medical, dental, and vision coverage
- Life, AD&D, and disability coverage
- Canadian fertility drug coverage
- Group Retirement Plan with employer match
- Employee Assistance Program and virtual care
- United Kingdom private medical insurance
- Virtual GP and at-home care
- Workplace pension with salary-sacrifice option