Senior Product Operations Manager, Evaluation Quality
Harvey is an AI platform for legal and professional-services organizations.
About Harvey
Harvey provides agents and collaborative tools for legal work, including legal research, document analysis, secure workspaces, contract intelligence, and organization-level analytics.
Skills
Candidate Availability
Required and preferred rules are kept separate and reflect the wording in the original posting.
About the Role
You will own the quality bar behind human evaluations, defining what good looks like for eval methodology and data analysis. You will author guidelines and instructions, standardize rubric design, onboard and calibrate contract attorneys, run QA on vendor deliverables, analyze eval results, and build dashboards to track eval-program health while partnering with Applied Legal Research, Product, Engineering and AI Research.
Requirements
- 6+ years in product operations, research operations, evaluation/QA operations, or quality program management
- Track record of owning quality inputs for expert-driven or human-in-the-loop work
- Experience onboarding, training and calibrating distributed expert raters, annotators or reviewers
- Grounding in measurement concepts such as calibration, inter-rater reliability, sampling and rubric design
- Experience running quantitative and qualitative data analyses and statistical tests
- Record of scaling evaluation quality processes with bias toward documentation and reproducibility
- Ability to work with domain experts and translate judgment into documented standards
- Strong cross-functional coordination
- Clear communicator
- Bias to action and high ownership
Responsibilities
- Own the quality bar for human evaluations and produce decision-grade outputs
- Author and maintain evaluation guidelines, instructions and databases
- Standardize rubric and evaluation design into repeatable templates and one documented methodology
- Own contract-attorney quality including onboarding, calibration training, inter-rater reliability and feedback loops
- Conduct quantitative and qualitative statistical analyses and investigate root causes
- Run QA on vendor and contract-attorney deliverables
- Ensure the quality bar holds across jurisdictions and non-English geographies
- Build lightweight operational dashboards to track rater capability and eval-program health
- Support ALR in a review step that certifies evaluations before scaling
- Partner with ALR and Analytics to align human eval with online signal
Benefits
- Equity
