Member of Technical Staff North Modelling Evals
Toronto-founded enterprise AI company building foundation models, agentic AI, enterprise search, and private deployment products.
About Cohere
Cohere provides enterprise-focused AI models and end-to-end products, including North for agentic workplace AI, Compass for enterprise search, and models for generation and retrieval. It supports private, VPC, and on-premises deployment options.
Skills
Candidate Availability
Required and preferred rules are kept separate and reflect the wording in the original posting.
About the Role
You will own the evaluation strategy for North workflows and build evaluation systems from product feedback, production failures, privacy-preserving usage logs, and customer needs. You will turn evaluation results and model failures into recommendations for model selection, updates, and improvements.
Requirements
- Experience improving LLM-powered, agent-powered, or AI-product systems through evaluations, feedback loops, data curation, prompting, model adaptation, or model selection
- Evaluation expertise including representative tasks, rubrics, data quality, failure analysis, and regression tracking
- Applied machine learning engineering judgment
- Ability to translate qualitative user and product signals into actionable measurement
- Understanding of model behavior, evaluation validity, product outcomes, and production tradeoffs
Responsibilities
- Own the evaluation strategy for agent workflows and enterprise knowledge work
- Build evaluations from user feedback, production failures, usage logs, customer needs, and product goals
- Create systems that continuously turn product learning into evaluations
- Translate evaluation results, customer data, and product context into recommendations for modelling teams
Benefits
- Weekly lunch stipend of $75/£75 or local-currency equivalent
- Health and dental benefits
- Mental health budget
- RRSP matching, 401K, and pension scheme
- Parental leave top-up for up to 6 months
- Arts, culture, fitness, wellness, quality-time, and workspace-improvement benefits
- Education and learning stipend
- 6 weeks of paid vacation
- Remote travel budget and annual company offsite
- Co-working benefit
- $500 home office stipend