Director of Evaluations
Montréal-based nonprofit AI research organization developing safe-by-design, non-agentic AI through its Scientist AI approach.
Funding history
About LawZero
LawZero conducts AI safety research and develops technical solutions for highly capable safe-by-design AI. Its Scientist AI research seeks transparent, evidence-based prediction without goal-directed behavior, with potential applications in scientific discovery and oversight of agentic AI systems.
Skills
Candidate Availability
Required and preferred rules are kept separate and reflect the wording in the original posting.
About the Role
You will define and lead evaluation strategy, build an independent evaluations function, and develop benchmarks, datasets, red-teaming programmes, and scalable tooling. You will evaluate AI capabilities, safety, explainability, and adversarial robustness, communicate results publicly, and represent evaluation work externally.
Requirements
- Advanced degree in machine learning, computer science, or a closely related field
- 10+ years of machine learning experience
- At least 5 years of technical team leadership experience
- Expertise designing and running large-scale evaluations of LLMs or frontier ML systems
- Experience building evaluation datasets, benchmarks, or interactive environments
- Written and verbal communication skills
Responsibilities
- Define the evaluations strategy and roadmap
- Build and scale the Evaluations Team
- Maintain independence from research and product streams
- Design benchmarks, datasets, tasks, and interactive environments
- Evaluate capabilities, safety, explainability, and adversarial robustness
- Lead guardrail evaluations for frontier models
- Establish automated and manual red-teaming programmes
- Build evaluation tooling and infrastructure
- Publish evaluation results through reports, system cards, publications, and blog posts
- Represent evaluation work with external organizations
Benefits
- Comprehensive health benefits
- Minimum 20 days of vacation per year upon start
- Minimum 4% employer retirement savings contribution
- Flexible benefits for well-being