Director of Evaluations
Montréal-based nonprofit AI-safety research organization developing safe-by-design advanced AI, centered on its Scientist AI approach.
Funding history
About LawZero
LawZero develops technical solutions and research for highly capable, non-agentic, safe-by-design AI systems. Its Scientist AI research aims to provide transparent, evidence-based prediction and potential oversight/guardrails for agentic AI.
Skills
Candidate Availability
Required and preferred rules are kept separate and reflect the wording in the original posting.
About the Role
You will define and lead evaluation strategy, build an independent evaluations function, and develop benchmarks, datasets, red-teaming programmes, and scalable tooling. You will evaluate AI capabilities, safety, explainability, and adversarial robustness, communicate results publicly, and represent evaluation work externally.
Requirements
- Advanced degree in machine learning, computer science, or a closely related field
- 10+ years of machine learning experience
- At least 5 years of technical team leadership experience
- Expertise designing and running large-scale evaluations of LLMs or frontier ML systems
- Experience building evaluation datasets, benchmarks, or interactive environments
- Written and verbal communication skills
Responsibilities
- Define the evaluations strategy and roadmap
- Build and scale the Evaluations Team
- Maintain independence from research and product streams
- Design benchmarks, datasets, tasks, and interactive environments
- Evaluate capabilities, safety, explainability, and adversarial robustness
- Lead guardrail evaluations for frontier models
- Establish automated and manual red-teaming programmes
- Build evaluation tooling and infrastructure
- Publish evaluation results through reports, system cards, publications, and blog posts
- Represent evaluation work with external organizations
Benefits
- Comprehensive health benefits
- Minimum 20 days of vacation per year upon start
- Minimum 4% employer retirement savings contribution
- Flexible benefits for well-being
