Software Engineer, Cybersecurity Benchmarks
Independent, data-first nonprofit research institute investigating AI progress, capabilities, infrastructure, and societal impacts.
Funding history
About Epoch AI
Epoch AI publishes data, research, benchmarks, and interactive tools on the drivers, progress, and impacts of artificial intelligence; it also provides commissioned research, model evaluations, and consultations.
Skills
About the Role
You will help build trustworthy, public evaluations of AI cybersecurity capabilities. Day to day you will build and maintain cybersecurity benchmarks, primarily in the Inspect library, harden the sandboxes they run in, and keep the benchmarks running so new model releases can be evaluated quickly. You will also use threat modelling to shape the evaluation strategy, integrate with AI providers, and work with researchers, analysts and engineers to make sure results are accurate and feed into research products.
Requirements
- Strong software engineering experience, including several years building and maintaining complex systems and contributing robust, maintainable code
- Experience sandboxing software against highly capable adversaries
- Red team experience ethically hacking software
- Familiarity with agentic AI use and coding harnesses
- Ability to generate own ideas for new benchmarks and experiments
- Motivation from Epoch AI's mission
- Professional level English proficiency
- Nice to have: experience benchmarking AIs and running evaluations
- Nice to have: experience setting up CTF infrastructure
- Nice to have: experience using modern AI for red teaming
- Nice to have: knowledge of inner workings of AI coding harnesses and tool calls
- Nice to have: familiarity with inspect_ai
- Nice to have: solid grasp of current AI trends
Responsibilities
- Design, build and maintain cybersecurity benchmarks for AI within the evaluation infrastructure, primarily using the Inspect library
- Pitch and prototype ideas for new benchmarks and experiments
- Keep the benchmark suite running to evaluate new model releases quickly
- Sandbox and harden evaluation environments so highly capable AI models cannot escape them
- Stay current on how AI agents and coding harnesses interact with their environments
- Use threat modelling to inform the benchmarking approach and decide what to measure
- Provide strategic input on AI cybersecurity evaluation strategy
- Integrate with AI providers and set up benchmarks to run on infrastructure
- Work with researchers, analysts and engineers to ensure evaluation data and outputs are accurate and integrated into research products and publications
Benefits
- Fully remote work
- Flexible work hours
- Three company retreats per year
