Evaluation Engineer Applied AI
Mistral AIVisit Mistral AI website
Paris-based AI company building frontier models, AI applications, developer tools, and compute infrastructure for enterprise and public-sector deployments.
Paris, France
About Mistral AI
Mistral AI develops open-weight and commercial language models and provides a full-stack AI platform spanning agents, application development, custom-model training, APIs, and AI cloud infrastructure.
Skills
Candidate Availability
Required and preferred rules are kept separate and reflect the wording in the original posting.
About the Role
You will discover priority use cases with customers and experts, design and score benchmarks, and build evaluations in real software. You will source or create data, run evaluations on models, identify flaws, collaborate with post-training teams, and present actionable results to leaders.
Requirements
- At least 2 years of industry experience in software, AI, or consulting, or a PhD
- Software engineering or research background
- Experience writing code that others can build on
- Experience working directly with customers, industry experts, or other teams
- Experience building with LLMs
- Experience working with large datasets and cleaning them
- Strong Python skills
- Experience working in a large shared codebase and code review
- Understanding of LLM training, inference, and evaluation
- Fluency in English
Responsibilities
- Run discovery with customers, industry experts, and internal teams to prioritize use cases
- Design benchmarks for industry use cases and frontier capabilities
- Define benchmark scoring methods
- Build and run evaluations where agents work in real software
- Maintain reliable evaluation results over time
- Stress-test evaluations for flaws and bugs
- Source data with expert vendors
- Build proxy or synthetic data when customer data cannot leave the customer
- Present findings to leaders
- Help teams build reusable evaluation tools and guides
Benefits
- Healthcare coverage
- Parental leave
- Retirement plans
- Relocation support
- Wellness programs
- Meal allowances
- Transportation allowances
- Location-specific perks