Machine Learning Engineer Evals
Nous ResearchVisit Nous Research website
Open-source AI company building models, the Hermes Agent, a hosted agent platform, and distributed-training infrastructure.
Nous Research on X (Twitter)Nous Research on DiscordNous Research on GitHubNous Research on Documentation
Austin, United States
Funding history
About Nous Research
Nous Research, Inc. trains models, builds AI agents, and develops infrastructure for open intelligence. Its current offerings include the installable Hermes Agent and Nous Portal’s paid model, tool, and cloud-agent services.
Skills
About the Role
You will run and improve end-to-end model evaluation pipelines. You will create judge-calibration protocols, extend benchmarks with tasks and automated graders, analyze model failures, and document recommendations. You will own recurring evaluation workflows and deliver tools that researchers use.
Requirements
- 3+ years in software engineering, ML engineering, data science, or a research-adjacent role
- Experience with an LLM evaluation framework
- Hands-on LLM experience, including prompting and few-shot design
- Python
- Git
- CI/CD
- Docker
- Linux command line
- Understanding of evaluation statistics, including accuracy, Cohen's kappa, and confidence intervals
- Experience with at least three specified evaluation competencies
- Communicate clearly with researchers and engineers
Responsibilities
- Run end-to-end evaluation pipelines and reproduce known results during onboarding
- Build and document judge-calibration protocols
- Extend benchmarks with tasks, prompts, environments, rubrics, automated graders, and quality assurance
- Analyze model-output failures, quantify failure modes, and recommend improvements
- Own recurring evaluation workflows and ship evaluation tooling
