Tech Lead Manager AI Evaluations
Glean is an enterprise AI platform that connects company knowledge and systems to provide permission-aware search, AI assistance, and agents.
Funding history
About Glean
Glean develops enterprise Work AI software. Its platform provides enterprise search, a conversational AI assistant, and tools to build, govern, and orchestrate AI agents using a company’s connected data and permissions.
Skills
Candidate Availability
Required and preferred rules are kept separate and reflect the wording in the original posting.
About the Role
You will lead applied AI scientists and engineers while setting the end-to-end technical direction for AI evaluation. You will build reusable evaluation infrastructure, define quality measurements, create representative evaluation sets, and integrate evaluations into development and production workflows. You will use results to diagnose failures, guide product decisions, and balance quality, latency, reliability, and cost.
Requirements
- 10+ years of experience in software engineering, ML infrastructure, data systems, or a related technical field, including building and operating production systems
- 5+ years of engineering or applied AI team management experience
- Experience designing, building, or operating evaluation systems for LLM or GenAI products
- Understanding of evaluation methods, datasets, metrics, human review, calibration, alignment, and statistical analysis
- Understanding of agent evaluation, including tool calling, multi-turn behavior, trajectories, task completion, and error recovery
- Strong coding ability in Python or another production language
- Experience with scalable data or developer platforms, experiment pipelines, or CI/CD integrations
- Sound engineering fundamentals in testing, APIs, data modeling, and reliable distributed systems
- Strong judgment about dataset representativeness, calibration, reproducibility, uncertainty, and metric optimization risks
- Ability to communicate findings and influence cross-functional roadmaps
- MS or PhD in Computer Science, AI, or a related field
Responsibilities
- Own the AI evaluation engineering roadmap and technical direction
- Build scalable, reusable evaluation infrastructure for datasets, experiments, graders, trace capture, analysis, and dashboards
- Create and maintain representative evaluation sets and protected acceptance sets
- Define quality measurements for AI products and agents
- Develop deterministic checks, model-based graders, and human evaluation processes
- Validate and calibrate automated judges against human judgments
- Integrate evaluations into CI, pre-release testing, production monitoring, and experiments
- Build feedback loops from user feedback and production traces into new test cases
- Connect evaluation outcomes to latency, reliability, and cost
- Lead, coach, and develop applied AI scientists and engineers
- Partner with product, infrastructure, security, and application teams
Benefits
- Medical coverage
- Vision coverage
- Dental coverage
- Generous time-off policy
- 401k plan
- Home office improvement stipend
- Annual education stipend
- Annual wellness stipend
- Regular company events
- Daily healthy lunches
Hiring Process
Brief AI-focused exercise or discussion as part of the interview process.