Software Engineer Evals
Glean is an enterprise AI platform that connects company knowledge and systems to provide permission-aware search, AI assistance, and agents.
Funding history
About Glean
Glean develops enterprise Work AI software. Its platform provides enterprise search, a conversational AI assistant, and tools to build, govern, and orchestrate AI agents using a company’s connected data and permissions.
Skills
Candidate Availability
Required and preferred rules are kept separate and reflect the wording in the original posting.
About the Role
You will design and build evaluation pipelines for assistant and agent quality, evaluate model releases, and create quality signals for regressions and launch readiness. You will build observability infrastructure, trace-enrichment and telemetry pipelines, dashboards, and debugging workflows. You will own backend systems through production rollout, reliability, monitoring, and iteration while connecting evaluation results to product improvements.
Requirements
- 6+ years of software engineering experience building backend systems, infrastructure, distributed systems, or data platforms
- Coding skills in Go, Python, Java, C++, or similar languages
- Experience with distributed data pipelines, production services, observability systems, or cloud-native infrastructure
- Analytical rigor in evaluating whether metrics reflect user experience
- Experience with LLM applications, evals, tracing, data warehouses, workflow orchestration, or ML infrastructure is a plus
Responsibilities
- Design and build large-scale evaluation pipelines
- Evaluate frontier and open-source model releases
- Build infrastructure and quality signals for regressions, tradeoffs, and launch readiness
- Build agent-observability infrastructure, trace enrichment, telemetry pipelines, dashboards, and debugging workflows
- Own backend systems from architecture through production rollout, reliability, monitoring, and iteration
- Connect evaluation results, customer feedback, regression analysis, and engineering workflows to product improvements
- Build systems that balance speed, reliability, security, and cost
- Mentor engineers and raise the bar for technical design
Hiring Process
AI-focused exercise or discussion