User Researcher AI Evaluations
Notion is an AI workspace for capturing team knowledge, finding answers, and automating work with agents.
Funding history
About Notion
Notion Labs, Inc. operates a collaborative workspace that combines documents, knowledge management, databases, projects, search, and AI-enabled automation. Its developer platform provides an API for connecting workspaces to external tools and automating workflows.
Skills
Candidate Availability
Required and preferred rules are kept separate and reflect the wording in the original posting.
About the Role
You will define reusable evaluation criteria for AI experiences and run recurring surveys, studies, comparisons, and human-in-the-loop evaluations. You will study workflows, failure modes, and recovery behavior, then turn findings into guardrails and prioritization guidance. You will partner with product, design, engineering, and data science to build scalable evaluation loops.
Requirements
- Ability to turn user expectations into rubrics, scoring guidelines, and observable metrics
- Experience evaluating AI-enabled products and working with data science or machine-learning partners
- Ability to communicate quality definitions and research findings to diverse partners
- Strong quantitative and qualitative UX research skills
- Experience with interviews, benchmarking, surveys, and experiments
- 5+ years of UX research experience in industry
Responsibilities
- Define reusable evaluation frameworks, rubrics, scoring guidance, and metrics
- Run recurring longitudinal and feature-specific surveys and studies
- Lead qualitative studies, side-by-side comparisons, and human-in-the-loop evaluations
- Ensure evaluations reflect user workflows, intent, and interaction journeys
- Identify failure modes, regressions, edge cases, and recovery behavior
- Translate findings into guardrails, fixes, and prioritization guidance
- Collaborate with product, design, engineering, and data science to build scalable evaluation loops
Benefits
- Equity