User Researcher AI Evaluations

Notion is the collaborative AI workspace where teams and agents think together. We're building one place where your knowledge projects meetings and AI tools live side by side so work is faster clearer and less fragmented. Millions of individuals small teams and large companies run their work on Notion.

Notinos (our employees) are customer zero in bringing this future of work to life. We care about craft building things that last and the belief that great work is still fundamentally human. Our goal isn’t to ship the next feature. Each and every team of Notinos is working to set the standard for how humans work together in the AI era. From building a business’s system of record to making and managing AI agents to automating away the busy work we care deeply about giving our customers more time for their life’s work.

We’re seeking an experienced UX Researcher to define and scale how we evaluate Notion’s AI-powered experiences—focusing on what “good” looks like not only for model output quality but for the end-to-end product experience where people discover set goals delegate work review results and build trust over time with AI.

This role sits at the intersection of research craft and evaluation operations: you’ll run studies that uncover user mental models expectations and failure/recovery behaviors then translate those insights into reusable rubrics workflows and measurement approaches that product design engineering and data science can apply consistently.

This role can be based in either San Francisco or New York City. We work from our offices on Mondays Tuesdays and Thursdays (our Anchor Days) because we do our best thinking and building together in person. We’re looking for someone who’s excited to work alongside the team during those days.

• Define what “good” looks like (frameworks & rubrics): Establish clear reusable evaluation criteria that reflect real user expectations—helpfulness trust tone control and transparency. You’ll translate qualitative insight into scoring guidance that can be applied consistently across teams and over time.

• Run recurring evals (longitudinal & feature-specific): Run recurring longitudinal and feature-specific surveys and studies to measure experience quality over time against defined rubrics. Lead qualitative studies side-by-side comparisons and human-in-the-loop evaluation efforts to deepen understanding of where experiences break down and how they can improve. You’ll help teams spot regressions benchmark improvements and understand when expectations shift.

• Anchor evaluation in real workflows (context > isolated feedback): Ensure evals reflect jobs-to-be-done user intent and the full interaction journey (goal setting delegation review iteration) not just decontextualized thumbs up/down. You’ll help teams understand who is evaluating what they’re trying to do and why outputs succeed or fail.

• Identify failure modes & recovery behavior (guardrails): Uncover breakdowns regressions and edge cases across the system—from model behavior to UI and integrations—and study how people notice issues correct them and continue their work. You’ll turn these insights into actionable guidance for guardrails fixes and prioritization.

• Operationalize evaluation with partners (process & tooling): Collaborate closely with Product Design Engineering and Data Science to align on target use cases and build scalable evaluation loops (human-in-the-loop review longitudinal studies and calibration of automated/LLM-judge approaches against human judgment).

• Ability to operationalize insight into measurement: You’re comfortable turning “soft” user expectations (trust tone usefulness clarity) into concrete rubrics scoring guidelines and observable metrics.

• AI fluency and systems thinking: You’re curious and hands-on with AI products and can reason about how model behavior uncertainty and system constraints shape user experience. You also have experience evaluating AI-enabled products (LLMs agents generative UI/workflow automation) and working with Data Science/ML partners on measurement strategy and evaluation tooling.

• Clear communication and impact orientation: You can align diverse partners around shared definitions of quality and create artifacts that enable teams to act consistently. You tailor storytelling to different audiences connect research to business outcomes and drive follow-through so insights translate into product change.

• Strong UX research craft (quant + qual): You can choose the right methods for the question— interviews benchmarking surveys experiments—and synthesize into actionable guidance. You also can prioritize ruthlessly work through ambiguity and balance scrappy iteration with deep dives when needed.

• Pragmatism in fast-moving environments: You can prioritize ruthlessly work through ambiguity and balance scrappy iteration with deep dives when needed.

• Experience: 5+ years doing UX research in industry

• Familiarity with LLM-as-judge methods prompt design for evaluators or “golden dataset” creation

• Experience using AI research tooling for rapid synthesis and communication (e.g. Dovetail Listen Labs Maze Outset etc.) as well as AI observability tooling like Braintrust

• Experience using data querying languages (e.g. SQL) scripting languages (e.g. Python) or statistical/mathematical software (e.g. R SAS Matlab etc.)

• Master’s or PhD in HCI Psychology Behavioral Science Anthropology Sociology or a related field

• You’re familiar with the work of computing heroes like Douglas Engelbart Alan Kay Bret Victor etc. — and understand why we're big fans.

Notion is committed to providing highly competitive cash compensation equity and benefits. The compensation offered for this role will be based on multiple factors such as location the role’s scope and complexity and the candidate’s experience and expertise and may vary from the range provided below. For roles based in San Francisco or New York City the estimated base salary range for this role is $196000-$230000 per year.

By clicking “Submit Application” I understand and agree that Notion and its affiliates and subsidiaries will collect and process my information in accordance with Notion’s Global Recruiting Privacy Policy and NYLL 144.

#LI-Onsite

You don’t need deep AI expertise for every role but we do expect every Notino to be intellectually curious drawn to tinkering and discovery and excited to use AI as a real collaborator in their work. For some roles AI fluency is a core requirement — when that’s the case we'll say so explicitly in the qualifications. People who thrive here don’t treat AI as a novelty. They use it to think better and make their work easier for others to build on.

We hire talented people from a wide range of backgrounds. If you’re excited about this role but don’t meet every bullet we still encourage you to apply. Notion is an equal opportunity employer and does not discriminate on the basis of any legally protected characteristic. Consistent with applicable law we will consider for employment qualified applicants with arrest and conviction records. Notion provides reasonable accommodations during the application process; if you need one please let your recruiter know.

Notion is proud to be an equal opportunity employer. We do not discriminate in hiring or any employment decision based on race color religion national origin age sex (including pregnancy childbirth or related medical conditions) marital status ancestry physical or mental disability genetic information veteran status gender identity or expression sexual orientation or other applicable legally protected characteristic. Notion considers qualified applicants with criminal histories consistent with applicable federal state and local law. Notion is also committed to providing reasonable accommodations for qualified individuals with disabilities and disabled veterans in our job application procedures. If you need assistance or an accommodation due to a disability please let your recruiter know.

What We Do

Notion is a collaborative AI workspace. Teams use it to store knowledge run projects take meeting notes and build AI workflows — all in one place so AI always has the context it needs to be useful.We're building the layer where AI stops being a chat window and starts being part of how work gets done. Knowledge decisions and action in one connected system.

Why Work With Us

We're building the layer where AI stops being a chat window and starts being part of how work gets done. Notion is where AI becomes useful at work: real workflows real users real context. People here move from idea to shipped without asking permission. If you want your fingerprints on work that millions of people use this is the place.

Gallery