Senior Applied AI Scientist

What You'll Do

• Design and own evaluation frameworks for production LLM features including LLM-as-a-judge evaluations regression suites synthetic datasets golden datasets and human review workflows.
• Analyze production behavior to identify quality issues hallucinations latency bottlenecks cost regressions and emerging failure modes.
• Design and run experiments including prompt variations workflow changes retrieval improvements and model comparisons; and quantify their impact on quality operational metrics and user outcomes.
• Define the metrics that matter and build dashboards that make AI performance visible across the organization.
• Partner with engineering to determine which optimizations should be productionized and how to measure ongoing success.
• Mentor teammates on experimental design statistical rigor evaluation methodology and measurement best practices.

Who You Are

• 5+ years of experience in data science applied machine learning experimentation or a closely related field with at least the last year focused on applied LLMs or AI evaluation.
• Strong Python and SQL skills with experience working on production data pipelines and experimentation.
• You have experience designing reproducible evaluation frameworks rather than relying on manual spot checks or qualitative assessments.
• You have strong statistical intuition: you think in terms of distributions confidence intervals variance and sample sizes rather than anecdotes.
• You’re comfortable working closely with engineers and product teams to translate experimental findings into production improvements
• Bonus: Experience with evaluation platforms (e.g. Braintrust LangSmith OpenAI Evals) experimentation platforms causal inference healthcare or operations-heavy environments.

What We Do

Ro is a direct-to-patient healthcare company with a mission of helping patients achieve their health goals by delivering the easiest most effective care possible. Ro is the only company to offer nationwide telehealth labs and pharmacy services. This is enabled by Ro's vertically integrated platform that helps patients achieve their goals through a convenient end-to-end healthcare experience spanning from diagnosis to delivery of medication to ongoing care. Since 2017 Ro has helped millions of patients in nearly every single county in the United States including 98% of primary care deserts.

Why Work With Us

Ro is powering quality care at scale. The Ro Operating System (ro.OS) vertically integrates the core parts of healthcare bringing together nationwide telehealth lab and pharmacy services on one platform. The result? ro.OS makes it easier for patients to access and providers to deliver high-quality care – millions of times over.

Gallery