Data Scientist – AI & Agentic Applications & Benchmarking

WFH Flexible • Los Angeles, CA • Product Description About CloudBees CloudBees is the leading software delivery platform for enterprise DevOps teams. As a high-growth startup, we empower developers to build, deploy, and manage software more efficiently. Now, we’re bringingagentic intelligenceinto our platform to supercharge developer workflows—and we need a data scientist who can both drive insights and tell the story behind the metrics. The Role CloudBees is seeking a startup-savvyData Scientistto help define, measure, and evangelize the impact ofAgentic Applicationsacross our platform. You’ll work closely with engineers and product teams toprototype and measure AI and Agentic experiences, using evals, telemetry, and AI benchmarksto help the company drive the conversation in the market and with customers. Translating performance into clear, compelling narratives to our customers and internal teams. As a founding member of the team, you will lead the charge as equal partsbuilder, evaluator, and communicator—with the technical depth to prototype in Python notebooks, Claude Code, and other tools to drive clarity to write about what matters. Key Responsibilities • Partner with our platform team to develop and prototypetelemetry, eval frameworks, and benchmarksfor emerging agentic systems. • Partner with product and engineering teams tomeasure AI outcomes and usage across customers and teams. • Help define KPIs and success metrics for AI and LLM-driven features and workflows. • Use Python notebooks toexplore data, visualize insights, and test hypotheses rapidly and share insights. • Tell the story behind the numbers: Write internal documentation, performance summaries, and thought leadership around outcomes. • Enable engineering teams toinstrument, log, and evaluateagent performance effectively. • Stay up to date with evolving metrics and evaluation techniques in the LLM and agentic AI ecosystem. Required Qualifications • 3+ years of experience in data science or ML analytics roles, ideally in startup or high-growth environments. • Proficiency inPython, including building and sharing analysis viaJupyter notebooks. • Experience working with evals, telemetry, A/B testing, and evaluating user-facing ML systems. • Experience with AI/ML tools such as MLFlow, Hugging Face, or other Model / LLM tools. • Ability to partner with technical teams to define meaningful metrics and benchmarks. • Clear communication skills—capable ofwriting about outcomes, sharing learnings, and influencing stakeholders. • Comfort working in fast-paced, ambiguous environments where speed and clarity matter. Preferred Qualifications • Experience withagentic or LLM-based applications(e.g., evaluating AI copilots, autonomous workflows). • Familiarity with tools like LangSmith, OpenInference, or custom eval stacks. • Background in developer tools, DevOps, or platform engineering environments. Why Join CloudBees • Shape the future of AI-driven DevOps with real user impact. • Join a nimble, passionate team at the forefront of agentic system development. • Work in a flexible, remote-first culture built on trust and innovation. • Competitive salary, startup equity, and excellent benefits. CloudBees is proud to be an Equal Opportunity Employer.We value diverse voices, ideas, and experiences as essential to building great products.