Senior Software Engineer - Model Evaluation & AI Systems

Deepgram is the leading platform underpinning the emerging trillion-dollar Voice AI economy providing real-time APIs for speech-to-text (STT) text-to-speech (TTS) and building production-grade voice agents at scale. More than 200000 developers and 1300+ organizations build voice offerings that are ‘Powered by Deepgram’ including Twilio Cloudflare Sierra Decagon Vapi Daily Cresta Granola and Jack in the Box. Deepgram’s voice-native foundation models are accessed through cloud APIs or as self-hosted and on-premises software with unmatched accuracy low latency and cost efficiency. Backed by a recent Series C led by leading global investors and strategic partners Deepgram has processed over 50000 years of audio and transcribed more than 1 trillion words. There is no organization in the world that understands voice better than Deepgram.

At Deepgram we expect an AI-first mindset—AI use and comfort aren’t optional they’re core to how we operate innovate and measure performance.

Every team member who works at Deepgram is expected to actively use and experiment with advanced AI tools and even build your own into your everyday work. We measure how effectively AI is applied to deliver results and consistent creative use of the latest AI capabilities is key to success here. Candidates should be comfortable adopting new models and modes quickly integrating AI into their workflows and continuously pushing the boundaries of what these technologies can do.

Additionally we move at the pace of AI. Change is rapid and you can expect your day-to-day work to evolve just as quickly. This may not be the right role if you’re not excited to experiment adapt think on your feet and learn constantly or if you’re seeking something highly prescriptive with a traditional 9-to-5.

Deepgram is looking for a Senior Software Engineer - Model Evaluation & AI Systems to join the team responsible for validating the quality of our speech audio and multilingual models before they reach customers.

This team owns the evaluation and quality assurance surfaces that ensure Deepgram's models — across speech-to-text text-to-speech and increasingly LLM- and multimodal-powered systems — meet their performance targets in both batch and streaming environments. We build the pipelines harnesses canaries and test frameworks that catch regressions hallucinations and quality issues before they impact customers and we partner closely with Research to turn model expectations into automated reproducible enforceable checks.

In this role you'll define evaluation methodology and build the infrastructure that measures model quality at scale. You'll create evaluation pipelines define pass/fail criteria grounded in Research benchmarks and build the monitoring that keeps models honest in production. Your work provides the trusted signals that inform release and optimization decisions and directly protects the customer experience.

We're looking for a strong engineer who is equally comfortable building test infrastructure and reasoning about model behavior. This role is aimed at senior engineers with broad instincts for quality measurement and automation; hands-on experience evaluating modern AI systems is a strong plus.

• Define and build evaluation methodologies for Deepgram's models spanning speech-to-text text-to-speech and emerging LLM RAG agent and multimodal systems.

• Design build and maintain automated evaluation pipelines across batch and streaming (e.g. WER runaway/hallucination detection latency and time-to-first-byte) with a focus on correctness reproducibility and ease of adoption.

• Build scalable reproducible evaluation infrastructure — harnesses orchestration and result-aggregation pipelines — running against production models and where needed large GPU clusters.

• Translate Research benchmarks and expected model metrics into automated enforceable pass/fail gates.

• Build and operate canaries and continuous-monitoring systems that detect quality regressions in production before they reach customers.

• Partner with DevOps/Infra to stand up ephemeral test environments and results-aggregation infrastructure.

• Work alongside Research model training inference and product teams to provide trusted evaluation signals that inform release and optimization decisions.

• Integrate evaluation and quality gates into CI/CD so quality is verified continuously not manually.

• Help raise the bar through code reviews technical design discussions and strong engineering and QA practices.

• BS MS or PhD in Computer Science AI Applied Math or a related field or equivalent experience.

• 5+ years of professional software or QA engineering experience with a track record of shipping test infrastructure or evaluation systems (senior candidates with significantly deeper experience welcome).

• Solid backend/scripting experience in a language such as Python Rust Go or similar.

• Experience designing and building automated test pipelines evaluation frameworks or data-processing systems.

• Strong analytical skills and comfort reasoning about metrics thresholds and statistical variation in results — able to distinguish real regressions from noise.

• Ability to take charge of ambiguous technical challenges and communicate effectively across research engineering and product teams.

• Hands-on experience evaluating modern AI systems such as LLMs RAG pipelines agents or multimodal models including model behavior analysis.

• Experience with React Native or other cross-platform mobile frameworks for building tooling that's accessible beyond the desktop.

• Experience building or improving evaluation frameworks benchmarks or ML infrastructure used by other teams or external users.

• A strong appreciation for evaluation quality — correctness reproducibility and consistency across environments.

• Experience with voice audio speech recognition or real-time systems and familiarity with metrics like WER MOS or latency/TTFB.

• Prior involvement in open-source projects through contributions reviews maintenance or community engagement.

• Experience acting as a technical bridge across teams or platforms (evaluation training inference agent frameworks) combining architectural understanding with clear communication and influence.

• Familiarity with cloud infrastructure containerized/ephemeral environments and monitoring tooling (e.g. Grafana canaries anomaly detection).

What We Do

Deepgram is the leading voice AI platform for developers building speech-to-text (STT) text-to-speech (TTS) and full speech-to-speech (STS) offerings. 200000+ developers build with Deepgram’s voice-native foundational models – accessed through APIs or as self-managed software – due to our unmatched accuracy latency and pricing. Customers include software companies building voice products co-sell partners working with large enterprises and enterprises solving internal voice AI use cases. The company ended 2024 cash-flow positive with 400+ enterprise customers 3.3x annual usage growth across the past 4 years over 50000 years of audio processed and over 1 trillion words transcribed. There is no organization in the world that understands voice better than Deepgram.

Why Work With Us

Our culture like our product is constantly learning and evolving but the heart of our team is enduring. We are a self-motivated positive passionate and competitive group of people. At Deepgram we put an emphasis on being ourselves being curious growing together and being human. We are a unique bunch who celebrate our differences.

Gallery