Senior Product Manager, AI Inference - Dynamo

This position is posted by Jobgether on behalf of a partner company. We are currently looking for a Senior Product Manager, AI Inference - Dynamo in United States.

This role sits at the cutting edge of AI infrastructure and large-scale inference systems, focusing on how next-generation models are served efficiently across distributed GPU environments. You will define and drive the product strategy for a high-performance inference framework that powers LLM and generative AI workloads at massive scale. Working at the intersection of hardware and software, you will help shape innovations in routing, caching, and memory optimization to dramatically improve latency, throughput, and cost efficiency. This is a deeply technical product leadership role requiring fluency in AI systems and distributed architectures. You will collaborate closely with engineering teams, researchers, and ecosystem partners to translate advanced system capabilities into scalable, production-ready products. Your work will directly influence how modern AI applications are deployed and experienced globally.

• Define and own the product strategy and roadmap for distributed AI inference systems and core framework components
• Drive development of intelligent routing systems to optimize inference performance, reduce redundant computation, and improve time-to-first-token
• Lead strategy for KV cache management, memory offloading, and multi-tier caching architectures for large-scale LLM serving
• Collaborate with engineering teams on hardware-software co-design to maximize performance across GPU-accelerated infrastructure
• Develop product requirements documents (PRDs) and system design documentation to guide engineering execution
• Define agentic inference capabilities including prioritization, multi-turn reasoning support, and dynamic output control
• Partner with ecosystem teams and open-source communities to integrate and align with frameworks such as vLLM, SGLang, and TensorRT-LLM
• Translate low-level system performance improvements into measurable business outcomes such as reduced cost and improved latency
• Work with technical program managers to align roadmaps, release cycles, and delivery timelines
• Engage with customers and stakeholders to incorporate real-world model evaluation into end-to-end inference workflows
• Monitor industry trends in LLMs, distributed systems, and AI infrastructure to inform long-term product direction

Requirements:

• 12+ years of experience in product management, technical leadership, or equivalent engineering/product hybrid roles in AI or infrastructure domains
• Strong background in AI inference systems, distributed computing, or GPU-accelerated platforms
• Deep understanding of LLM inference workflows including prefill/decode stages and KV cache mechanics
• Experience working with large-scale distributed serving systems and performance optimization techniques
• Ability to translate complex technical systems into clear business value and customer impact
• Strong technical fluency with the ability to collaborate effectively with research and engineering teams
• Experience working in highly matrixed, cross-functional environments with strong influencing skills
• Customer-focused mindset with the ability to build products aligned with real-world usage needs
• Strong analytical and data-driven decision-making skills across product lifecycle management
• Familiarity with agentic AI frameworks, MLOps, or generative AI systems is highly desirable
• Hands-on technical background in AI/ML systems is a strong plus, including reading or applying research in product strategy

Benefits:

• Competitive base salary ranging from $208,000 – $327,750 USD
• Equity participation as part of total compensation package
• Comprehensive health, dental, and vision insurance coverage
• Strong focus on employee and family wellbeing benefits
• Access to cutting-edge AI infrastructure and world-class engineering teams
• Opportunity to shape foundational AI inference systems at global scale
• Inclusive, innovation-driven culture focused on accelerated computing and AI advancement
• Career growth within a leading organization in AI and high-performance computing