Technical Lead Manager, Training Accelerator Optimization

The ML Infra Training team supports Waymo's efforts to build ML models. The team focuses on the inner loop of ML training and model optimization. This includes resource consumption, job reliability, training duty cycle, and TFLOPs device utilization. To create solutions for the unique challenges in machine learning for autonomous driving, the team works with the JAX and TensorFlow teams at Google on the one hand, and with Waymo's ML practitioners in Perception, Behavior Prediction, and Planning on the other hand.

We are looking for a Technical Lead Manager with 2+ years of experience managing a team and a background in ML training frameworks, ML accelerator profiling, and scalable, distributed model training. One of the main responsibilities of this position is to identify performance bottlenecks and optimization opportunities of different neural networks and to increase the training efficiency on different types of GPUs and TPUs.

At a minimum, we'd like you to have:

• BS in Computer Science, Math, or equivalent real-world experience
• Experience improving the performance of optimization algorithms on different devices
• Real-world experience with practical challenges of mixed-precision and quantization aware training
• Prior experience on ML compiler optimization such as XLA
• Prior work on ML compiler infrastructure, such as of HLO and MLIR
• Practical expertise in scaling model training to multiple TPUs/GPUs
• Practical familiarity with using ML accelerator profiling tools to uncover performance bottlenecks
• Quantify impact of issues to prioritize work to both improve system efficiency and maintain code health
• Passionate about infrastructure work, building libraries, tools, and pipelines for machine learning practitioners
• Solid Python or C++ skills

It's preferred that you also have …

• Gained practical experience with distributed model training on popular frameworks such as Pytorch, Jax, and Tensorflow,
• Designed and built large-scale, production quality backend systems, especially in applied machine learning or data pipeline,
• Experience with knowledge distillation using Foundation Models / large LLMs,
• Previously worked on fine-tuning and PEFT, such as LoRA, and prompt tuning,
• Worked on multi-threaded and stream-based programming models,
• Experience contributing to software architecture and design, and design patterns, reliability, and challenges of scaling
• Become practiced at rolling out changes to production code reliably,
• Previously mentored or coached junior ICs, scoped and planned work for multiple engineers, chosen benchmarks, set expectations, and evaluated outcomes.

While at Waymo, you will enjoy benefits that cover…

Health and wellness: Our people are at the heart of everything we do. At Waymo, you can enjoy top-notch medical, dental and vision insurance, mental wellness support, a Flexible Spending Account (FSA), a Health Saving Account (HSA), and special wellness programs.

Financial wellness: Your financial peace of mind is important to us. At Waymo, we offer competitive compensation, bonus opportunities, equity, a generous 401(k) plan, 1-on-1 financial coaching, a 529 College Savings Plan and lots of other perks and employee discounts.

Flexibility and time off: Take the time you need to relax and recharge. Enjoy the flexibility to work from another location for four weeks per year. We support an on-site or hybrid work model and offer remote working opportunities, paid time off, bereavement, sick, and parental leave.

Supporting families: When it comes to growing your family or caring for your loved ones, you have our full support. Enhanced leave options include paid parental leave (birthing parent gets 24 weeks of paid leave with up to 4 weeks of additional leave before their due date, and non-birthing parent gets 18 weeks of paid leave), and 20 days of subsidized backup childcare or adult/elder care.. Access to fertility care or adoption support as you grow your family.

Community and personal development: At Waymo, you'll find a range of opportunities to grow, connect, and give back. We offer education reimbursement, personal and professional development, mentorship, and other ways to connect through Employee Resource Groups (ERGs), other internal groups, and even time off to volunteer.

Cool perks: Access to Google offices, cafes, wellness centers, massages, and so much more. To support your wellbeing at home, you can enjoy at-home fitness and cooking classes, and more.

#LI-Hybrid