Staff Software Engineer

Now, more than ever, the Toast team is committed to our customers. We’re taking steps to help restaurants navigate these unprecedented times with technology, resources, and community. Our focus is on building the restaurant platform that helps restaurants adapt, take control, and get back to what they do best: building the businesses they love. And because our technology is purpose-built for restaurants, by restaurant people, restaurants can trust that we’ll deliver on their needs for today while investing in experiences that will power their restaurant of the future.

Are you bready* for a change?

At Toast, our Site Reliability Engineers (SREs) are responsible for keeping all customer- facing services and other Toast production systems running smoothly. SREs are a blend of pragmatic operators and software craftspeople who apply sound software engineering principles, operational discipline, and mature automation to our environments and our codebase. Our decisions are based on instrumentation and continuous observability as well as through predictions and capacity planning.

About this roll* (Responsibilities)

• Define, implement and evolve a world-class observability technology stack that allows rapid detection of issues in our system and enables root cause analysis (25%)
• Provide scalable metrics and dashboarding solutions for R&D 
• Provide distributed tracing capabilities to visualize and track issues across our complex system 
• Provide log aggregation and insights for R&D using best in class technology 
• Provide a global view of the true customer experience through usage of Real-User Monitoring & external cloud-based solutions
• Act as a champion for reliability and work with partner teams in different lines of business  to influence product roadmaps to improve resiliency and reliability of all services. Champion our uptime targets and enable other teams  to improve the way we measure the reliability of the system (25%)
• Provide technical leadership in production triage, incident resolution, and retrospective/root cause analysis to maintain the reliability and uptime of our platform (20%)
• Leverage a strong understanding of Cloud Architecture 
• Knowledge of Java and the JVM (Java Virtual Machine) to triage and understand issues within services 
• Implement strategies to increase system reliability and performance through on-call rotation and process optimization
• Lead incident post-mortem/retrospectives to surface reliability improvements and drive to completion
• Mentor and coach peers and reliability champions on SRE best practices. Contribute to running an SRE Guild (15%)
• Design, build and drive adoption of a platform that enables service resilience testing/chaos engineering to validate and test Toast’s architecture is resilient to failure. Build and own a performance testing framework/environment to enable our R&D teams to understand the constraints of their services and improve  performance (15%)

Do you have the right ingredients*? (Requirements)

• Extensive and broad industry experience with at least 7+ years in building and running production systems and participating in incident calls 
• Comfortable reading, writing, and debugging object oriented languages - Java and Python etc.
• Well-versed in software architecture and deep understanding of cloud and microservices 
• Demonstrated experience working with at least one major cloud platform (AWS, GCP, or Azure)
• Exposure to complex, mission critical, and large scale distributed systems
• Ability to set an example for the team with positive and inclusive leadership and discussion on work.
• General knowledge of most technical expertise areas, with deep knowledge in two areas.
• Observability platforms (Datadog, NewRelic, Splunk, AppD, etc.) - APM, RUM, Synthetic monitoring
• Prometheus, Thanos, and Grafana: service catalog metrics and recording rules for alerts
• Log shipping pipelines and incident debugging visualizations
• Block and object storage configuration and debugging
• Database experience (DynamoDB, Aurora, RDS)
• Advanced Terraform syntax and GitLab CI/CD configuration, pipelines, jobs
• Containers: cluster provisioning and new services

Bonus ingredients*

• Deep experience with Database technologies
• Experience working with android and/or IoT devices

*Bread puns encouraged but not required