Senior Observability Engineer

As a Senior Observability Engineer at AffiniPay, you will help develop and execute our observability program that will enhance existing Cloud and Site Reliability Operations, driving innovation, modernization, and delivery of reliable and performant cloud capabilities at scale.  You will be a member of the newly emerging Platform Engineering Organization and will be responsible for the delivery and management of the  availability, performance and security of the infrastructure and core service layers that make up our core payment platform and practice management software serving over 200,000 legal professionals and processing over $18 billion every year!  

Our ideal candidate will have a deep engineering background (specifically with various cloud technologies), a strong advocacy for observability as a means for delivery of products that are secure and performant, be passionate about surfacing and resolving problems at scale, and ability to collaborate with many teams and stakeholders to make a significant impact on the business and what we deliver to our users. This position is responsible for providing design and implementation leadership of the Observability tooling stack, consolidation of existing tools into a common program for all products, and being the resident SME for Observability to the Engineering and Product Organization. As this team grows, this critical role will be the leader on the team and a trusted advisor to Senior Engineering Leadership.

Your day to day: 

• Lead Engineer in the design, build, management and adoption of the Observability Program for the Enterprise Platform of products that is 100% cloud based.
• Leverage your experience in making tooling decisions based on value, adoption, ease of delivery and usability for a growing, international Product and Engineering organization.
• Assist leadership with the development and execution of a comprehensive strategy for ensuring the reliability, scalability, and performance of our SaaS based product lines.
• Own the Observability function for the organization and the tooling stack that supports it.
• Optimize and enforce best practices for site reliability engineering, incident management, and monitoring.
• Energetically advocate for SLOs that scale effectively by communicating issues to the product owners and engineers.
• Optimize incident response protocols to ensure minimal downtime and rapid issue resolution ideally with automated assignment based on Error Budgets and SLOs.
• Collaborate with Engineering teams in the Payments and Practice Management products to define and deliver a common Observability program with common tooling.
• Drive towards zero-downtime deployments leveraging tooling and scopes, writing tools to automate and monitor services, functionalities and other tools using technologies, including Python scripting, Ruby, Terraform, Bash, JSON, or YML or XML templates
• Monitor system performance and identify opportunities for optimization, load balancing, and resource allocation.
• Understand, shape and enrich rigorous security practices across both all Platform resources.
• Collaborate with cross-functional teams, including development, security, and compliance, to ensure alignment and successful product delivery.
• Able to design tooling and streams of metrics and monitors that support different stakeholders and functions in the organization like compliance, InfoSec, Accounting, Sales
• Stay informed about industry trends, emerging technologies, and best practices in SRE and infrastructure.
• Drive innovation and process improvements to enhance system reliability and operational efficiency.

What we are looking for:

• Bachelor's degree in Computer Engineering, Computer Science, or a related field (or equivalent experience).
• 5+ years of experience with developing and deploying automation services for continuous monitoring and observability tools such as DataDog (preferred), ELK, Dynatrace, New Relic. Splunk or other mature APM platforms.
• 5+ years of experience in software delivery as a DevOps Engineer or SRE delivering verifiable value to the DevX of an organization.
• Deep experience with large scale distributed systems, cloud based infrastructure, advanced container concepts, container runtimes (containerd, docker) and container orchestration frameworks (kubernetes, openshift) and instrument monitoring of all  environments.
• Effective communication skills with demonstrated ability to effectively work with ICs and Management alike communicating needs, potential enrichment of feature sets and gap analysis.
• Skill with infrastructure-as-code, configuration management, and CI/CD a must.
• Advanced to expert practitioner of instrumenting and interpreting database performance metrics in MySQL, postgress or other RDMS.
• Fully proficient with Terraform having leveraged it in day-to-day working environment to codify everything from INF to configurations to deployment pipelines and security.
• Expert level knowledge of monitoring solutions, incident management, and SRE principles with verifiable experience. 
• Extensive AWS experience is a must, experience with GCP or Azure is also desirable.
• If you live near one of our offices in either San Diego or Austin - we work in a hybrid capacity!