Principal Software Engineer, Elastic Search

Comcast's Technology, Product & Experience organization works at the intersection of media and technology. Our innovative teams are continually developing and delivering products that transform the customer experience. From creating apps like TVGo to new features such as the Talking Guide on the X1 platform, we work every day to make a positive impact through innovation in the pursuit of building amazing products that are enjoyable, easy to use and accessible across all platforms. The team also develops and supports our evolving network architecture, including next-generation consumer systems and technologies, infrastructure and engineering, network integration and management tools, and technical standards. Because our employees are our most valuable asset, we offer a flexible working environment to balance the need to work independently wherever you choose, with days that require collaboration at one of our offices. Our roles primarily reside inside or around one of our Tech Hubs (Philadelphia and surrounding suburbs, Denver, Austin, Silicon Valley, Washington DC/N. Virginia).

Job Summary

The DevX Run Observability Team develops, delivers, and operates platforms and services to enable Comcast's engineering and operations teams to understand the health of our products and services in real-time. Our massively scaled systems support thousands of users and store petabytes of data.

The DevX Team is an entire organization committed to improving the developer experience at Comcast. Within DevX, the Observability Team endeavors to reduce the friction for developers to integrate with outstanding observability platforms using best practices and efficient deployment patterns.

With this goal in mind, we seek an Principal Elastic Engineer to join our team in a highly-scaled Elasticsearch environment. The ideal candidate will have a strong understanding of data structures and experience working with Elasticsearch in a high-scale environment.

Job Description

The successful candidate will have a strong understanding of distributed systems and experience with large data sets. In addition, you should be comfortable working with data in various formats (e.g., JSON, CSV, etc.) and be able to design and implement data pipelines using technologies such as Vector, Logstash, and Beats.

In addition to your technical responsibilities, you will also be expected to mentor a team of engineers and assist the manager in guiding team members' development and career growth. Therefore, you should have a proven track record of successfully mentoring and leading teams and be able to provide mentorship and support to team members as they grow in their roles.

What You'll Do:

• Demonstrate your technical expertise to maintain and improve a highly scaled ELK environment that ingests over 200 TB of data per day.
• Provide technical leadership to a team of engineers and a manager
• Provide SME-level feedback to stakeholders on environmental improvements
• Manage/Perform upgrades and performance testing.
• Assist with data normalization
• Assist in fixing and remediating Elastic Search-related issues
• Employ best practices for managing systems and services across all environments
• Fault finding, analysis, and logging information for reporting performance exceptions
• Proactively monitoring system performance and capacity planning
• Create and modify scripts or applications to perform tasks
• Lead development efforts for automation strategies and deployment processes
• Work closely with supporting platform teams to ensure optimal environment operation
• All other duties as assigned.

What You Need:

• 10+ years of software or related technology experience
• 3+ years of experience working with high-volume Elasticsearch environments
• Configuration management experience through Ansible/Terraform/Chef/Puppet or like tools
• Experience with distributed systems and large data sets
• Experience with data pipelines and ingestion tools (e.g., Vector, Logstash, Beats)
• Experience with data management: hot/warm/cold architectures, shard allocation/re-allocation, snapshots & restoration
• Experience securing the Elastic stack and hardening hosting environments
• Experience sizing infrastructure and deploying services
• Experience supporting containerized services in a production environment
• Experience integrating Elasticsearch with external systems
• Intermediate to expert-level software development skills in Python, Java, Javascript, Go, or similar language class
• Cloud infrastructure in AWS
• Demonstrated team mentorship and product ownership
• Knowledge of Kubernetes is a plus

Employees at all levels are expected to:

• Understand our Operating Principles; make them the guidelines for how you do your job.
• Own the customer experience - think and act in ways that put our customers first, give them flawless digital options at every touchpoint, and make them promoters of our products and services.
• Know your stuff - be hardworking learners, users, and advocates of our groundbreaking technology, products, and services, especially our digital tools and experiences.
• Win as a team - make big things happen by working together and being open to new insights.
• Be an active part of the Net Promoter System-a way of working that brings more employee and customer feedback into the company - by joining huddles, making callbacks, and helping us elevate opportunities to do better for our customers.
• Drive results and growth.
• Respect and promote inclusion & diversity.
• Do what's right for each other, our customers, investors, and our communities.

Disclaimer:

• This information has been designed to indicate the general nature and level of work performed by employees in this role. It is not designed to contain or be interpreted as a comprehensive inventory of all duties, responsibilities, and qualifications.

Comcast is an EOE/Veterans/Disabled/LGBT employer.

We believe that benefits should connect you to the support you need when it matters most, and should help you care for those who matter most. That's why we provide an array of options, expert guidance and always-on tools that are personalized to meet the needs of your reality-to help support you physically, financially and emotionally through the big milestones and in your everyday life.

Please visit the benefits summary on our careers site for more details.

Education

Bachelor's Degree

While possessing the stated degree is preferred, Comcast also may consider applicants who hold some combination of coursework and experience, or who have extensive related professional experience.

Certifications (if applicable)

Relative Work Experience

10 Years +

Comcast is proud to be an equal opportunity workplace. We will consider all qualified applicants for employment without regard to race, color, religion, age, sex, sexual orientation, gender identity, national origin, disability, veteran status, or any other basis protected by applicable law.