Senior Data Engineer

What You'll Do

As a Senior Data Engineer on the data platform team, we’ll rely on your expertise across multiple disciplines to develop, deploy and support data systems, data pipelines, data lakes, and lakehouses.  Your ability to automate, performance tune, and scale the data platform will be key to your success.

Your initial areas of focus will include:

• Collaborate with stakeholders to make effective use of core data assets
• With Spark and Pyspark libraries, load both streaming and batched data
• Engineer lakehouse models to support defined data patterns and use cases
• Leverage a combination of tools, engines, libraries, and code to build scalable data pipelines
• Work within an IT managed AWS account and VPC to stand up and maintain data platform development, staging, and production environments
• Documentation of data pipelines, cloud infrastructure, and standard operating procedures
• Express data platform cloud infrastructure, services, and configuration as code
• Automate load, scaling, and performance testing of data platform pipelines and infrastructure
• Monitor, operate, and optimize data pipelines and distributed applications
• Help ensure appropriate data privacy and security
• Automate continuous upgrades and testing of data platform infrastructure and services
• Build data pipeline unit, integration, quality, and performance tests
• Participate in peer code reviews, code approvals, and pull requests
• Identify, recommend, and implement opportunities for improvement in efficiency, resilience, scale, security, and performance

What You Need to Get the Job Done (if you don’t have all, apply anyway!)

• Experience developing, scaling, and tuning data pipelines in Spark with PySpark
• Understanding of data lake, lakehouse, and data warehouse systems, and related technologies
• Knowledge and understanding of data formats, data patterns, models, and methodologies
• Experience storing data objects in hadoop or hadoop like environments such as S3

• Demonstrated ability to deploy, configure, secure, performance tune, and scale EMR and Spark 
• Experience working with streaming technologies such as Kafka and Kinesis
• Experience with the administration, configuration, performance tuning, and security of database engines like Snowflake, Databricks, Redshift, Vertica, or Greenplum
• Ability to work with cloud infrastructure including resource scaling, S3, RDS, IAM, security groups, AMIs, cloudwatch, cloudtrail, and secrets manager
• Understanding of security around cloud infrastructure and data systems
• Git-based team coding workflows

Bonus Skills (Not Required, So Apply Anyway!)

• Experience deploying and implementing lakehouse technologies such as Hudi, Iceberg, and Delta
• Experience with Flink, Presto, Dremio, Databricks, or Kubernetes
• Experience with expressing infrastructure as code leveraging tools like Terraform
• Experience and understanding of a zero trust security framework
• Experience developing CI/CD pipelines for automated testing and code deployment
• Experience with QA and test automation
• Exposure to visualization tools like Tableau

Beyond the technical skills, we’re looking for individuals who are:

• Clear communicators with team members and stakeholders
• Analytical and perceptive of patterns
• Creative in coding
• Detail-oriented and persistent
• Productive in a dynamic setting

If you love to learn, you’ll be in good company. You’ll likely have a Bachelor’s degree in computer science, information systems, or equivalent working experience.