Senior Data Engineer

About Us

At Viaduct, we are developing an end-to-end machine learning platform to empower automakers to build safer, more intelligent, and personalized vehicles. Our platform increases the accessibility and actionability of connected vehicle data for automakers and their partners and end-customers. We are a diverse team motivated to solve the hardest problems in the automotive industry.

Who You Are

You are a thoughtful engineer. Comfortable being a high level IC as you lead teams. You understand the complexities of distributed systems and how to triage and solve issues that arise with them. Scalability is top of mind when designing any system or writing code. You believe building a better ETL system requires close collaboration with the machine learning and data science teams. You avoid reinventing the wheel unless necessary and are excited by opportunities to contribute to the open-source community.

Expected Skills

• 3+ years as a data engineer
  
• Experience leading teams
• Proficiency in Python and SQL
• Hands on experience with Spark or equivalent technologies
• Working proficiency with a workflow scheduler (Airflow, Prefect, Argo, etc) 
• Experience with distributed file-systems (HDFS, S3, etc)
• Familiar with the tools in open-source data ecosystem (Apache, CNCF, etc)
• Experience with incremental or real-time processing (Delta Lake, Apache Hudi, Kafka Stream, Spark Streaming, etc) 

Security and Privacy Responsibilities

• Follow our policy and procedure documents related to security and privacy
• Follow the guidelines in the Employee Handbook
• Participate in new hire and annual training for security and privacy
• Treat data security and privacy as one of your primary job responsibilities

About the Role

Day 5 

• Learn about Viaduct’s history and mission
• Get to know every team member
• Set up your development environment
• Understand Viaduct’s ETL pipelines and run your first DAGs
• Deep dive into the nuances of vehicle data
• Attend our weekly ML lunch

Day 30

• Take ownership of ETL pipelines
• Identify scalability bottlenecks in the existing ETL pipelines
• Be familiar with the day-to-day work of machine learning engineers and data scientists
• Learn the architecture of data engineering systems and services

Day 90

• Be the ETL pipeline expert at Viaduct
• Improve overall data quality and discoverability
• Confident in the scalability of Viaduct’s ETL pipelines 
• Present your work at our weekly ML lunch
• Comfortable contributing to our engineering infrastructure and systems