Associate Staff Engineer

Company Description

We are a Digital Product Engineering company that is scaling in a big way! We build products, services, and experiences that inspire, excite, and delight. We work at scale — across all devices and digital mediums, and our people exist everywhere in the world (19000+ experts across 33 countries, to be exact). Our work culture is dynamic and non-hierarchical. We are looking for great new colleagues. That is where you come in!

Job Description

We are seeking a skilled and experienced Data Engineer who can assist with managing scheduled DAGs and data workflows. The ideal candidate should have expertise in modern data engineering tools and technologies, particularly in the Azure ecosystem, and be comfortable working in a virtual machine environment via terminal.

Key Responsibilities:

• Design, develop, and manage workflows using Apache Airflow, ensuring scheduled DAGs run efficiently and reliably.
• Work with Azure Data Factory (ADF) and Azure Data Lake Storage (ADLS) for data integration and storage solutions.
• Develop and maintain data pipelines and ETL processes using Databricks, PySpark, and Python.
• Monitor, troubleshoot, and optimize data workflows and pipelines to ensure high availability and performance.
• Collaborate with cross-functional teams to understand data requirements and deliver scalable solutions.
• Manage and operate within virtual machine environments using terminal commands for development and debugging.

Qualifications

Must have Skills: Azure Data Factory, Azure Pipeline Management (Strong), Python (Strong), Databricks, Airflow.

• Proficiency in Apache Airflow for orchestrating workflows and managing DAGs.
• Strong experience with Azure cloud services, including ADF and ADLS.
• Hands-on experience with Databricks and PySpark for big data processing and analytics.
• Advanced programming skills in Python, with a focus on data manipulation and automation.
• Comfort with terminal operations and commands in VM environments.
• Solid understanding of data engineering principles, including data modeling, ETL processes, and data pipeline optimization.
• Strong problem-solving skills and the ability to work independently in a dynamic environment.
• Experience with other Azure services or similar cloud platforms is a plus.
• Familiarity with DevOps practices and tools for CI/CD in data engineering is a plus.
• Knowledge of database technologies such as SQL and NoSQL systems is a plus.
• Understanding of data security and compliance standards is a plus.