Senior Big Data Engineer

We seek a Senior Big Data Engineer to join our remote team for an exciting project focused on developing ETL and ELT pipelines for aggregating commerce purchase and catalogue data from various sources into the data lake.
In this role, you will be responsible for architecting and maintaining our code base for ETL and ELT pipelines, large batch/micro-batch processing, and streaming systems. You will also build the infrastructure required for optimal extraction, transformation, and loading of data from various sources using ADF, Spark, Kafka, or similar technologies.

#LI-DNI#LI-TG3

Responsibilities

• Design and maintain our code base for ETL and ELT pipelines, large batch/micro-batch processing, and streaming systems
• Build out the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using ADF, Spark, Kafka, or similar technologies
• Identify, design, and implement internal process improvements, automate manual processes, optimize data delivery, re-designing infrastructure for greater scalability, etc
• Act as a single point of responsibility over any delivery-related matters, including escalations, upsells, ramp-downs for top management and stakeholders
• Ensure a sound and future-proof architecture is planned and the implementation meets the technical quality standards
• Coordinate between multiple disciplines and stakeholders, and ensure that projects are delivered in line with the client's processes and methodologies
• Establish a strategy of continuous delivery risk management that enables proactive decisions and actions throughout the delivery life cycle
• Measure and improve delivery productivity, and serve as a consultant to the Data Engineers in support of quality and timely delivery of work
• Perform production support and deployment activities

Requirements

• Bachelor's or Master's degree in Computer Science, Engineering, or a related field
• Minimum of 3 years of experience in Data Software Engineering
• Expertise in Apache Spark, Databricks, Scala, and SQL
• Proficiency in Apache Kafka, PySpark, and Spark Streaming
• Skills in building and deploying data pipelines in a cloud environment
• Strong knowledge of data warehousing concepts, data modeling, and ETL/ELT processes
• B2+ English level, both verbal and written

Nice to have

• Familiarity with Microsoft Azure cloud platform

EPAM Georgia is a team of innovators united by a passion for technology. The dynamic and inclusive culture we embrace helps positively impact our communities, clients, and employees. Here you will collaborate with multi-national teams, contribute to numerous cutting-edge projects, deliver the most creative solutions, and have an opportunity to learn. Our people are at the heart of our success, and we are proud to provide talents with a solid ground to develop and grow.