Lead Data Engineer(8+ Years of Exp)

At 3Pillar, our focus is on leveraging cutting-edge technologies that revolutionize industries by enabling data driven decision making. As a Lead, Data Engineering, you will hold a crucial position within our dynamic team, actively contributing to thrilling projects that reshape data analytics for our clients, providing them with a competitive advantage in their respective industries.

Key Responsibilities

• Understanding the client requirement and architect robust data platforms on multiple cloud technologies.
• Take the lead on creating, defining and driving the data engineering solutions for complex client engagements
• Develop ETL/ELT processes to help extract and manipulate data from multiple sources.
• Work with the leadership to set the standards for software engineering practices within the data engineering team
• Maintain and optimize the data infrastructure required for accurate extraction, transformation, and loading of data from a wide variety of data sources.
• Monitor data systems performance and implement optimization strategies.
• Ensure data accuracy, integrity, privacy, security, and compliance through quality control procedures.
• Partner with data scientists and functional leaders in sales, marketing, and product to deploy machine learning models in production.
• Choose and use the right analytical libraries, programming languages, and frameworks for each task.
• Lead data engineering team.

Minimum Qualifications

• Demonstrated expertise with a minimum of 7+ years of experience in data engineering domain
• Advanced SQL skills and experience with relational databases and database design.
• Experience with data modeling
• Experience working with cloud Data Warehouse solutions (e.g., Snowflake, Redshift, BigQuery, Azure Synapse, etc.).
• Strong Python skills with hands-on experience on Pandas, NumPy and other data related libraries
• Experience with working on large data sets and distributed computing (e.g. Hive/Hadoop/Spark/Presto/MapReduce). ​​
• In depth knowledge and understanding of data streaming tools like Apache Kafka, AWS Kinesis.
• Proficient in data pipeline and workflow management tools e.g. Airflow, AWS Step functions etc
• Experience with data visualization tools like PowerBI, Tableau, AWs Quicksight etc
• Experience with the AWS/Azure/GCP data engineering services
• Knowledge of AWS services viz., EC2, EMR, S3, Lambda, EBS, IAM, Redshift, RDS, GLUE ETL
• OR
• Knowledge of Azure services viz., ADF, Databricks, Azure Synapse Analytics, ADLS Gen2, Azure Devops
• Good understanding of NoSQL databases like Redis, Cassandra, MongoDB, or Neo4j.
• Excellent problem-solving, communication, and organizational skills.
• Proven ability to manage a highly skilled data engineering team.