ETL Technical lead

ETL Technical lead

Location: Chennai

ETL experience: 5+years

Azure Synapse experience: 3+ years

Must have skills

• 5+ years of experience in designing, developing, and managing ETL processes, with at least 3 years of hands-on experience in the Azure ecosystem.
• Proficiency with Azure Synapse Pipelines, Azure Dedicated SQL Pool, Azure Data Lake Storage (ADLS), and other related Azure services.
• Experience with audit logging, data governance, and implementing data integrity and data lineage best practices.
• Experience in securing and managing secrets using Azure Key Vault.
• Strong experience working with data formats such as Parquet, JSON, and XML.
• Hands-on experience implementing the Medallion architecture (Bronze, Silver, Gold layers).
• Experience with Logic Apps and pipeline automation, including setting up email notifications for alerts.
• Demonstrated experience in integrating ETL pipelines with ML models.
• Excellent SQL skills, with the ability to optimize complex queries in a large-scale data environment.
• Proven ability to lead a technical team, resolve technical issues, and mentor team members.
• Strong understanding of data warehousing principles, data governance, and best practices in data management.

Key Responsibilities:

• Lead and manage the ETL team, providing mentorship, technical guidance, and driving the delivery of key data initiatives.
• Design, develop, and maintain ETL pipelines using Azure Synapse Pipelines for ingesting data from various file formats, such as Parquet, JSON, and XML, and securely storing them in Azure Data Lake Storage (ADLS).
• Architect, implement, and manage data solutions following the Medallion architecture for effective data processing and transformation.
• Leverage Azure Data Lake Storage (ADLS) to build scalable and high-performance data storage solutions, ensuring optimal data lake management.
• Manage the Azure Dedicated SQL Pool to optimize query performance and scalability.
• Automate data workflows and processes using Logic Apps and ensure seamless integration between different Azure services.
• Implement and maintain pipeline automation, including setting up email notifications for monitoring pipeline health and issues.
• Ensure secure and compliant data handling by implementing audit logging for all data transformations and access controls.
• Oversee and maintain data governance strategies, ensuring adherence to organizational policies and regulatory compliance.
• Implement and enforce data integrity checks to ensure the accuracy, consistency, and reliability of data across pipelines.
• Develop and maintain data lineage tracking to provide visibility into the lifecycle of data across various systems and processes.
• Secure sensitive data and manage secrets within the data pipeline using Azure Key Vault, ensuring encryption and best practices for data security.
• Collaborate with data scientists and integrate ETL pipelines with Machine Learning models to support predictive analytics and advanced data science use cases.
• Troubleshoot and resolve complex data pipeline issues, ensuring data integrity, quality, and reliability.
• Monitor and optimize the performance of ETL processes, ensuring efficiency and cost-effectiveness in the Azure ecosystem.
• Act as the primary technical point of contact for the ETL team, assisting with technical challenges and ensuring that best practices are followed.