Principal Data Engineer

Job Description

Main Responsibilities:

• Design and develop cloud native applications using Big Data stack, handling end to end data flow and processing
• Thought-leader in data architectures, ETL/ELT frameworks, and highly scalable data model design
• Thought leader in defining the roadmap to tune performance of complex big data applications.
• Experience in full product lifecycle from feature feasibility study/ defining solution to production and maintenance, taking ownership of complete feature delivery with quality 
• Mentor and advise the scrum teams to meet product expectations
• Work with distributed teams, Dev Leads, Architects/Experts, Managers, and counterparts in different locations, as well as with external teams. 

About the ideal candidate

• 12+ years of industry experience with strong technical experience of which 10+ years in Big Data technologies, Cloud, and ETL/ELT frameworks etc.
• Deep expertise working with Spark - Scala, Python, on platforms like Databricks, and deep expertise in cloud platforms preferably Azur
• Must have knowledge of CI CD deployments
• Experience in Micro-service architecture is a must-have.
• Experience with logging management using Splunk/ELK including monitoring best practices
• Should have cracked problems on the scale, and performance in Spark using cloud based platforms
• Expertise in best practices, design patterns, and anti-patterns.
• Competency in professional engineering best practices and processes relating to software development lifecycle (scrum, architecture/design/coding standards, design/code reviews, unit testing, build processes, testing, operations, etc)
• Excellent collaboration as well as written and oral communication skills.
• Cloud/SaaS (particularly Azure) experience will be an added advantage but not mandatory
• Experience working with Openshift and Kubernetes, Helm, and Jenkins is added advantage 

Keywords: Spark, Scala, Python, Big Data Engineer, Azure Databricks, Event Driven, Microservices, Java, API, Cloud, Azure, Event hub