As a Data Engineer you will play a key role in designing building and optimizing data pipelines and processing solutions that enable data-driven decision-making across the organization. Leveraging expertise in SQL Spark and Python alongside AWS data integration and processing technologies (Glue Athena Redshift) you will ensure that our data infrastructure is scalable secure and high performing. You will work closely with data analysts data scientists and business stakeholders to deliver reliable well-tested and high-quality data solutions. You will contribute to modernizing data architectures implementing performance optimizations and supporting the adoption of lakehouse design principles.
In this role you'll work in one of our IBM Consulting Client Innovation Centers (Delivery Centers) where we deliver deep technical and industry expertise to a wide range of public and private sector clients around the world. Our delivery centers offer our clients locally based skills and technical expertise to drive innovation and adoption of new technology.
In this role you'll work in one of our IBM Consulting Client Innovation Centers (Delivery Centers) where we deliver deep technical and industry expertise to a wide range of public and private sector clients around the world. Our delivery centers offer our clients locally based skills and technical expertise to drive innovation and adoption of new technology.
What will you do?
- Design develop and maintain scalable data pipelines for ingestion transformation and loading.
- Implement AWS-based data integration and processing solutions using tools such as Glue Athena and Redshift.
- Write clean efficient and maintainable code using SQL Spark and Python.
- Conduct functional and automated testing of data pipelines to ensure accuracy reliability and performance.
- Perform performance tuning optimization and debugging of data workflows.
- Collaborate with cross-functional teams to understand data requirements and translate them into technical solutions.
- Contribute to the design and development of lakehouse architectures.
- Ensure best practices for data governance quality and security are followed.
- Maintain technical documentation including data flow diagrams pipeline specifications and troubleshooting guides.
Experience:
- Minimum 5+ years of experience in data engineering or related technical roles.
- Proven experience in developing and maintaining large-scale data pipelines.
- Hands-on experience with AWS data services (Glue Athena Redshift).
Technical Skills:
- Strong knowledge of SQL Spark and Python.
- Proficiency in data analysis and transformation workflows.
- Experience with functional and automated testing of data pipelines.
- Understanding of performance tuning and debugging for data processes.
- Some knowledge of lakehouse architecture and design principles.
- Familiarity with cloud security best practices and compliance requirements.
Soft Skills:
- Strong problem-solving and analytical abilities.
- Excellent communication skills for working with technical and non-technical stakeholders.
- Ability to work collaboratively in cross-functional teams.
- Attention to detail and commitment to delivering high-quality solutions.
- Strong organizational skills with the ability to manage multiple priorities.