Spark Developer

Introduction
At IBM work is more than a job – it’s a calling: To build. To design. To code. To consult. To think along with clients and sell. To make markets. To invent. To collaborate. Not just to do something better but to attempt things you’ve never thought possible. Are you ready to lead in this new era of technology and solve some of the world’s most challenging problems? If so lets talk.

Your Role and Responsibilities
We are seeking an experienced Senior Spark / AWS Engineer to join our team.
As a key member of our technical team you will be responsible for designing developing and deploying scalable efficient and reliable Spark-based applications on AWS.
If you have hands-on experience working with AWS EMR Airflow Terraform and GitLab and are passionate about Spark development refactoring performance tuning and AWS engineering we’d love to hear from you!

Responsibilities:
Spark Development
* Design and develop scalable efficient and reliable Spark-based applications
* Collaborate with cross-functional teams to understand requirements and deliver high-quality solutions
Spark Refactoring
* Analyze and optimize existing Spark code for performance scalability and maintainability
* Refactor code to improve readability efficiency and reliability
* Collaborate with the team to ensure smooth code deployments
Spark Performance Tuning
* Identify performance bottlenecks in Spark applications
* Analyze and optimize Spark configurations such as memory CPU and network settings
* Implement optimizations to improve application performance and scalability
AWS Engineering
* Collaborate with cross-functional teams to ensure seamless integration with AWS services
* Ensure AWS security monitoring and logging best practices

Required Technical and Professional Expertise
*Strong understanding of Spark: APIs configurations performance tuning and best practices
*Hands-on experience working with AWS: EC2 S3 EMR Glue Lambda API Gateway etc.
* EMR experience: creating managing and optimizing EMR clusters
* Airflow experience: creating managing and optimizing Airflow workflows
* Terraform experience: designing and deploying infrastructure as code
* GitLab experience: using GitLab for version control CI/CD and collaboration
* Scala knowledge: preferred but not required
* Excellent problem-solving skills: ability to troubleshoot complex technical issues
* Strong communication skills: ability to collaborate with cross-functional teams and communicate technical ideas effectively

Preferred Technical and Professional Expertise

• Relevant applicable certifications