At IBM Global Sales we bring together innovation collaboration and expertise to help clients solve their most complex business challenges. Working across industries and geographies you’ll partner with colleagues clients and partners to co-create solutions that drive digital transformation and lasting impact.Success in Global Sales is built on curiosity empathy and collaboration. You’ll connect technical understanding with strong people skills building trusted relationships and shaping solutions that improve business and society. With world-class onboarding continuous learning and a supportive culture IBM offers the tools and opportunities to grow your career. Join us and be part of a global team that’s passionate about driving innovation and making a difference.
We are seeking a highly skilled and experienced Spark Cluster Admin to join our dynamic team. This role is critical for building optimizing and supporting our cutting-edge data platform leveraging Apache Spark Apache Iceberg and a robust DevOps approach within an OpenShift environment. The ideal candidate will be adept at both developing high-performance data solutions and ensuring their stability and reliability in a production setting.
Key Responsibilities
- Design develop and optimize scalable and resilient data processing applications using Apache Spark (batch streaming and real-time).
- Implement and manage data pipelines ensuring data quality consistency and performance.
- Perform Spark job performance tuning and optimization to handle large-scale datasets efficiently.
- Manage and automate the deployment of Spark applications within OpenShift clusters utilizing Docker and Kubernetes.
- Establish and maintain CI/CD pipelines for automated testing deployment and release management of Spark workloads.
- Provide comprehensive production support for critical Spark jobs including proactive monitoring troubleshooting debugging and participation in on-call rotations.
- Work extensively with Apache Iceberg table format leveraging its capabilities for schema evolution time travel hidden partitioning and ACID transactions.
- Collaborate closely with data scientists other data engineers and operations teams to deliver robust and integrated solutions.
- Develop and maintain documentation for data pipelines job configurations and operational procedures.
- 9+ years of hands-on experience in Spark cluster administration roles.
- Expert-level proficiency in Apache Spark (Spark Core Spark SQL Spark Streaming).
- Strong programming skills in Scala Python (PySpark) or Java .
- Significant experience with OpenShift including deploying managing and automating containerized applications within the platform.
- Solid understanding of Docker and Kubernetes for containerization and orchestration.
- Proven experience implementing and maintaining CI/CD pipelines using tools like Jenkins GitLab CI or similar.
- Demonstrable experience with Apache Iceberg including practical application of its features like schema evolution time travel queries and ACID compliance.
- Strong background in production support for data applications including monitoring troubleshooting and incident resolution.
- Understanding and practical application of DevOps principles (Infrastructure as Code automation continuous monitoring).
- Strong SQL skills and experience working with various data sources.
- Excellent analytical and problem-solving abilities for diagnosing and resolving complex issues in distributed environments.
- Familiarity with distributed systems concepts and architectures.
- Experience with other data lake table formats (e.g. Delta Lake Apache Hudi).
- Familiarity with cloud platforms (AWS Azure GCP) beyond OpenShift.
- Experience with messaging queues or streaming platforms like Apache Kafka.
- Contributions to open-source data projects.