Senior Software Engineer, Gluten Engine

Introduction
At IBM work is more than a job – it’s a calling: To build. To design. To code. To consult. To think along with clients and sell. To make markets. To invent. To collaborate. Not just to do something better but to attempt things you’ve never thought possible. Are you ready to lead in this new era of technology and solve some of the world’s most challenging problems? If so lets talk.

Your Role and Responsibilities
We are seeking highly skilled Senior Engineers to join our team working on the Apache Gluten project a cutting-edge initiative aimed at enhancing the performance of JVM-based SQL engines like Apache Spark by offloading execution to native engines. This role requires deep expertise in distributed systems native code development and familiarity with big data frameworks.

Responsibilities:

• Architecture & Development: Design develop and optimize the core components of the Apache Gluten project focusing on the transformation of Spark SQL’s physical plans to Substrait plans and their execution using native libraries like Velox and ClickHouse.
• Performance Optimization: Lead efforts to offload performance-critical data processing tasks from JVM-based SQL engines to native libraries improving overall query execution efficiency.
• Integration & Support: Work on integrating Gluten with various backends and frameworks such as Apache Arrow for data format standardization and extend support for additional backends and file formats.
• Collaboration: Collaborate with other contributors including those from leading tech companies like Intel and Kyligence to enhance the project’s functionality and ensure robust performance across various environments.
• Documentation & Community Engagement: Contribute to the project’s documentation assist in community engagement and mentor junior engineers.

Required Technical and Professional Expertise

• Technical Expertise: Extensive experience in C++ Java/Scala and native code optimization. Knowledge of Spark SQL internals and experience with native execution engines like Velox is highly desirable.
• Distributed Systems: Strong background in distributed systems and experience with large-scale data processing frameworks.
• Open Source Contribution: Proven track record of contributing to open-source projects preferably within the Apache ecosystem.
• Problem Solving: Strong analytical skills with the ability to troubleshoot complex performance issues and provide innovative solutions.

Preferred Technical and Professional Expertise

• Experience with the Apache Spark ecosystem and its plugin mechanisms.
• Familiarity with Apache Arrow and Substrait specifications.
• Knowledge of various data formats like Parquet ORC and columnar data processing techniques.
• Understanding of JNI (Java Native Interface) and its application in cross-language development.