Engineering Manager - PySpark

At Databricks, we are passionate about enabling data teams to solve the world's toughest problems — from making the next mode of transportation a reality to accelerating the development of medical breakthroughs. We do this by building and running the world's best data and AI infrastructure platform so our customers can use deep data insights to improve their business.

As PySpark continues to gain traction within the open source Spark community and the Databricks Lakehouse platform, we are looking for an Engineering Manager to spearhead PySpark development initiatives, encompassing both open source and Databricks-specific components. Your primary mission will be to solidify Apache Spark's position as the go-to data processing and analysis framework within the Python community. Your key responsibilities include:

• Leading and growing a talented engineering team developing PySpark features and capabilities.
• Promoting the adoption of Spark and the Databricks Lakehouse platform among Python users.
• Overseeing sustained recruitment of top-tier talent, fostering a well-organized and synergistic team structure.
• Implementing robust processes to efficiently execute on product vision and strategy in alignment with organizational goals and priorities.
• Ensure strong alignment by collaborating closely with leaders across the company, within engineering, as well as with product management, field engineering, recruiting, and HR.

By taking on this pivotal role, you will play an instrumental part in nurturing a thriving Python user base and driving the success of Spark and the Databricks Lakehouse platform.

The impact you will have:

• Lead product development for one of the fastest growing libraries in the open source Spark project, as well as the Databricks Lakehouse platform. 
• Make company wide impact by driving Python adoption across the Databricks product portfolio.
• Develop and deepen understanding and expertise in PySpark and the PyData ecosystem, a well adopted yet still hyper-growing product.
• Define, shape, and drive the future of Spark and Databricks Lakehouse platform for Python users.
• Grow a world class team of software engineers working on our compute fabric. Increase headcount by 5+ engineers in next 18 months, with continued growth beyond that according to product objectives. Hire top-notch staff+ level talent.
• Ensure consistent delivery against milestones and strong alignment with the field working "two-in-a-box" with product leadership.
• Evolve organizational structure to align with long term initiatives, build strong "5 ingredient" teams with good comms architecture.
• Manage technical debt, including long term technical architecture decisions and balance product roadmap.

What we look for:

• 5+ years experience with distributed systems, databases, compilers, performance optimization, or big data systems (Spark, Hadoop).
• A passion for database systems, storage systems, distributed systems, language design, or performance optimization
• Can ensure the team builds high quality and reliable infrastructure services. Have experience being responsible for testing, quality, and SLAs for a product. Have built and lead teams in a complex technical domain, such as distributed data systems or database internals
• Ability to attract, hire, and coach engineers who meet the Databricks hiring standards. Can uplevel existing team via hiring top-notch senior talent, growing leaders and helping struggling members. Can gain trust of the team and provide career development and guidance. Experience managing distributed teams preferred.
• Comfort working cross functionality with product management and directly with customers; ability to deeply understand product and customer personas.