Senior Data Engineer (Java, Spark)

Who we are? We are Big Data experts, working with international clients, creating and leading innovative projects related to the Big Data environment. We offer tailor-made solutions. It does not matter whether we are talking about building a Data Lake, conducting training in the field of data management, or performing detailed Big Data analysis. We don’t just focus on one technology, instead, we specialize in a whole range of open-source and public cloud tools. Our team brings together over 130 specialists in their fields. We have participated in dozens of conferences, and written countless amounts of code, we are the organizers of Big Data Tech Summit Warsaw, the largest Polish conference related to Big Data topics. We run webinars, share knowledge on blogs, create whitepapers and more. Why? Because we believe that Big Data is an indispensable future of business.
Thanks to that, we always select the most optimal Big Data solutions.

Customer

We are working on the project for a global financial institution with a strong European base, offering services to customers in over 40 countries, widely recognized as being one of the most innovative.

Project

The initial goal of the project was to build an ingestion solution with the use of open-source and closed-source technologies (NiFi, Spark, and HDP/CDP). The solution must be highly scalable, fault-tolerant, and extremely efficient. Further improvements related to event-based ingestion, making the data lake available to consumers, and migration to Cloudera stack are envisaged in the nearest future. 

The candidate is expected to drive the discussions/refinements, based on their knowledge and experience, focusing on building rock-solid solutions and delivering value.

Responsibilities

• Analysis of current code

• Development of high-quality code applying engineering best practices (tests, reviews, CI/CD)

• Organizing and leading the technology decision process, preparing detailed designs and implementation proposals

• At least 5 years of experience in Data and Software Engineer roles
• Java programming experience
• Good data engineering background especially with distributed data processing (like Spark), distributed data datastores (Hive), and tools for building data pipelines (NiFi, Airflow)

Nice to have:

• Understanding of container-based platforms and tooling including Docker, familiarity with Kubernetes and/or OpenShift

• Salary : 150-190 PLN net + VAT/h B2B (depending on knowledge and experience)

• 100% remote work

• Elastic working hours

• Possibility to work from the office located in the heart of Warsaw

• Opportunity to learn and develop with the best Big Data specialists in Poland

• International projects

• Possibility of conducting workshops and training

• Clear career path and certifications

• Co-financing sport card

• Co-financing health care

• All equipment needed for work