Quantexa developer
Job Description
- Implement data transformation, aggregation, and enrichment processes to support various data analytics and machine learning initiatives
- Collaborate with cross-functional teams to understand data requirements and translate them into effective data engineering solutions
- Design, develop, and implement Spark Scala applications and data processing pipelines to process large volumes of structured and unstructured data
- Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data
- Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch
- Implement data transformations, aggregations, and computations using Spark RDDs, DataFrames, and Datasets, and integrate them with Elasticsearch
- Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards
- Troubleshoot and resolve issues related to data processing, performance, and data quality in the Spark-Elasticsearch integration
- Monitor and analyze job performance metrics, identify bottlenecks, and propose optimizations in both Spark and Elasticsearch components
- Ensure data quality and integrity throughout the data processing lifecycle
- Design and deploy data engineering solutions on OpenShift Container Platform (OCP) using containerization and orchestration techniques
- Optimize data engineering workflows for containerized deployment and efficient resource utilization
- Collaborate with DevOps teams to streamline deployment processes, implement CI/CD pipelines, and ensure platform stability
- Implement data governance practices, data lineage, and metadata management to ensure data accuracy, traceability, and compliance
- Monitor and optimize data pipeline performance, troubleshoot issues, and implement necessary enhancements
- Must be Quantexa certified data engineer / data architect and proficient with the tool.
- Proven experience as a Data Engineer, working with Hadoop, Spark, and data processing technologies in large-scale environments
- Proficiency in Scala programming language and familiarity with functional programming concepts
- Experience with Quantexa tool is highly preferred.
- In-depth understanding of Apache Spark architecture, RDDs, DataFrames, and Spark SQL
- Experience with containerization platforms such as OpenShift Container Platform (OCP) and container orchestration using Kubernetes
- Proficiency in programming languages commonly used in data engineering, such as Spark, Python, Scala, or Java
- Knowledge of DevOps practices, CI/CD pipelines, and infrastructure automation tools (e.g., Docker, Jenkins, Ansible, BitBucket)
- Experience with Graphana, Prometheus, Splunk will be an added benefit
Date Posted
10/07/2024
Views
0
Similar Jobs
Software Architecture Engineering and Cloud Computing Engineer - The Aerospace Corporation
Views in the last 30 days - 0
The Aerospace Corporation is seeking a Senior Project Engineer with expertise in software architecture engineering and cloud computing The role involv...
View DetailsSoftware Engineering Manager - Cargill
Views in the last 30 days - 0
The Software Engineering Manager job involves setting goals for a team responsible for software project development and delivery ensuring quality stan...
View DetailsSales Development Representative - UK (Remote) - Dscout
Views in the last 30 days - 0
Dscout is a company that specializes in experience research solutions helping innovative companies like Salesforce Sonos Groupon and Best Buy to build...
View DetailsIntern People Experience - Personio
Views in the last 30 days - 0
Personio is an HR platform that simplifies complex tasks for small and mediumsized organizations With a team of over 1800 employees across Europe and ...
View DetailsSenior Finance Business Partner (d/f/m) - Personio
Views in the last 30 days - 0
Personio an intelligent HR platform is seeking a Senior Manager for FPA to lead financial planning and analysis for key departments The ideal candidat...
View DetailsSenior Lead, Talent Acquisition - Sales (Relocation to Munich) (d/f/m) - Personio
Views in the last 30 days - 0
Personio a leading HR platform is seeking a Senior Lead Talent Acquisition professional to drive growth in the Revenue and Success functions across Eu...
View Details