Data Engineer (Spark ,Scala & Elastic Search )
Job Description
- Design, develop, and implement Spark Scala applications and data processing pipelines to process large volumes of structured and unstructured data
- Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data
- Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch
- Collaborate with data engineers, data scientists, and other stakeholders to understand requirements and translate them into technical specifications and solutions
- Implement data transformations, aggregations, and computations using Spark RDDs, DataFrames, and Datasets, and integrate them with Elasticsearch
- Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards
- Troubleshoot and resolve issues related to data processing, performance, and data quality in the Spark-Elasticsearch integration
- 5+ Years as Data Engineer
- 5 Years experience in Spark, Scala, Elastic Search
- Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards
- Monitor and analyze job performance metrics, identify bottlenecks, and propose optimizations in both Spark and Elasticsearch components
- Stay updated with emerging trends and advancements in the big data technologies space to ensure continuous improvement and innovation
- Mandatory Skills : Spark, Scala & Elastic Search
Date Posted
09/16/2024
Views
2
Similar Jobs
Senior Software Engineer (Scala/Java) - HERE Technologies
Views in the last 30 days - 0
HERE Technologies is seeking an experienced backend engineer with strong Java or Scala skills to join the Map Processing Pipelines team The role invol...
View DetailsSoftware Architecture Engineering and Cloud Computing Engineer - The Aerospace Corporation
Views in the last 30 days - 0
The Aerospace Corporation is seeking a Senior Project Engineer with expertise in software architecture engineering and cloud computing The role involv...
View DetailsSenior Data Analyst - Customer Experience - WISE
Views in the last 30 days - 0
Wise is a global technology company aiming to revolutionize international money transfers by offering minimal fees maximum ease and full speed They ar...
View DetailsLead Data Analyst - Mitigation - WISE
Views in the last 30 days - 0
Wise is a global technology company seeking an Operations Analyst with 4 years of experience in analytics particularly in operational team analytics T...
View DetailsLead Technical Support Engineer - HERE Technologies
Views in the last 30 days - 0
This role Senior Technical Support Engineer at HERE Technologies involves supporting a diverse portfolio of products and services acting as a technica...
View DetailsPrincipal / Lead Software Engineer- RUST (Algorithmic and Mathematics) - m/w/d - HERE Technologies
Views in the last 30 days - 0
HERE Technologies is seeking a Principal Software Engineer to lead the development of extended services for their VRP solver Tour Planning The role in...
View Details