Data Engineer (Spark ,Scala & Elastic Search )
Company
Unison Consulting
Location
Other US Location
Type
Full Time
Job Description
- Design, develop, and implement Spark Scala applications and data processing pipelines to process large volumes of structured and unstructured data
- Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data
- Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch
- Collaborate with data engineers, data scientists, and other stakeholders to understand requirements and translate them into technical specifications and solutions
- Implement data transformations, aggregations, and computations using Spark RDDs, DataFrames, and Datasets, and integrate them with Elasticsearch
- Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards
- Troubleshoot and resolve issues related to data processing, performance, and data quality in the Spark-Elasticsearch integration
- 5+ Years as Data Engineer
- 5 Years experience in Spark, Scala, Elastic Search
- Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards
- Monitor and analyze job performance metrics, identify bottlenecks, and propose optimizations in both Spark and Elasticsearch components
- Stay updated with emerging trends and advancements in the big data technologies space to ensure continuous improvement and innovation
- Mandatory Skills : Spark, Scala & Elastic Search
Date Posted
09/16/2024
Views
2
Similar Jobs
Senior Engineering Manager, Micros Foundations - Atlassian
Views in the last 30 days - 0
Atlassian is seeking a Senior Engineering Manager to lead a team of Backend Software Engineers The role involves guiding technical decisions prioritiz...
View DetailsSenior Frontend Engineer - Simply Business
Views in the last 30 days - 0
Simply Business is seeking a Senior Frontend Engineer to join their Front End Tooling team The role involves developing products using best practices ...
View DetailsDevelopment Underwriter - Simply Business
Views in the last 30 days - 0
Simply Business is seeking a Development Underwriter with an Underwriting background to support their new MGA brand Nupro which aims to disrupt the sm...
View DetailsE2E Solution Architect - Ahold Delhaize USA
Views in the last 30 days - 0
Ahold Delhaize USA is seeking a Solution Architect with extensive experience in IT architecture BigData Analytics and various software designs and dev...
View DetailsE2E Solution Architect - Ahold Delhaize USA
Views in the last 30 days - 0
Ahold Delhaize USA is seeking a Solution Architect with extensive experience in IT architecture BigData Analytics and various software designs and dev...
View DetailsE2E Solution Architect - Ahold Delhaize USA
Views in the last 30 days - 0
Ahold Delhaize USA a division of a global food retailer is seeking a Solution Architect for its US operations The role involves translating business r...
View Details