Big Data Engineer I

Job Description

Big Data Engineer I

Technology Solutions Team

The Big Data Engineer I primary responsibilities are to create, implement, and maintain various automated data solutions within DataDelivers big data ecosystem. The Engineer will work closely with other teammates to design optimum solutions using best practices. The Engineer is responsible to monitor and measure the data ecosystem's performance, troubleshoot as needed, and report on status and metrics.

The Big Data Engineer also collaborates with DataDelivers data scientists and account team to creates ETL, batch, and automated processes to best suite DataDelivers and its client's needs.

They proactively follow and contribute to the teams coding and programming standards and offer suggestions to how better improve them. They have a high standard for the quality of code and documentation they themselves produce.

Qualifications:

• (5 years of relevant experience) or (2 years of relevant experience and an advanced degree in Computer Science/IT or related field)
• Good understanding of distributed computing principles.
• Proficiency with Big Data frameworks such as Hadoop, Spark, MapReduce, HDFS.
• Proven experience ingesting data from multiple data sources such as REST API, SFTP flat files, Streaming data etc.
• Proven experience with Big Data querying tools such as Athena/Presto, Pig, Hive, and Impala.
• Proven experience with NoSQL databases, such as HBase, Cassandra, Redshift, DynamoDB.
• Proven experience with various ETL techniques and frameworks, such as Flume, Glue Jobs, Step Functions.
• Proven experience with Big Data ML toolkits, such as Mahout, SparkML, or H2O
• Proven experience with AWS Lambda and leveraging it in various solutions such as Glue, Step Functions, CloudWatch, S3 Events, etc.
• Strong experience with using Python scripts & libraries.
• Experience desired with Database Warehousing Design Concepts; Dimensional.
• Modeling, Star/Snowflake Schemas, ETL/ELT, Data Marts, Analytic Playgrounds, Reporting techniques.
• Experience working with Agile software development methodologies, namely Scrum.
• Proven experience with team collaboration, release management, system and performance monitoring.
• Ability to work well with people from many different disciplines and varying degrees of technical experience.
• Excellent analytical, problem resolution, organization and time management skills.
• Ability to handle multiple tasks at a time.

Experience and Skills

• Work closely with SMEs, make solution recommendations, and implement agreed upon solutions using best practices.
• Select and integrate any Big Data tools and frameworks required to provide requested capabilities.
• Design and implement ETL and automated processes.
• Monitor performance and advise of any necessary improvements and changes.
• Management of EMR clusters, Glue Jobs, Athena Tables, S3 data lakes; with all included services.
• Provide technical support to members of TS and SA team, as well as project support across client engagements.
• Work with geographically dispersed teams, embracing Agile and DevOps strategies for themselves and others while driving adoption to enable greater technology and business value.
• Stays current with relevant technology in order to maintain and/or improve functionality for authored applications.
• Assume other responsibilities as requested/required.
• Acts as a subject matter expert for systems worked on. Ensures DataDelivers data solutions are using the latest versions and code base.
• Actively listen to and work with end users to gather feedback and input, and make suggestions and solutions based on said feedback.

Job Benefits

• We constantly strive to achieve a strong work/life balance; we are an employee-centric culture, with room for flexibility in work location and hours where possible
• We provide excellent benefits, including 401K, medical, dental, vision, disability, and life insurance, and generous compensated time off policy
• We offer a competitive compensation package with opportunities for performance rewards based on company success