Azure Cloud Data Engineer

Cloud Data Engineer

Reports to: Senior Manager IT Last Update: 9/14/2022

Overview:

3+ years of expertise in building scalable distributed data processing solutions with Azure Data Lake Storage and Azure Synapse Data Warehouse, understanding of the functionalities - Identity Management, Security, Data Governance, Devops, Operations on Azure Platform.

Expert in managing Setup Dataiku DSS install, configuration and optimization Manage Python Recipes via Dataiku, docker container Troubleshoot, Python jobs failure issues Optimization of existing ML jobs Optimize Spark for better performance Run R/Python jobs via docker container and optimize it for better performance.

Essential Duties (Approximate % of Time Spent)

List duties and group similar items :

• Work as an azure cloud data engineer, an individual contributor, and a team player

• Analyze, design, and determine coding, programming, and integration activities required based on specific objectives

• Dataiku DSS platform installation, configuration and optimization

• Manage data pipeline jobs failure issues

• Optimization of existing ML jobs Optimize Spark for better performance

• Python jobs via docker container and optimize it for better performance

• Develop processes, techniques, and tools to analyze and monitor platform performance

• Strong working knowledge in Azure Data Warehouse and Data lakes

• Experience in Azure cloud eco system

• Experience in managing databases and objects

• Experience Azure Data Lake Storage with Azure Synapse and Azure data factory

• Develop Azure Data Factory/Synapse pipelines and experience with Azure SQL Pools, Apache Spark

• Identify, troubleshoot and resolve issues related to slow / failed jobs

• Support ingestion pipelines from data engineering standpoint

• Experience with SQL (Queries, Functions, Stored Procedures) and Python languages

• Participation in the design of data models for reports

• Experience with Azure Functions and API integration management

Non-Essential Duties: Performs other duties as assigned.

Negotiate and influence changes outside of the team that continuously shape and improve the Data strategy.

Participate or run the critical engagements, consistently delivering quality services. Drive high-quality work products within expected timeframes and on budget.

Collaborate with advance analytics team.

All employees shall be in compliance with the Archrock Health Safety and Environmental Policy and local, statutory and/or regulatory requirements at all times.

Skills and Abilities:

Ability to:

- Dataiku DSS admin tasks

- Dataiku platform installation and configuration

- Docker containerization and optimization

- Azure Synapse/Data Factory

- Azure Synapse/Data Factory

- Azure Data Lake Storage /ADLS Gen2

- Oracle PL/Sql, SQL (Queries, Functions, Stored Procedures)

- Databricks

Knowledge:

- Powershell, Python, Pyspark

- Azure DevOps

Minimum Education and Certification:

- Bachelor's degree

Minimum Experience:

- 5+ years of IT and Data Warehouse experience

Leadership:

- Customer-focused professional who is motivated to drive the creation of great data engineering platforms

- Contribute to the continued evolution of Corporate Analytics Platform

- Responsible for managing a growing cloud-based data ecosystem

Problem Solving and Innovation:

- Ability to define and create complex data models to build insights, predictions, and innovations from data.

- Expected to identify opportunities and to provide ideas, methods and innovations to enhance teamwork, efficiency and quality.

This job description is a summary of essential job functions. It is not intended as an employment contract, nor is it intended to describe all duties someone in this position may perform. All employees are expected to perform tasks, as assigned by supervisor, regardless of job title or routine job duties.