Consultant - Business Consulting (Tech Transformation))

Job Overview:

We are looking for a skilled and experienced Data Engineer to join our data team. The Data Engineer will be responsible for designing, building, and maintaining scalable and efficient data pipelines and architectures that support various data-driven initiatives. The ideal candidate will have strong technical expertise in data processing, data storage, and data integration, and will work closely with data scientists, analysts, and other stakeholders to ensure the integrity, availability, and reliability of the data.

Key Responsibilities:

Design and Build Data Pipelines:

• Develop, construct, test, and maintain scalable data pipelines for ingestion, processing, and storage of large data sets from multiple sources.

• Optimise data architectures for both structured and unstructured data to support batch and real-time data processing.

Data Integration and Management:

• Integrate data from different source systems (APIs, databases, cloud services) into the data lake or data warehouse.

• Implement and manage ETL (Extract, Transform, Load) processes for efficient data movement, cleaning, and transformation.

• Ensure data quality, consistency, and integrity through proper validation and testing.

Database and Data Warehousing Solutions:

• Design and implement database solutions that support data storage, transformation, and querying.

• Build and maintain data warehouse solutions to facilitate business intelligence and analytics needs.

Cloud Infrastructure:

• Work with cloud platforms such as AWS, Azure, or Google Cloud to implement data storage, processing, and streaming architectures.

• Set up and manage cloud-based data infrastructure, ensuring scalability, security, and performance optimisation.

Collaboration and Stakeholder Management:

• Collaborate with data scientists, analysts, software engineers, and business teams to understand data requirements and deliver solutions that meet their needs.

• Document data flows, schemas, and pipeline designs to ensure transparency and understanding across the team.

Data Governance and Security:

• Implement data governance standards, data security, and compliance measures to protect sensitive data.

• Ensure compliance with data privacy regulations such as GDPR, CCPA, etc.

• Define and maintain data retention policies and ensure adherence to best practices for data management.

Performance Monitoring and Optimisation:

• Monitor data pipeline performance, identify bottlenecks, and troubleshoot issues as they arise.

• Continuously improve data processes to ensure they are efficient and scalable for growing data sets.

Automation and Tools Development:

• Automate repetitive tasks and processes to streamline data engineering workflows.

• Develop tools and frameworks for data access, management, and analysis, enabling the business to easily work with data.

Skills and Qualifications:

Essential Skills:

• Bachelor's degree in Computer Science, Engineering, or a related field.

• Proven experience as a Data Engineer, Software Engineer, or similar role in large scale data platform implementation (project team size of >= 10 FTEs and/or project duration 6 months or more).

• Strong experience in SQL and relational database systems (e.g., MySQL, PostgreSQL, Oracle).

• Proficiency in programming languages such as Python, Java, or Scala for data manipulation and ETL tasks.

• Minimum 5 years of hands-on experience with ETL tools like Apache Nifi, Talend, Informatica, or equivalent.

• Familiarity with big data technologies such as Hadoop, Spark, Kafka, and HBase.

• Minimum 3 years of hands-on experience with cloud-based data services (e.g., AWS S3, Azure Data Lake, Google BigQuery, Redshift).

• Knowledge of data modelling, database design, and data architecture best practices.

• Experience with version control (e.g., Git) and agile development practices.

Preferred Skills:

• Experience with NoSQL databases like MongoDB, Cassandra, or DynamoDB.

• Experience with Databricks and Dataiku.

• Familiarity with machine learning pipelines and integration with data systems.

• Knowledge of CI/CD pipelines and automation tools like Jenkins, Docker, or Kubernetes.

• Exposure to real-time data processing tools such as Apache Flink or Storm.

• Understanding of data privacy regulations (GDPR, CCPA, HIPAA) and how they impact data storage and processing.

Personal Attributes:

• Strong problem-solving skills and ability to work independently or as part of a team.

• Excellent communication skills, with the ability to explain technical concepts to non-technical stakeholders.

• Attention to detail and a focus on data accuracy and quality.

• Proactive mindset with a passion for innovation and staying up-to-date with emerging technologies.

Given that this is just a short snapshot of the role we encourage you to apply even if you don't meet all the requirements listed above. We are looking for individuals who strive to make an impact and are eager to learn. If this sounds like you and you feel you have the skills and experienc