Senior Software Engineer - DB Developer

Databricks Data Engineer/Architect

Title: Databricks Data Engineer / Architect

Location: Capgemini Hubs (NY NYC, Chicago IL, Columbia SC, Atlanta GA, Houston TX, Nashville TN)

Job Description

Capgemini is seeking a Databricks Data Engineer, who will design, develop, and manage the data infrastructure on the Databricks platform within the Azure cloud environment. This includes: configuring the data lake (ADLSGen2), creating and optimizing data pipelines, and closely monitoring pipelines to ensure data quality and scalability.

The Data Engineer will integrate data from different sources, conduct data transformations, configure security data sharing, and ensure data cleanliness. Additionally, the Data Engineer will seek understanding of the data requirements and provide appropriate solutions. By completing these tasks the Data Engineer will contribute significantly to customers digital transformation initiatives and facilitate data-driven decision-making while advancing AI/ML journey.

Success in this role comes from effective collaboration with various internal and client teams - including product owners and developers.

Responsibilities

• Design, develop, and maintain data processing workflows and analytics solutions using Azure Databricks
• Use business requirements to drive the design of data solutions/applications and technical architecture
• Create technical, functional, and operational documentation for data pipelines and applications
• Develop and maintain ETL (Extract, Transform, Load) pipelines using Databricks to process and transform large datasets
• Collaborate with data engineers and data scientists to design and implement scalable and efficient data processing workflows
• Build and optimize Apache Spark jobs and clusters on the Databricks platform
• Develop and maintain data ingestion processes to acquire data from various sources and systems
• Implement data quality checks and validation procedures to ensure accuracy and integrity of data
• Perform data analysis and exploratory data mining to derive insights from complex datasets
• Design and implement machine learning workflows using Databricks for predictive analytics and model training
• Coordinate and participate in structured peer reviews/walkthroughs/code reviews
• Work effectively in an Agile Scrum environment (JIRA/Azure DevOps)
• Stay updated with the latest advancements in big data technologies and contribute to the improvement of existing systems and processes'

Required Skills

• B.S. in Computer Science/Engineering or relevant field
• 8+ years of experience in the IT industry
• 3+ years of hands-on experience in data engineering/ETL using Databricks Notebook programming on Azure or any cloud infrastructure and functions
• Prove Databricks development experience with significant Python, PySpark, Spark SQL, Pandas, NumPy in Azure environment
• Hands on experience of building data pipelines using Databricks and Apache Spark
• Hands on experience designing and delivering solutions using Terraform and Azure DevOps agents
• Creating mount points for ADLS Gen2 storage in DBFS to implement RBAC for end users
• Strong understanding of distributed computing principles and experience with large-scale data processing frameworks
• Experience with CI/CD on Databricks using tools such as AZDO Git, and Databricks CLI
• Experience working with structured and unstructured data
• Strong understanding of Data Management principles (quality, governance, security, privacy, life cycle management, cataloging).
• Unity Catalog experience desirable
• Experience with Delta Lake, Unity Catalog, Delta Sharing, Delta Live Tables (DLT)
• Able to work independently
• Excellent oral and written communication skills
• Experience in the following: Azure: 3 years (Required); Cloud development 5 years (Required); Python: 3 years (Required)
• Nice to have: Azure Synapse, Databricks Lakehouse Architecture, Azure Data Factory (ADF), PowerBI, Predictive Analytics, AI/ML, Medallion architecture; Microsoft Azure Databricks and Azure Data Engineer certifications

About Capgemini

Capgemini is a global leader in partnering with companies to transform and manage their business by harnessing the power of technology. The Group is guided everyday by its purpose of unleashing human energy through technology for an inclusive and sustainable future. It is a responsible and diverse organization of over 360,000 team members in more than 50 countries. With its strong 55-year heritage and deep industry expertise, Capgemini is trusted by its clients to address the entire breadth of their business needs, from strategy and design to operations, fueled by the fast evolving and innovative world of cloud, data, AI, connectivity, software, digital engineering and platforms. The Group reported in 2022 global revenues of €22 billion.

Get The Future You Want | www.capgemini.com

Disclaimer

Capgemini is an Equal Opportunity Employer encouraging diversity in the workplace. All qualified applicants will receive consideration for employment without regard to race, national origin, gender identity/expression, age, religion, disability, sexual orientation, genetics, veteran status, marital status or any other characteristic protected by law.

This is a general description of the Duties, Responsibilities and Qualifications required for this position. Physical, mental, sensory or environmental demands may be referenced in an attempt to communicate the manner in which this position traditionally is performed. Whenever necessary to provide individuals with disabilities an equal employment opportunity, Capgemini will consider reasonable accommodations that might involve varying job requirements and/or changing the way this job is performed, provided that such accommodations do not pose an undue hardship.

Capgemini is committed to providing reasonable accommodations during our recruitment process. If you need assistance or accommodation, please reach out to your recruiting contact.

Click the following link for more information on your rights as an Applicant http://www.capgemini.com/resources/equal-employment-opportunity-is-the-law

Salary Transparency

Capgemini discloses salary range information in compliance with state and local pay transparency obligations. The disclosed range represents the lowest to highest salary we, in good faith, believe we would pay for this role at the time of this posting, although we may ultimately pay more or less than the disclosed range, and the range may be modified in the future. The disclosed range takes into account the wide range of factors that are considered in making compensation decisions including, but not limited to, geographic location, relevant education, qualifications, certifications, experience, skills, seniority, performance, sales or revenue-based metrics, and business or organizational needs. At Capgemini, it is not typical for an individual to be hired at or near the top of the range for their role. The base salary range for the tagged location is $110841 - $145000 / year.

This role may be eligible for other compensation including variable compensation, bonus, or commission. Full time regular employees are eligible for paid time off, medical/dental/vision insurance, 401(k), and any other benefits to eligible employees.

Note: No amount of pay is considered to be wages or compensation until such amount is earned, vested, and determinable. The amount and availability of any bonus, commission, or any other form of compensation that are allocable to a particular employee remains in the Company's sole discretion unless and until paid and may be modified at the Company's sole discretion, consistent with the law.

Ref: 1715129

Posted on: Sep 8, 2023

Experience level: Experienced (non-manager)

Contract Type: Permanent

Location: