Senior Data Engineer – Data Mesh, New York

We are seeking an experienced Data Engineer with AWS experience to join our newly created Service Consultant Sub-Practice and exclusively work with a mature and impactful long-term client in the financial services industry. This role requires a balance of independent work and teamwork, focusing on advancing the client's agenda. The ideal candidate will possess some understanding of the financial services sector, coupled with a proven track record of independent work and team collaboration. Join us to make a meaningful impact and contribute within our innovative team. 

What you’ll do:

• Profiling of source system data for target state schema/table design
• Delivery of required datasets from the source systems into curated S3 buckets or folders, processed automatically via Databricks following medallion architecture
• Ensure data validations, normalization of datasets as per each business area’s requirements 
• Support testing and fine-tuning performance of workflows in Databricks 
• Manage User Group identification per use case requirements (Creation of Okta groups, roles, permissions) 
• Manage environment deployment process via CI/CD 

Requirements:

• 5+ years’ experience as a Data Engineer
• 5+ years' experience with Python and Java
• Strong knowledge of SQL 
• Hands-on experience and understanding of API integrations 
• Good understanding of the Databricks platform and the data lakehouse / medallion architecture 
• Hands-on experience with Starburst / Trino and/ or AWS Athena 
• Experience with AWS-native technologies like Lambda, Glue and S3 
• Database migrations experience with tools like Flyway, Liquibase. Atlas or equivalent 
• CI / CD experience with Gitlab, GitHub or equivalent 

Nice to haves:

• Collibra & AWS integration 
• Collibra & Databricks integration 
• IaC experience with Terraform or CloudFormation

Lab49 is a specialist strategy, design, and technology consulting company dedicated to the financial services industry. We were founded in 2002 to fill a need in the Financial Technology marketplace for high-end execution and reliable delivery of technology projects. 

We have office hubs in Midtown Manhattan, Washington DC, London, Budapest, New Delhi and Sydney. 

Our Hybrid Work Model 

At Lab49, we embrace a hybrid work model, offering you the flexibility to work from the office or remotely. We expect our Lab49ers to average 2-3 days per week in person and this could be at our midtown office in NYC or at a client site. We believe that both the flexibility and in person connection to each other and our clients build a collaborative culture and the opportunity to accelerate growth and innovation. 

Lab49/ION is committed to maintaining a supportive and inclusive environment for people with diverse backgrounds and experiences. We respect the varied identities, abilities, cultures, and traditions of the individuals who comprise our organization and recognize the value that different backgrounds and points of view bring to our business. 

Lab49/ION adheres to an equal employment opportunity policy that prohibits discriminatory practices or harassment against applicants or employees based on any legally impermissible factor.