Databricks Data Engineer

Company Description

Blend360 is an acclaimed, forward-thinking Data, Digital Marketing, & AI Solutions Company, dedicated to fueling remarkable outcomes for our Fortune 500 clients. Our trajectory is one of continuous expansion, emerging at the crossroads of cutting-edge analytics, data proficiency, technology, and digital marketing excellence.

Job Description

We are seeking a Databricks Data Engineer to support our client, a community-based health system serving 1.3 million people across Ohio and Indiana, to design, build, and optimize data pipelines and workflows in support of our healthcare analytics initiatives. This role will be integral in transforming raw data into actionable insights that support clinical, financial, and operational improvements. The ideal candidate will have a strong background in data engineering with specific expertise in Databricks and cloud platforms.

The Details:

• Location: Fully remote within U.S. (Eastern or Central Time preferred)
• Duration: Approximately a 6-month consulting role through Blend360, with the possibility of extension
• Benefits: We do offer benefits to our full-time W2 consultants, including Health, Vision, Dental, 401K plan, Life Insurance, Pretax Commuter Benefits, and an incredibly supportive team cheering you on!

What you’ll do:

• Data Pipeline Development: Design, build, and manage scalable data pipelines and ETL processes in Databricks, utilizing PySpark, SQL, and other tools to transform data into usable formats for analysis and reporting.
• Data Integration: Integrate data from various clinical, financial, and operational systems, ensuring data quality, accuracy, and availability.
• Data Lake and Warehouse Management: Develop and maintain the data lake and data warehouse environments, ensuring they align with best practices for storage, access, and scalability.
• Data Quality and Governance: Collaborate with data governance and compliance teams to ensure data privacy, security, and accuracy, adhering to healthcare regulations such as HIPAA.
• Collaboration and Support: Work closely with data scientists, analysts, and business stakeholders to understand data needs and develop solutions that meet clinical and business objectives.
• Performance Optimization: Identify performance bottlenecks and optimize Databricks workflows for efficiency and cost-effectiveness, leveraging cluster management and Spark optimization techniques.
• Documentation and Training: Create and maintain clear, comprehensive documentation for data processes and provide training or support to other data team members as needed. Continuously provide training and mentorship to team members on Databricks best practices.

Qualifications

• Strong experience in SQL, PySpark, CI/CD, CLI, Azure DevOps, Unity Catalog, various cluster settings, performance optimization, and end-to-end pipeline setup.
• Experience with parsing and loading semi-structured data (JSON).
• Experience with parsing and loading unstructured data (clinical notes, DICOM images - radiology reports, MRI scans, etc.).
• Ability to set up streaming data in the Azure environment (Azure Event Hub).

Preferred Qualifications:

• Prior healthcare experience.
• Familiarity with Medical Language Models from John Snow Labs.
• Knowledge of healthcare NLPs and Spark NLPs.
• Knowledge of HL7 Data format.

Additional Information

This role is not eligible for immigration sponsorship.

The starting pay range for this role is $70.00 - $75.00 per hour. Actual compensation within the range will be dependent on several factors including but not limited to relevant experience, skills, certifications, training, and location. It is not typical for an individual to be hired at or near the top of the range and determining factors for compensation are considered for each individual circumstance. BLEND360 also offers a competitive benefits program to meet the health and financial well-being of our team and their families. You can look forward to a range of benefits including medical, dental, vision, 401K, PTO, commuter benefits, spending accounts, life insurance, disability coverage, and EAPs.

A diverse workforce is a strong workforce
To deliver growth at BLEND360 and for our clients, we believe as a Talent Solutions Company, we have a responsibility and unique opportunity to positively impact the workforce. Diversity has played a critical role in our history, our growth, and continues to have a profound impact on our success. We are determined to have equality in the workplace, within our team and as an extension of our clients’ team.

BLEND360 is an equal opportunity employer.