Overview:
Highlight is seeking an Azure Databricks Engineer to remotely support a cloud development program supporting our federal customer.Responsibilities:
• Design, build, and maintain scalable ETL/ELT pipelines using Databricks workflows, Delta Lake, and Apache Spark to process large volumes of structured and unstructured data
• Architect and optimize data lakehouse solutions, implement data modeling best practices, and ensure efficient data storage and retrieval using Delta Lake format and partitioning strategies
• Configure and manage Databricks workspaces, clusters, and compute resources while implementing security controls, access management, and cost optimization strategies
• Develop streaming data solutions using Structured Streaming and implement batch processing jobs for data transformation, aggregation, and integration across multiple data sources
• Collaborate with data scientists to deploy machine learning models, implement MLflow for model lifecycle management, and create automated ML pipelines for training and inference
• Monitor data pipeline performance, troubleshoot issues, implement data quality checks, and optimize Spark jobs for improved efficiency and reduced processing costsQualifications:
• BA/BS/MS in Computer Science, Engineering, Data Science, or equivalent experience.
• 5+ years experience as a Data Engineer, with 3+ years specifically in Azure Databricks and related Azure data services.
• Expertise in Apache Spark, Delta Lake, Python, SQL, and Scala.
• Hands-on experience designing and optimizing large-scale data pipelines.
• Experience with Azure Data Factory, Azure Data Lake Storage, Azure Synapse Analytics, and other Azure services. Strong understanding of ETL/ELT processes, data modeling, and big data technologies.
• Microsoft Certified: Azure Data Engineer Associate (DP-203) or Databricks Certified Data Engineer or equivalent certification or experience.
• Experience with Agile methodology and implementing data governance and security.About Highlight:
For over ten years, Highlight has provided Development and Modernization, Secure IT, and Mission Solution services to our federal government customers. Our team knows the technology; we understand how our customers and their stakeholders work; and we know how to implement industry best practices to deliver high-quality, end-to-end solutions that minimize risk and maximize results.
Since our inception, Highlight has had an employee-first mindset. Our mission is to provide employees with rewarding and impactful career opportunities. In 2021, Highlight’s founder, Rebecca Andino, implemented an Employee Stock Ownership Plan to embody and expand our culture of transparency, teamwork and rewarding the work of our employees. By becoming an ESOP, our employee-owners share in the success of the company through their ownership stake. To learn more about ESOPS, check out: www.esopinfo.org.
We’re an Equal Opportunity Employer (EOE) that empowers our people to fearlessly drive change - regardless of their race, color, religion, sex, gender identity, sexual orientation, national origin, disability, veteran status, or other characteristics. Our team is dedicated to foster diversity within our teams to promote creativity, innovation, and teamwork to deliver the best solutions for our customers.
To receive compensation and benefits information for this role, contact us or email us
[email protected] include the Req ID (this is at the top of the posting under the position title) in the subject line of the email.-:
#nwhp