Senior Data Engineer II

At Compass, we’re on a mission to help everyone find their place in the world. Since 2012, we’ve been transforming the real estate industry with our end-to-end technology platform, empowering residential real estate agents to deliver outstanding service to their clients. Our culture thrives on interpersonal connectivity, collaborative impact, and bold, innovative solutions.

Data is the foundation of Compass technologies. Our team is responsible for architecting, building, and maintaining a unified, scalable, and cost-effective analytics platform, including a data lake, data warehouse, data pipelines, and operational tools to support data stakeholders across the company.

Want more jobs like this?

Get jobs in Manhattan, NY delivered to your inbox every week.

Get Jobs

Send me The Muse newsletters for the best in career advice and job search tips.

By signing up, you agree to our Terms of Service & Privacy Policy.

As a data engineer, you will be responsible for building, optimizing, and maintaining scalable data pipelines using distributed computing on the cloud. You are a data expert who understands and optimizes data systems from the ground up. You will collaborate with data analysts and scientists to support data initiatives and ensure consistent, optimal data delivery architecture for ongoing projects. You must be self-directed and comfortable supporting the data needs of multiple teams, systems, and products.

Responsibilities:

• Data Infrastructure Development: Design and implement fault-tolerant, scalable cloud data solutions, ensuring optimal performance, cost, and scalability. Automate infrastructure configuration using Infrastructure as Code (IaC).

• Infrastructure Operations: Oversee platform support, troubleshooting, updates, and feature adoption. Ensure platform health and cost optimization.

• Security & Compliance: Implement best practices for data security and encryption. Maintain regulatory compliance (e.g., CCPA).

• Engineering Excellence: Stay updated with emerging technologies in data engineering, identifying opportunities for improvement and operational efficiency.

• Leadership & Mentorship: Guide junior/mid-level engineers through mentorship and code reviews, fostering professional growth.

• Collaboration: Partner with analysts, engineers, and stakeholders to align data infrastructure with business goals. Contribute to planning and decision-making.

Qualifications:

• Bachelor’s/Master’s in Computer Science, IT, or related field.
• 8+ years in cloud data platform development (e.g., Databricks, Spark, Kafka, Airflow).
• Proficient in Python, Java, or C# with 8+ years of programming experience.
• 3+ years using IaC technologies like AWS CloudFormation or Terraform.
• Expertise in data compliance/governance and process automation.
• Strong skills in SQL and cloud platforms (e.g., AWS Glue, GCP, Azure Data Factory).

Desirable Skills:

• SME in cloud data platforms (e.g., Databricks, AWS Lake Formation).
• Proven ability to enhance service performance and productivity.
• Analytical and problem-solving skills with strong stakeholder management.
• Experience with machine learning workflows and integration with LLMs.

Compensation: The base pay range for this position is $168k -$180k annually; however, base pay offered may vary depending on job-related knowledge, skills, and experience. Bonuses and restricted stock units may be provided as part of the compensation package, in addition to a full range of benefits. Base pay is based on market location. Minimum wage for the position will always be met.