Data Engineer I

New Visions works to make the public education system in New York City a place where students from every background can graduate high school and successfully transition into their post-secondary future. The School Systems and Data Analytics Department aims to accomplish this mission by creating a comprehensive data management ecosystem to support school and district users in making decisions that maximize their students’ likelihood of graduating and succeeding beyond high school. We currently support over 900,000 students in nearly 1600 schools, working with teachers and administrators to help students progress towards graduation and beyond into post-secondary success.

The School Systems and Data Analytics department is at the core of supporting New Visions staff and schools in translating data into action. Multiple times per day, we manage the processing of data from multiple internal and external source systems into New Visions databases and into live tools, providing staff with actionable, timely, and accessible information to make data-informed decisions. 

The Data Engineer I plays a crucial role within the unit and the organization, working closely with the Data Management team and Infrastructure team to develop and maintain robust data systems, monitor and troubleshoot core data processing pipelines, and manage the New Visions’ data tools ecosystem to provide the right data quickly to key stakeholders. The Data Engineer I is primarily responsible for building and maintaining the infrastructure used to operate and scale the data platforms that the organization supports. 

You are excited about public service and the prospect of solving problems that are challenging and affect urban schools everywhere.

You are detail oriented and enjoy organizing data in a way that will facilitate the work of team members.

You care about creating a data model that is optimized for performance and quality

You enjoy working with analysts, designers and product managers to determine the best way to grow our data model to accommodate new features and tools.

You love working in teams to solve complex challenges.  You thrive in a fast-paced, highly collaborative environment. 

• Develop, monitor, and improve the NV data model and its pipelines
• Use a combination of languages (Python, SQL, R) and tools to manipulate and transfer data
• Create cross-sectional and longitudinal data sets from raw data files
• Collaborate with software engineers and product managers  to create schema in mongoDB and deliver data that fulfills feature requests
• Create systems for assuring data quality and accuracy
• Create alerts and process-monitoring tools to understand the flow of data
• Manage infrastructure for processing large data sets for use in NV data tools and the NV data portal
• Maintain code repository for automated overnight data processing and incorporate new data streams as needed
• Research, test, and integrate methods to streamline the NV data model
• Support the integration of additional data sources into New Visions data warehouse and data tools
• Support the operationalizing of robust data quality assurance within data infrastructure
• Monitor and evaluate core data processing efforts to identify areas for improvement as well as troubleshoot technical issues
• Collaborate with internal colleagues to support current and forthcoming features for NV tools and the NV data portal
• Communicate best-practices in data engineering to the respective teams 
• Provide metrics of data processing to highlight areas for growth in core data processing efforts

• Proficiency in Python required
• Demonstrated ability in SQL, as well as common practices in schema design and data storage
• Proficiency with ETL development
• Experience with SQL databases (Redshift, Postgres, etc.)
• Experience with noSQL databases (mongoDB, Cassandra, etc.)
• Demonstrated ability in R, and the data.table package
• Exceptional strategic, analytical, and critical thinking skills
• Strong project management and organizational skills.
• Excellent written and oral communication skills.
• Close attention to detail
• Demonstrated ability to prioritize, multi-task, work under pressure and meet deadlines
• Demonstrated persistence and independence in learning technical subject matter, and in solving technical problems.
• Demonstrated ability to identify problems and suggest solutions for discussion
• Demonstrated ability to identify problems and lead improvements to existing codebases and processes
• Knowledge of public education data in New York State

• A minimum of 2 years of experience in data engineering, software engineering, or advanced data analytics
• Expertise in Git and Github - including branching, merging, diffs, and hotfixes
• Expertise in Python, SQL and R
• Proficiency with SQL databases (Redshift, Postgres, etc.)
• Proficiency with noSQL databases (mongoDB, Cassandra, etc.)
• Experience building scalable solutions with AWS Redshift, MongoDB, and PostgreSQL
• Experience using big data technologies (Hadoop, Spark, etc.)
• Experience building data pipeline tools to manage data quality, ensuring production data is always accurate and available for key stakeholders and business processes that depend on it
• Expertise managing data pipelines to support continuing increases in data volume and complexity
• Demonstrated ability to manage Airflow instances and develop best practices for scheduling jobs

• Data/Database Layer 
• AWS (Redshift, S3, RDS), MongoDB, PostgreSQL 
• Code 
• Python, SQL, Linux/Bash, R
• Orchestration Layer
• Apache Airflow, Docker, AWS ECS

Disclaimer:

The statements herein are intended to describe the general nature and level of work being performed by the employee in this position.  These statements are not intended to be construed as an exhaustive list of all responsibilities, duties, and skills required of a person in this position.

Pay Range:

$83,278 minimum to $137,409 maximum

While the majority of new hires’ starting salaries shall be expected to be placed in the first half of the range(s) to allow space to grow in the job, for those who already have significant relevant experience at that level of the job, placement at the higher end of the range(s) is possible.

Additional Information:

NYC-based headquarters. Remote options available, ideally working from locations in Eastern Standard Time. Additional details will be shared if selected for an interview