Senior Data Engineer

Role description

At Dandi, we are looking for our first Data Engineers to join and grow our existing core team.
As a Data Engineer, you will be working on our core systems, engines and product services using the most sophisticated cutting edge technologies to solve challenging problems that are unique to the DEI world

We have primed ourselves since day one to build one of the most resilient and scalable systems to support what we saw as one of the most interesting challenges in our careers. Our system currently processes TBs/s of data with upwards of 2m writes per second - and as we grow those numbers are ought to grow with us.

We have also built our infrastructure and systems to allow innovation - You have an idea to implement, or new technology to try out - this is the right place for you

Dandi engineering infrastructure is built on top of GCP utilizing cutting edge technologies & softwares such as Bigtable, BigQuery, Apache Spark, AI Platform & Tensorflow, Apache Airflow, K8s to mention a few

What you’ll do:

You’ll be responsible to build and expand our core systems, engines and product services:

• Work with Apache Spark batch & real time streaming to process data at scale
• Work with Scala microservices hosted on K8s (GKE) to support our products
• Work with MlLib and Tensorflow on Vertex AI to solve forecasting, simulations in addition to a variety of machine learning challenges
• Deploy new models and engines for new ideas and analyses

What you won’t do:

• You won’t be responsible for any frontend application work
• You won’t be managing other team members
• You won’t work on technical debt as that is something we don’t have
• You won’t be bored as there’s always new technologies to implement and new innovations to test

The Senior Data Engineer at Dandi is a full-time position, reports to the CTO and comes with remote, in-person or hybrid options. The Sr. Data Engineer comes with a $200,000 base salary, with 0.5% fully diluted ownership with standard four-year vesting.

Ideal background

We encourage all applicants to apply. These are the ideal skills and attributes for the position, but we’re quite open minded, so reach out! See below: 

• B.S. in Computer Science or a higher degree
• Advanced knowledge of different non-relational schema models (Column Family, Graph, Document, Object)
• Experience with advanced topologies and architecture design such as KAPPA and LAMBDA architectures
• Experience in Scala or any similar JVM based language
• Experience in Python
• Experience with high throughput systems
• Experience in Go is a great bonus to have