Senior Software Engineer- Data Acquisition

We are looking for people with experience gathering data from the web at scale. You’ll be part of Luma’s infrastructure team, working cross-functionally with Applied AI and Research on mission critical work-streams.

Responsibilities

• Own / lead engineering projects for data acquisition.
• Work with Research and Applied AI to ensure their data and technical needs are met.
• Develop and deploy highly scalable distributed systems capable of handling petabytes of data.
• Build and gather high quality datasets by collaborating closely with the research team.
• Build and maintain backend services for data storage.
• Architect and implement algorithms for data management including indexing & search.
• Conduct and analyze system performance experiments.

Experience

• Proficiency in Python.
• Experience of 5+ years in web-scraping at-scale.
• Experience building and working efficiently in very large distributed systems.
• Experience working with Kubernetes, proxies and other relevant tools.
• Experience working with research datasets.
• Please note this role is not meant for recent grads.

All your applications are reviewed by real people.