Data Engineer

Job Scope

The Data Engineer will design and develop the infrastructure necessary to capture and make sense of the massive amount of information made available to 500 Global by virtue of its relationships with a portfolio of over 3,000+ startups from their earliest stage. The ideal candidate is 50% thinker and 50% do-er, has a bias toward action, has an insatiable curiosity around ways to apply AI to business processes, and knows where/when to cut corners (and where/when not to.)

Essential Functions

• Lead integrations and develop connectors with various data sources, ensuring smooth and secure data extraction, including Salesforce, AirTable, Skyvia, Allvue, SaplingHR, and proprietary web portals.
• Design and implement scalable and reliable data infrastructure using appropriate technologies (e.g. data lakes, etc.)
• Document processes, best practices, and knowledge for transparency and reproducibility while staying updated on emerging technologies and industry best practices.
• Build and maintain the infrastructure required for machine learning (ML) and NLP models and LLMs, ensuring scalability, reliability, and performance.
• Collaborate closely with domain experts, and other stakeholders to understand and address business needs, participating in relevant and company-wide decisions.
• Additional projects as determined by business needs.

Minimum Qualifications

• Bachelor's degree in Computer Science, Engineering, or a related field.
• 4+ years of experience in designing and implementing scalable and reliable data infrastructure using cloud technologies.
• Solid understanding of data governance, security, and compliance standards in a cloud environment.
• Hands-on experience building API integrations with various data sources
• Proficiency in cloud platforms such as Amazon Web Services (AWS), Microsoft Azure, or Google Cloud Platform (GCP), including cloud services relevant to data storage, processing, and analytics.
• Excellent problem-solving skills and ability to troubleshoot complex data infrastructure and integration issues.
• Demonstrated experience building something from the ground up
• Experience staying updated on emerging technologies and industry best practices relevant to data engineering and analytics.
• Strong documentation skills with a focus on documenting processes, best practices, and knowledge for transparency and reproducibility.
• Proven track record of collaborating closely with cross-functional teams, domain experts, and stakeholders.

Preferred Qualifications

• Experience working Data Lakes, Data warehouses, and ETLs.
• Experience building and maintaining infrastructure for machine learning (ML) and natural language processing (NLP) models, ensuring scalability, reliability, and performance.
• Knowledge of web development concepts (HTML, CSS, JavaScript, TypeScript, Python).Previous experience in PE, VC, or financial sectors.
• Experience working with (or at) startups in fast-pacing environments.
• A public GitHub repository with unlocked packages you have built