Research Scientist

This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Research Scientist based in Brazil.

Join a cutting-edge AI research environment where you will help define how frontier language models are evaluated and benchmarked. In this highly technical role, you will design rigorous evaluation methodologies, develop high-quality benchmarks, and collaborate with subject matter experts to assess advanced capabilities across reasoning, coding, multimodal systems, and agentic AI. You will play a key role in ensuring evaluation frameworks are scientifically robust, reproducible, and aligned with the evolving needs of leading AI research organizations. This opportunity is ideal for a researcher who enjoys solving complex methodological challenges and contributing to the advancement of trustworthy AI evaluation.

• Design and develop original evaluation methodologies that accurately measure the capabilities of frontier AI models across reasoning, coding, agentic workflows, tool use, and multimodal tasks.
• Create comprehensive benchmark packages supported by expert-verified ground truth, rigorous quality control processes, deterministic verification methods, and well-calibrated scoring rubrics.
• Evaluate benchmark quality by ensuring strong construct validity, appropriate task discrimination, sufficient performance headroom, and minimal benchmark contamination.
• Recruit, assess, and coordinate subject matter experts to develop, validate, and review high-quality evaluation datasets and benchmark content.
• Serve as the final technical reviewer for evaluation quality, benchmark correctness, and task complexity before external delivery.
• Collaborate with AI research organizations to understand evaluation objectives and translate research needs into effective benchmarking strategies.
• Lead evaluation pilots from initial concept through final delivery, ensuring all outputs meet demanding scientific and technical standards.
• Continuously refine evaluation methodologies to support emerging capabilities and evolving best practices in frontier AI research.

Requirements

• Demonstrated research experience in machine learning evaluation, benchmarking, AI assessment methodologies, or a closely related field.
• Proven expertise designing or contributing to AI benchmarks, evaluation research, published datasets, or similar research initiatives relied upon by AI organizations.
• Strong understanding of large language model evaluation, including benchmarking methodologies, scoring rubrics, pass rates, contamination risks, construct validity, and discriminative task design.
• Deep knowledge of code model evaluation and technical assessment frameworks for advanced AI systems.
• Experience working with subject matter experts and maintaining rigorous quality standards across technical research projects.
• Excellent analytical, research, and problem-solving skills with a strong attention to scientific rigor.
• Outstanding written and verbal communication skills in English.
• Ability to work independently in a remote, research-driven environment while collaborating effectively with multidisciplinary teams.
• Spanish language proficiency is considered a plus.

Benefits

• Fully remote contract position based in Brazil.
• Opportunity to work on frontier AI research and evaluation projects with global impact.
• High level of ownership and influence over research methodology and benchmark design.
• Collaboration with leading AI experts, researchers, and technical specialists.
• Flexible remote working environment that supports autonomy and innovation.
• Exposure to cutting-edge advancements in large language models and AI evaluation.