Search and AI Evaluations

Summary Join NIST to lead research and development for AI and Generative AI testing, evaluation, and validation across diverse modalities. You will independently design and code web-based measurement applications for data quality, scoring, and analysis. This role involves leading technical publications and providing programming expertise to core research areas. By driving these evaluation processes, you will ensure the integrity of measurement and the rigorous standards of critical agency AI systems. Learn more about this agency Duties As a Search and AI Evaluations technical expert, you will be responsible for advancing the safety and efficacy of next-generation AI systems. In this position, you will: Plan and execute comprehensive research activities focused on Test, Evaluation, Validation, and Verification (TEVV) processes for Artificial Intelligence and Generative AI across diverse modalities (e.g., text, image, audio) and use cases. Independently design, program, and maintain web-based applications and support tools used for data quality assurance, automated scoring, complex analysis, and the generation of technical reports. Lead the development of technical briefs, white papers, and peer-reviewed publications; present findings to stakeholders to influence AI policy and technical standards. Requirements Conditions of employment U.S. citizenship Males born after 12-31-59 must be registered for Selective Service Suitable for Federal employment Bargaining Unit Position: No Qualifications Basic Requirements: Bachelor's degree in computer science or bachelor's degree with 30 semester hours in a combination of mathematics, statistics, and computer science. At least 15 of the 30 semester hours must have included any combination of statistics and mathematics that included differential and integral calculus. All academic degrees and coursework must be from accredited or pre-accredited institutions. In addition to the above requirements, all applicants must have one year (52 weeks) of specialized experience equivalent to at least the GS-12 level (ZP-III at NIST). Specialized experience is defined as: Experience in engineering custom web-based applications (Python/React preferred) for manual and automated data labeling, quality assurance, scoring, and performance measurement for text and multimodal AI systems. Experience executing test, evaluation, validation, and verification (TEVV) protocols across text, vision, and audio modalities. Experience in designing scoring frameworks to evaluate generative AI outputs and LLM-as-a-judge protocols. OR Master's or equivalent graduate degree OR A combination of education and experience. Experience refers to paid and unpaid experience, including volunteer work done through National Service programs (e.g., Peace Corps, AmeriCorps) and other organizations (e.g., professional, philanthropic, religious, spiritual, community, student, social). Volunteer work helps build critical competencies, knowledge, and skills, and can provide valuable training and experience that translates directly to paid employment. You will receive credit for all qualifying experience, including volunteer experience. The qualification requirements in this vacancy announcement are based on the U.S. Office of Personnel Management (OPM) Qualification Standards Handbook. If requesting reconsideration of your qualification determination, please refer to the following site: Applicant Reconsideration Education This position has an education requirement. Transcripts must be submitted to validate that the education requirement has been met. Unofficial transcripts will be accepted in the application package. However, an official copy will be required before a final offer of employment. Use of foreign education for qualifications. An accredited organization must evaluate education completed outside of the U.S. to ensure that it is comparable to education received in accredited institutions in the U.S. Click here to view a listing of accredited organizations from the Department of Education's website. A copy of the foreign education evaluation (containing the results with a course-by-course listing) is required with your application. Additional information This notice is issued under direct-hire authority to recruit new talent to occupations for which NIST has a severe shortage of candidates. This position is covered under NIST's Alternative Personnel Management System (APMS), a pay-for-performance system with excellent HR flexibilities to help NIST recruit and retain top talent. Learn more about the APMS here! We may share your application package with other selecting officials. Additional selections may be made through this vacancy. Click all links in this vacancy announcement to view additional information or instructions. You MUST select at least ONE ELIGIBILITY to be considered for this position. All documents submitted for this announcement must be legible to make qualification or eligibility determinations. A probationary period may be required. NIST strives to build a flexible and encouraging work environment to bring out the best in our employees. To help our employees balance responsibilities at home and at work, NIST offers a variety of work-life flexibilities. For more information, please visit our Careers website. Expand Hide  additional information Candidates should be committed to improving the efficiency of the Federal government, passionate about the ideals of our American republic, and committed to upholding the rule of law and the United States Constitution. Benefits A career with the U.S. government provides employees with a comprehensive benefits package. As a federal employee, you and your family will have access to a range of benefits that are designed to make your federal career very rewarding. Opens in a new windowLearn more about federal benefits. Review our benefits Eligibility for benefits depends on the type of position you hold and whether your position is full-time, part-time or intermittent. Contact the hiring agency for more information on the specific benefits offered.