Linguist II

Iron Systems is an innovative, customer-focused provider of custom-built computing infrastructure platforms such as network servers, storage, OEM/ODM appliances & embedded systems. For more than 15 years, customers have trusted us for our innovative problem-solving combined with holistic design, engineering, manufacturing, logistics, and global support services.

Job Title: Linguist II

Location: US - WA - Redmond

Job Description: Summary

• We seek a skilled Linguistic Engineer to maintain high-quality data annotation practices for our topline quality evaluation pipelines.
• In this role, you will be instrumental in assessing the quality of the data processed by human annotation teams.
• Using new and existing data quality oversight tooling and methods, you will collaborate closely with a cross-functional team of engineers, research scientists, project managers, and data scientists to report on and guide improvements in annotation quality, ensuring their accuracy and reliability.

Responsibilities

• Design and conduct experiments for evaluating Rater quality, and inter-rater reliability; write summary reports based on findings and speak on those findings in meetings.
• Deliver rater feedback and design and execute rater quality improvement plans.
• Develop manual and automated processes for multiple concurrent projects ensuring high-quality labeled data.
• Create and perfect LLM Quality grading queues and LLM-related component quality error attribution grading queues.
• Build datasets and write guidelines in a rapid fashion for new and emergent LLM data rating, creation, and annotation needs.
• This is not an annotator position, though annotation will occasionally be required to accommodate urgent requests.

Skills

• Basic Python, and intermediate SQL with a strong desire to grow required for this position.
• Basic knowledge of linguistics (some Phonetics, Syntax, Semantics, and Dialectology) is required.
• Ability to analyze numerical rating results and draw conclusions based on data.
• Intermediate stats/math for data analysis.
• Written and oral communication skills necessary for communicating project milestones and for reporting findings.

Education/Experience

• Bachelor's degree in computational linguistics, speech science, or related field OR graduate degree in linguistics OR industry similar experience required.
• Master's degree or beyond preferred OR equivalent industry experience.
• Additional learning via Coursera, Udemy, or other online resources is great too.