Linguist
Remote
1 Year Contract
Years of Experience: 4-6 year of experience
Must-Have Skills
1 Data Annotation
2 SQL Python
3 Linguistic Background
Nice-to-have Skills
1 Computational Linguistics (Master's preferred)
2 Statistics & Data Science
Reponsibilites
● Dataset Building & Curation: Receive requests from engineers and research scientists, then design and execute synthetic data collections or sample from existing data to produce clean, labeled datasets for LLM training and evaluation.
● Annotation Queue Management: Stand up and maintain grading queues — define rubrics, write annotation guidelines, onboard raters, and iterate on queue design to keep pace with rapidly evolving LLM features.
● Quality Auditing & Error Attribution: Review annotated data daily, flag inconsistencies, run inter-rater reliability checks, and trace quality issues back to specific LLM subcomponents (e.g., factuality, coherence, brevity).
● Data Analysis & Reporting: Use AI to write SQL queries and Python scripts to pull metrics, compute agreement scores, and generate dashboards; summarize findings in weekly reports shared with cross-functional partners.
● Experiment Design: Plan and run controlled experiments to test rater calibration, guideline clarity, or model performance changes — then present results and recommendations in team meetings.
Guideline Authoring: Draft, test, and refine annotation guidelines for new LLM capabilities on tight timelines, ensuring raters can consistently apply them at scale.
● Cross-Functional Collaboration: Attend syncs with engineering, research, and PM stakeholders to align on data priorities, surface quality insights, and inform roadmap decisions for smart glasses LLM features.
Skills:
• Basic Python, basic SQL with strong desire to grow required for this position. Intermediate level preferred.
• Basic knowledge of linguistics (some of Phonetics, Syntax, Semantics, Dialectology) required.
• Ability to analyze numerical rating results and draw conclusions based on data.
• Basic stats / math for data analysis.
• Written and oral communication skills necessary for communicating on project milestones and for reporting findings .
Education/Experience:
• Bachelor’s degree in Linguistics, Computational Linguistics, Speech Science, or related field OR graduate degree in linguistics OR industry similar experience required.
• Master’s degree or beyond preferred OR equivalent industry experience.
• Additional learning via Coursera, Udemy, or other online resources is great too