Overview
Senior data annotation jobs in Montreal focus on creating and evaluating high-quality training data for AI/ML systems. At Rex.zone, you will support LLM training pipelines through data labeling, RLHF evaluation, prompt evaluation, and QA review to improve model performance and safety in a remote, full-time role.
Key Responsibilities
- Deliver high-accuracy data labeling and review across NLP, LLM evaluation, computer vision annotation, and content safety labeling
- Design, maintain, and enforce annotation guidelines compliance and taxonomy standards
- Lead QA evaluation workflows including sampling plans, inter-annotator agreement checks, and adjudication
- Execute RLHF tasks such as pairwise ranking, preference labeling, and consistency audits
- Conduct prompt evaluation and rubric-based scoring aligned to model requirements
- Perform dataset audits, ambiguity logging, and label error analysis to improve training data quality
- Partner with engineering/ML stakeholders to define acceptance criteria and feedback loops into LLM training pipelines
- Document edge cases and escalation paths to reduce guideline drift
- Support onboarding and calibration using examples, gold sets, and retraining plans
- Track quality metrics and recommend process improvements that improve throughput without sacrificing accuracy
Required Qualifications
- Senior data annotation or data labeling operations experience with QA evaluation ownership
- Strong understanding of taxonomy design, guideline adherence, and ambiguity resolution
- Hands-on experience with LLM evaluation, RLHF, or prompt evaluation workflows
- Familiarity with NLP tasks such as named entity recognition and text classification
- Comfortable working remotely with cross-functional stakeholders
Nice to Have
- Content safety labeling or policy-driven evaluation experience
- Inter-annotator agreement metrics and quality sampling methodologies
- Computer vision annotation exposure (bounding boxes, polygons, segmentation)
- Multilingual evaluation or locale-specific guidelines
Work Model
Remote, Full-time.
Apply with relevant annotation experience and examples of QA evaluation or guideline development work.