About The Role
AI prompt engineers design, test, and evaluate prompts that steer LLMs toward accurate, safe, and useful outputs across production workflows. You will collaborate with engineering and AI/ML data operations to improve model behavior through prompt iteration, evaluation, and RLHF-adjacent feedback.
What You Will Do
- Design system prompts, user prompts, and tool instructions for LLM-powered products (chat, search, summarization, extraction, coding assistants).
- Run prompt evaluation and QA evaluation using rubrics to measure correctness, groundedness, safety, and style adherence.
- Create and maintain datasets for prompt testing, including data labeling and preference data aligned to RLHF objectives.
- Analyze failures (hallucinations, instruction gaps, unsafe content) and propose prompt, policy, or data fixes.
- Write and enforce annotation guidelines; ensure training data quality and compliance.
- Support LLM training pipelines, regression testing, and model performance improvement.
- Contribute to content safety labeling, refusal behavior testing, and jailbreak/misuse resistance.
- Document prompt libraries, evaluation sets, and decision logs for reproducible experiments.
Required Qualifications
- Mid-Senior experience building or optimizing prompts for LLM applications in production or rigorous evaluation settings.
- Strong writing and reasoning skills to translate product intent into precise instructions and constraints.
- Hands-on experience with rubric-based prompt evaluation and QA evaluation.
- Understanding of RLHF concepts (preference data, calibration, helpfulness/harmlessness tradeoffs).
- Ability to define measurable criteria for quality and safety under ambiguity.
Preferred Qualifications
- Experience with NLP tasks (summarization, classification, NER, extraction) and RAG evaluation.
- Familiarity with content safety labeling, policy writing, and red-teaming.
- Experience building evaluation datasets and running QA programs.
- Basic scripting/analytics (spreadsheets, SQL, Python) for tracking and error analysis.
Compensation
Hourly rate: $30–$50/hr (BASE_SALARY).
Remote-ready roles across Canada time zones.