Job title: Agentic QA Engineer – Generative AI & Agentic Systems (Agent, Multi‑Agent Testing)
Location: Dallas, TX (Onsite)
Duration: Long term Contract
Mode of interview: 2 F2F rounds interviews in Dallas, TX (Must)
No Third Party
Required Qualifications
- 7+ years in Software QA/Testing, with 2+ years in AI/ML or LLM-based systems; hands-on experience testing agentic/multi-agent architectures.
- Strong programming skills in Python experience building test harnesses, simulators, and fixtures.
- Experience with LLM evaluation (exact/soft match, BLEU/ROUGE, BERTScore, semantic similarity via embeddings), guardrails, and prompt testing.
- Expertise in distributed systems testing latency profiling, resiliency patterns (circuit breakers, retries), chaos engineering, and message queues.
- Familiarity with orchestration frameworks (LangChain, LangGraph, LlamaIndex, DSPy, OpenAI Assistants/Actions, Azure OpenAI orchestration, or similar).
- Proficiency with CI/CD (GitHub Actions/Azure DevOps), observability (OpenTelemetry, Prometheus/Grafana, Datadog), and feature flags/canaries.
- Solid understanding of privacy/security/compliance in AI systems (PII handling, content policies, model safety).
- Excellent communication and leadership skills; proven ability to work cross-functionally with Ops, Data, and Engineering.
Preferred Qualifications
- Experience with multi-agent simulators, agent graph testing, and tooling latency emulation.
- Knowledge of MLOps (model versioning, datasets, evaluation pipelines) and A/B experimentation for LLMs.
- Background in cloud (AWS), serverless, containerization, and event-driven architectures.
Prior ownership of cost/latency/SLAs for AI workloads in production.