About the Role
Our AI Conversation Agent is one of the most complex products in our portfolio. As the dedicated QA Engineer for this product, you will own quality end to end, designing customer-oriented test scenarios, driving non-functional testing, and building the automation infrastructure that lets us ship fast without leaking defects to production.
Key Responsibilities
Customer-oriented use case design
- Design and execute test scenarios that mirror real customer conversations—interruptions, overlapping speech, background noise, accents, off-script digressions, and edge-of-spectrum user behavior.
- Proactively explore the product as a user would, hunting for inconsistencies in conversation flow, response quality, and experience across channels.
- Translate customer-reported issues into reproducible test cases and regression suites.
Non-functional requirements testing
- Own systematic non-functional testing: interruption handling, noise robustness, weak-network conditions (high latency, jitter, packet loss), and sustained-load stability.
- Build and operate automated call-testing and load-testing platforms, simulating thousands of concurrent calls and continuously monitoring system stability and key metrics.
- Evaluate and quantify core experience metrics - ASR recognition accuracy, TTS naturalness, end-to-end latency, and drive issues to closure with engineering.
Quality infrastructure & observability
- Establish end-to-end logging and tracing across the conversation pipeline to speed up root-cause analysis and reduce time-to-diagnosis.
- Build maintainable automated test frameworks (Python/Java) integrated into CI/CD, covering both functional and non-functional scenarios.
- Define, track, and report quality metrics; make release quality visible and data-driven for stakeholders.
Requirements
- 4+ years of QA/testing experience, ideally with voice, AI, or real-time communication products.
- Proficient with Python or Java, experienced in building automated test frameworks—not just writing scripts, but engineering test infrastructure.
- Deep quality mindset with a talent for uncovering boundary conditions and failure scenarios others miss.
- Patience and rigor for detailed manual exploratory testing—genuine care for product detail and consistency.
- Ability to work independently as the dedicated QA owner of a complex product, collaborating closely with engineering and product teams.
Preferred Qualifications
- Experience testing VoIP, WebRTC, or other real-time network protocols.
- Hands-on experience evaluating ASR/TTS systems or LLM-based conversation products.
- Experience with observability tooling (logging, tracing, dashboards) and load-testing platforms.
- Background in fintech, banking, or other regulated enterprise domains.