Shape the quality and safety of AI systems from the ground up.
AI is changing the way software is built — and traditional testing approaches are no longer enough.
Join a fast-growing tech company that’s redefining how businesses connect with their investors. My client has built a powerful cloud-native platform trusted by hundreds of listed and private companies — and they’re only just getting started. Following a recent acquisition and resulting cash injection (by a well-known US FinTech Investor & Incubator), we’re looking for a Senior Quality Assurance Engineer to join a technology team at the forefront of building and deploying AI-enabled products, agents and platforms.
This is a genuinely interesting opportunity for an experienced QA / Quality Engineer who wants to move beyond traditional functional testing and play a key role in AI assurance, evaluation and responsible production deployment.
You’ll combine strong software quality engineering with emerging AI testing practices, developing the frameworks, automation and evaluation strategies that determine whether AI systems are reliable, secure and ready for production.
What you’ll be doing
You’ll have significant ownership across the AI quality lifecycle, including:
- Define quality strategies for AI applications, agents, RAG systems and APIs
- Build automated frameworks covering functional, integration, E2E, regression, performance and security testing
- Design AI evaluation frameworks measuring accuracy, relevance, completeness, groundedness and consistency
- Test for hallucinations, unsupported claims, prompt injection, data leakage and unsafe behaviour
- Validate AI agent workflows, tool selection, decision paths and failure handling
- Build evaluation datasets covering normal, edge-case, ambiguous and adversarial scenarios
- Establish measurable quality thresholds and evidence-based release gates
- Test retrieval pipelines, document ingestion, search relevance and source attribution
- Validate authentication, authorisation, tenant isolation and data-access boundaries
- Perform load, concurrency, latency and resilience testing
- Investigate complex failures using logs, traces, metrics and test evidence
- Integrate automated testing and AI evaluations into CI/CD pipelines
- Monitor production quality, model/data drift and emerging failure patterns
- Work closely with engineering teams to identify root causes and validate fixes
- Provide clear release recommendations and confidently communicate residual risks
What we’re looking for
We’re looking for a senior-quality engineer with a strong automation background and a genuine interest in how AI systems behave in the real world.
You’ll bring:
- Strong experience in quality engineering, software testing or test automation
- Strong Python programming skills, ideally with Pytest
- Experience building automated tests for APIs, distributed services and data-driven applications
- Hands-on experience testing AI, ML, Generative AI or RAG applications
- An understanding of the non-deterministic nature of LLMs and why traditional pass/fail testing isn't always enough
- Experience creating AI evaluation datasets, test cases, scoring criteria and measurable quality thresholds
- Strong knowledge of functional, integration, regression, performance, resilience and security testing
- Experience with REST APIs, asynchronous workflows, databases and cloud applications
- CI/CD experience with tools such as GitHub Actions, Azure DevOps, Jenkins or equivalent
- Cloud experience, preferably AWS
- A strong understanding of data privacy, access control and secure software development
- Excellent analytical and problem-solving skills
- The confidence to challenge assumptions and recommend that a system should not be released when the evidence doesn't support it
AI experience that will set you apart
Experience in any of the following would be highly advantageous:
- Agentic AI or multi-agent systems
- AI systems that invoke external tools and APIs
- LLM evaluation and model-based evaluation techniques
- Prompt injection, adversarial testing or AI red teaming
- Retrieval technologies, embeddings, vector databases and semantic search
- Testing ML models for bias, drift, explainability and performance degradation
- AWS AI services including Amazon Bedrock, SageMaker, AgentCore or CloudWatch
- Financial services or another highly regulated environment
Why this role?
This isn't a role where you'll simply be writing test cases against an existing application.
You'll help establish how AI quality is measured, how risks are identified and what evidence is required before an AI system reaches production.
You'll have the opportunity to work on complex, evolving technology where the boundaries of conventional QA are being pushed — helping engineering teams build AI systems that are not just impressive in a demonstration, but reliable, secure, measurable and fit for real-world use.
If you're a senior QA or automation engineer who is fascinated by AI and wants to be part of defining the next generation of software quality, this is an opportunity worth exploring.