AI Quality Assurance (QA) Engineer

Apply

Your mission

You’ll be an enthusiastic and innovative AI Quality Assurance (QA) Engineer with a passion for cutting-edge AI and machine learning technologies. This role is integral to ensuring the reliability, scalability, and fairness of our AI systems. You will develop and maintain AI-focused test automation frameworks, evaluate model performance, and embed quality into every step of our AI/ML lifecycle. You will work closely with Design, Front-end and Back-end engineers, and product teams to build robust, automated, and intelligent QA solutions that drive trust in our AI products.

About Journey

Journey is a team of digital specialists based in Auckland and London. We love what we do, and we do it well. We look after each other and have fun together.

We create digital customer experiences through a blend of consulting, design, and development. Our focus is on digital products as well as mobile apps, and software for national and international clients. Our goal is to make Journey the place where you can do the best work of your career and enjoy yourself at the same time.

What’s the role about?

This is a full-time, permanent role (guaranteed minimum 40 hours per week).
Work-from-home option of 2 days per week.
Location: Hobsonville Point, Auckland

Key responsibilities‍

AI & Automation Testing

Design and implement automated test frameworks tailored for AI/ML pipelines, APIs, and model endpoints.
Design and develop intelligent automation frameworks for web and mobile platforms using tools such as Selenium, Appium, Playwright, or equivalent. Leverage AI-based solutions to enhance test stability and automated test case generation.
Develop test strategies and scenarios that validate ML model accuracy, bias, drift, explainability, and fairness.
Conduct functional and non-functional testing of AI systems across a range of inputs, environments, and failure cases.
Build regression suites to test model behavior against continuous training cycles.

Tools, Pipelines & Engineering

Develop and maintain automation using tools like pytest, Selenium, Playwright, Appium, or custom AI QA tools.
Use programming languages such as Python (preferred), Java, Typescript, or C# to write scalable test automation code.
Implement and optimize CI/CD pipelines (e.g., GitHub Actions, Azure DevOps) that integrate AI testing into release processes.

Cross-Platform & Cloud

Validate AI workloads deployed on cloud platforms (AWS, Azure, GCP) and on-premise infrastructures.
Ensure performance consistency across deployment environments.

Innovation & AI Integration

Stay current with AI developments (e.g., GPT models, transformer architectures, reinforcement learning).
Innovate internal QA processes by integrating AI for test generation, prediction of test coverage gaps, or anomaly detection.
Utilise AI-powered IDEs and coding assistants such as Cursor, GitHub Copilot, Windsurf, or similar to accelerate test development and code reviews.

Functional & Non-Functional Testing

Execute comprehensive testing across browsers, devices, and platforms using cloud-based tools like BrowserStack . Ensure functionality, usability, performance, and compatibility through both manual and AI-augmented test strategies.

Mobile App Testing

Conduct comprehensive mobile app testing (iOS and Android) through manual validation and AI-powered automation. Use real devices and simulators to test functionality, UI/UX, performance, and edge cases.

Collaboration & Quality Culture

Collaborate with data scientists and MLOps teams to define testing checkpoints across the model lifecycle.
Translate technical model insights into QA metrics and documentation accessible to non-technical stakeholders.
Champion a "shift-left" approach to embed quality earlier in AI feature development.

About you

4–5 years of experience in test automation, with at least 2+ years focused on AI/ML or data-intensive systems.
Strong software engineering skills with Python (preferred) or Java/C#/Typescript.
Experience with automated testing tools (e.g., Pytest, Selenium, Playwright, Appium).
Solid understanding of AI/ML concepts: model training, evaluation metrics (accuracy, precision, recall), overfitting, bias, fairness.
Familiarity with model validation, performance monitoring, and testing strategies for ML models.
Proficiency in working with CI/CD and DevOps tools (e.g., Git, GitHub Actions, Azure DevOps).
Exposure to cloud-based AI services (AWS SageMaker, Azure ML, Google Vertex AI) and on-prem environments.
Strong analytical thinking and debugging capabilities.
Experience testing LLM applications, prompt engineering, or deploying AI agents in production.
Knowledge of AI ethics and fairness testing methodologies.
Experience with BDD/TDD and writing automated tests as part of the development process.
Exposure to tools like TensorFlow Model Analysis (TFMA), MLflow, Great Expectations, or similar.
Bachelor’s or Master’s in Computer Science, Software Engineering, Data Science, or related field.

Benefits

Competitive $$
A team that works hard, has fun and loves a challenge.
Be surrounded by some of the best digital minds in the business.
…and drink great coffee!

Important notes

For this role you must have the legal right to live and work in New Zealand and be based in Auckland or within an easy commute to Auckland. We will not be offering a visa to overseas candidates for this role, and you will not be able to work from offshore.

A competency test and other assessments will form part of the interview process.

Apply now with your CV and a brief cover letter describing your experience in AI/ML testing and what excites you about this role.

If you have any questions, please contact Sarah Ritchie, Head of People & Culture, jobs@journey-digital.com.

We’re invested in your health & happiness.

Caring and supportive team

We're not just a company, we're a tight-knit family that truly cares. We're all about the people here and we've got a culture that thrives on collaboration, innovation, and a decent dose of fun. Think social shindigs, fridge loaded with drinks, snacks at arm's length, and an office that feels like a second home. Oh, and did we mention we love playing the long game? Stick with us and we'll celebrate your commitment with our long-service recognition.

100% radiation

One of our Journey values is ‘be a radiator’, which is all about radiating positivity and joy and being solutions focused – and that’s a prerequisite for each new person who joins our team. That means that you’ll be working with 100% radiation all around you. You’ll see it and feel it and it will brighten your day!

Professional growth

We’re on a never-ending quest for knowledge and we’re bringing you along for the ride. From team training to individual learning opportunities, we’ve got your back. Plus, our Career Progression Framework is your personal roadmap to skill development and career advancement.

A great place to work

We care deeply about the wellbeing of our people. We prioritise a genuinely supportive culture: regular check-ins with Team Leaders, realistic hours, and active management of stress. Our serene Auckland office is just steps from the water – perfect for a lunchtime stroll or a moment of sunshine to reset your day. Our London office sits in the heart of Old Street’s creative and tech hub – buzzing with energy and great food!

Joy

At Journey, joy is part of the job. Whether it’s playing team games, light-hearted meetings, a round of pool or table tennis, or the friendly dogs that visit the office – we believe in laughter, connection and creating a workplace where people feel valued, respected and genuinely happy to be.

Flexible working

We get it, life’s not a one-size-fits-all deal. All our team can work from home for two days each week. Plus we also offer flexible start and finish times so you can fit your work hours around school pick-ups, morning gym workouts or beating the rush hour traffic.