Quality engineering in the AI era: keep the discipline, move the feedback closer

Test the user journey, not only the component

Start with the tasks customers need to complete: discover a service, sign in, submit a form, pay, find information or recover from an error. Map the risks and dependencies in each journey. Component checks are valuable, but they do not prove that the whole flow works across the interface, API and data layer.

Layer automation with human review

Use unit tests for logic, integration tests for service boundaries, end-to-end tests for critical flows and exploratory testing for unexpected behaviour. Automate the checks that are stable and repeatable. Keep human review for ambiguous, visual, accessibility and high-impact decisions where judgement matters.

Give AI features their own evaluation set

AI output is probabilistic, so a simple pass/fail test on one prompt is not enough. Build representative test cases, expected source references, unacceptable outputs and scoring criteria. Include adversarial inputs, ambiguous questions, out-of-date sources, refusal cases and fallback behaviour. Compare results across model or prompt changes.

Make release decisions explicit

Define severity levels, acceptance criteria and who can approve a release. Record reproducible defects with environment, steps, expected behaviour and actual behaviour. Retest the fix and run regression checks for nearby functionality. A release checklist should reflect the product’s risk rather than being a generic tick-box exercise.

Bring feedback into everyday engineering

Run tests in pull requests and deployment pipelines where practical. Track escaped defects, flaky tests, time-to-detect, time-to-recover and user-reported friction. Use those signals to improve the test strategy rather than simply increasing the number of automated tests.

Putting the ideas into practice

Modern quality engineering is not less testing; it is earlier and more targeted feedback. AI can accelerate implementation, while risk-based testing, evaluation and release discipline protect the people who rely on the product.