A QA Bot That Checks Website Changes Before Clients Do
How we gave a WordPress agency a QA bot that checks every finished change request on the real pages, in a median of 20 seconds, and sends misses back to the team with screenshots before the work goes to the client.
20 sec
Median check per finished request
25
Client sites watched
4 days
From first version to live
CLIENT OVERVIEW
A digital marketing agency that maintains WordPress sites for its clients. When a client asks for a change, the team makes it and marks the request done. A baseline audit found that only about 60% of finished change requests came back ready to send to the client. The rest needed another round of fixes before the client could see them.
THE PROBLEM
What was broken
Done didn't mean ready
A baseline audit found only about 60% of finished change requests came back ready to send to the client. Every miss meant another round of fixes before the client could see the change.
One change, many places
A single request can need the same change in several places on a site. Checking only the page named in the request misses the rest, and checking every place by hand is slow.
Site health needed someone to look
Whether a site was up, whether it looked different from the day before, and whether a new client's site came through its move intact all depended on someone opening the pages and checking.
THE GOALS
What success looks like
Check every finished change request on the real pages before it goes back to the client
Work out every place a change must appear, not only the page named in the request
Send misses back to the team with screenshots
Watch client sites for uptime and visual changes, and check a new client's site when it moves over
OUR APPROACH & SOLUTION
Checking the real pages, not the ticket
We built the bot around the moment a request is marked done. It reads the request, works out every place the change must appear, opens the real pages, and checks each one. We shipped a version a day: the first on November 3, 2026, a second on November 4 that checked real requests in 7 to 29 seconds each, and a third on November 5 with before-and-after checks. It went live for the agency on November 6, and the team is reviewing its first findings.
Phase 1
Three Versions in Three Days
The first version shipped on November 3, 2026, and read a finished request, worked out where the change must appear, and checked each place on the real pages. The second, on November 4, checked real requests in 7 to 29 seconds each. The third, on November 5, added before-and-after checks.
Phase 2
Live for the Agency
The bot went live for the agency on November 6, 2026, with 252 automated tests on the worker that runs the checks. The team is reviewing its first findings.
Phase 3
Site Monitoring
The bot has 25 client websites set up, with uptime checks on 23 of them. It also runs a daily screenshot comparison and checks a new client's site when it moves over.
Phase 4
AI Ticket Triage
Also in rollout at the same agency: AI triage that sorts each incoming ticket into work or conversation and posts an internal routing note. Routing notes have posted since late September 2026, and the first production backfill classified 427 tickets for $0.98.
WHAT WE BUILT
Features and capabilities
01
Starts When a Request Is Marked Done
The bot picks up a website change request when the team marks it done and reads what the client asked for.
02
Works Out Every Place a Change Must Appear
From the request, the bot plans every place the change has to show up on the site, not only the page the request names.
03
Checks the Real Pages
The bot opens the real pages and checks each place on its plan. In a 21-day look back over finished requests, a check took a median of 20 seconds.
04
Sends Misses Back With Screenshots
When a place doesn't show the change, the bot sends it back to the team with a screenshot, so the fix starts from what the page shows.
05
Before-and-After Checks
Since the third version, the bot compares a page before and after the change, so it can tell what the change did.
06
Uptime and Daily Screenshot Comparison
Uptime checks run on 23 of the 25 sites set up, and a daily screenshot comparison shows when a page looks different from the day before.
07
Site Move Checks
When a new client's site moves over, the bot runs checks on it, so the team sees what came through the move.
08
AI Ticket Triage
In rollout at the same agency, AI triage sorts each incoming ticket into work or conversation and posts an internal routing note. The first production backfill classified 427 tickets for $0.98.
09
A Tested Worker
The worker that runs the checks has 252 automated tests, so a change to the bot gets tested before it checks a client's site.
KEY TAKEAWAYS & IMPACT
What changed for this business
20 sec
Median check per finished request
In a 21-day look back over finished requests, a check took a median of 20 seconds, so the bot can check each request as it's marked done.
25
Client websites set up
The bot has 25 client websites set up, and 23 of them have uptime checks. A daily screenshot comparison runs alongside the change checks.
$0.56
For 50 planning calls
The bot plans each check with AI before it opens a page. In early production, 50 planning calls cost $0.56.
THE OUTCOME
What it's built to deliver
-
Every finished change request gets checked on the real pages before it goes back to the client, in a median of 20 seconds.
-
Misses go back to the team with screenshots, so the fix starts from what the page shows.
-
A change gets checked everywhere it has to appear, not only on the page the request names.
-
Uptime checks and a daily screenshot comparison watch client sites without anyone opening them.
-
The team is reviewing the bot's first findings against a baseline of about 60% of finished requests ready to send.
-
Incoming tickets get a routing note that sorts them into work or conversation, and the first 427 cost $0.98 to classify.