AI Call Grading That Trains Account Managers
How we graded 452 of a marketing agency's own client reporting calls with AI, found what its best calls do differently, and turned those moments into training so account managers can run the calls on their own.
1,816
Call transcripts mined
452
Reporting calls graded on one rubric
9
Weighted call fundamentals
CLIENT OVERVIEW
A digital marketing agency serving personal injury law firms and home service businesses, with a director of account management and a team of account managers running a monthly reporting call for each client. The CEO and the head of SEO had a way of telling the client's story that worked, but it lived in their heads. So they kept joining the calls. Across the calls we reviewed, the CEO was on 265 of 422 graded reporting calls. Every one of those hours came out of running the business.
THE PROBLEM
What was broken
The CEO was the reporting process
Monthly reporting calls are the one hour a month a client sees the agency. The CEO joined most of them because that hour decides renewals. The director of account management had been working to move the CEO into a coaching seat, but there was nothing to hand the account managers that captured how the CEO ran a call.
No shared standard for a good call
Account managers, SEO leads, and leadership each used different tools and told different stories. Feedback on calls came down to gut feel. Nobody could say what a great call did differently from an average one, so nobody could train it.
Numbers without meaning
Without a framework, calls drifted toward reading the dashboard: average position, impressions, traffic. Clients wanted to know about leads and cases. When the number on screen doesn't connect to their business, the client ends up doing the math out loud, and the call stops building trust.
THE GOALS
What success looks like
Find what the agency's best reporting calls actually do, using its own recordings
Grade every reporting call against one weighted, repeatable rubric
Give account managers a playbook, a scorecard, and real call clips to train on
Move the CEO from monthly reporting calls to quarterly strategy calls and coaching
OUR APPROACH & SOLUTION
Learning from the calls that already worked
We didn't start from a generic sales framework. We started from the agency's own recordings. We pulled every transcript, isolated the reporting calls, read the strongest ones line by line, and only then wrote the rubric. Every script in the training comes from a real call the team already ran.
Phase 1
Call Inventory
Pulled 1,816 call transcripts from the agency's recording tool, sorted out 1,026 client-facing calls, and isolated 457 monthly reporting calls across 84 client accounts. The median call ran 41 minutes.
Phase 2
Pattern Mining
Read the 42 strongest calls line by line and pulled out what they had in common: opening on the client's business, setting an agenda, turning every metric into leads and cases, leading with a real win, getting to bad news before the client does, honesty, real dialogue, clear next steps, and checking that it landed.
Phase 3
AI Grading at Scale
Turned those nine fundamentals into a weighted rubric and had AI graders score every call from its full transcript. 452 calls were graded and 422 were scoreable. The data also killed three beliefs the team held, including that the best calls talk less and that the CEO's presence alone raised the score.
Phase 4
Training Kit
Built a reporting call playbook, a 14-slide training deck, a scorecard that flags any call with two or more fundamentals under a 3 for coaching, and a video library of 47 timestamped clips from real calls, good and bad. The kit is built so a new account manager watches the clips, reads the playbook, and scores a recorded call before running one.
Phase 5
Measurement and Handoff
Re-graded 165 calls from before the training and 19 after it in one blind run, so grader drift couldn't fake a result. Added a tenth fundamental at the CEO's request, anchoring each call to the client's 90-day plan, and mapped the call flow onto the agency's reporting platform.
WHAT WE BUILT
Features and capabilities
01
Call Inventory Pipeline
Python scripts pull every transcript and recording from the agency's call recorder, separate client-facing calls from internal meetings, and parse how each call is structured. Grading starts from the 457 calls that are actually monthly reporting calls, not every meeting on the calendar.
02
Weighted Call Rubric
Nine fundamentals, each scored 1 to 4 against written anchors that describe what a 1, 2, 3, and 4 look like. Turning metrics into the client's money and handling bad news carry the most weight, 18 points each. Every call rolls up to a 0 to 100 score in one of five bands, from Needs work to Elite.
03
AI Grader With Reconciliation
AI graders read each call's full transcript, score every fundamental, and pull out the call's standout moment and its best line worth reusing. A reconciliation script checks every batch against the call list, because graders silently dropped calls in 2 of the 46 batches.
04
Missed-Moment Finder
A second AI pass finds up to three timestamped moments per call that explain where it lost points. An account manager clicks a timestamp and watches the exact moment the call slipped.
05
Reporting Call Playbook
A playbook of about 5,300 words with an eight-segment flow for a 35 to 45 minute call, guidance on presenting each screen, a two-role model for larger accounts, and a swipe file of real lines from the agency's best calls.
06
Self-Serve Scorecard
A spreadsheet with a "Score a Call" tab and a "Log Many Calls" tab that computes the 0 to 100 score and band and flags any call with two or more fundamentals under a 3. A companion guide gives an anchor and an example for every score, so a manager can grade a recorded call without an AI run.
07
Video Training Library
A training app that holds all 457 reporting calls, plus 47 timestamped clips sorted into 13 themes: 8 on what great calls do and 5 on where calls go sideways. The rubric and a scorecard practice sit in the same place.
08
Blind Before-and-After Grading
Earlier and later calls are re-graded together in one shuffled, blind run. Separate runs drifted 8.1 points apart, so grading everything in one run keeps grader drift from faking a change in either direction.
09
Platform Call Guide
A call guide and training page map each page of the agency's reporting platform to the fundamentals, such as the status page to the agenda and bad news, and the value page to turning metrics into money. It adds a tenth fundamental, anchoring the call to the client's 90-day plan, with proposed weights for leadership to confirm.
KEY TAKEAWAYS & IMPACT
What changed for this business
61%
Of calls under the bar, made visible
Leadership got its first objective read on reporting quality: an average of 51 out of 100, with 258 of 422 calls under 55. Honesty was already the team's strongest habit. Setting an agenda and turning metrics into money were the weakest, so the training went straight at those.
47
Real moments to train on
Instead of a slide of tips, account managers watch 47 timestamped clips from their own calls, each tied to a fundamental. They see the exact moment a call went well or slipped, then practice the fix.
2 for 2
Blind grades matched the human picks
Before any AI grading, our line-by-line review picked the two strongest calls. The blind grade ranked those same two calls first and second, a check that the rubric scores what experienced people already recognize as a great call.
THE OUTCOME
The wins
-
Leadership has one objective, repeatable read on reporting quality across 422 calls, in place of feedback that came down to gut feel.
-
Coaching targets the habits with the widest gap between the best and worst calls, like handling bad news, where top-quarter calls averaged 3.30 out of 4 and bottom-quarter calls averaged 2.06.
-
Three beliefs about what makes a good call were tested against the data and dropped, so training time goes to behaviors the data supports.
-
Every script in the training is a real line from one of the agency's own calls, so account managers learn from their colleagues' best moments instead of a generic sales framework.
-
A manager can score any recorded call in a spreadsheet and get a score, a band, and a coaching flag without running the AI graders.
-
Before-and-after checks run in one blind batch, so any change in scores reflects the calls and not drift between grading runs.
-
The reporting platform and the playbook follow the same call flow, so an account manager moves through the platform in the order the training teaches.