This rubric is for reviewing your own practice session: five things interviewers say they look for when AI is allowed, each described as it shows up in a transcript at three levels. Use it to find the moments that helped or hurt. It is a practice rubric and not a validated score, because no published study has validated a transcript rubric against an independent measure of skill.
How should you use it?
After a practice round, read your session, or have your own agent read it with the prompt in the round's REVIEW.md, and for each dimension find two or three quoted moments as evidence. Mark each dimension strong, middling, weak or not enough evidence. Do not add the marks up: a total would suggest a precision the rubric does not have.
1 Breaking the problem down
Turning a loose brief into a plan you own, then handing the AI one well defined step at a time.
| Level | In the transcript |
|---|---|
| Strong | Reads the brief and the code before asking for anything. Lists the open decisions and settles them, in a plan the candidate wrote or edited. Each prompt asks for one step, and the result is checked before the next step starts. |
| Middling | A plan exists, but it is the agent's, accepted as it came. Steps are large, such as a whole feature in one go, and are reviewed afterwards rather than along the way. |
| Weak | The whole brief is pasted into one prompt and whatever comes back is accepted. Later prompts are versions of fix it. |
Why it is here. Rootly's reviewers read for problem decomposition first, and their weak example pasted the entire spec into one prompt. Canva's strongest candidates gave the AI well defined pieces of work. In the Microsoft study, issues solved incrementally succeeded 83% of the time against 38% for one shot delegation.
2 Context given to the AI
Supplying the facts the AI cannot guess: the constraint in a config file, the rule in a document, how the system is deployed, the helper that already exists.
| Level | In the transcript |
|---|---|
| Strong | Prompts name the files, constraints and rules that matter and say what done looks like. When the candidate learns something the AI missed, they pass it on in precise words. |
| Middling | Some context, mostly generic, such as make it production ready. The AI is left to find the important facts and sometimes does not. |
| Weak | No domain context. The AI guesses, and the one fact that mattered is never mentioned. |
Why it is here. Rootly reads whether prompts show domain understanding. In the Microsoft study, issues where developers supplied expert insight succeeded 64% of the time against 29% without. Anthropic's classifier of apparent expertise looks at how precisely users frame their directions.
3 Verifying output
Checking that what the AI produced is right before relying on it.
| Level | In the transcript |
|---|---|
| Strong | Runs the tests after each meaningful change and reads the diff before accepting it. Asks for evidence instead of taking claims on trust, and adds a test for the case that matters most, not only the cases the brief spelled out. |
| Middling | Runs the tests once at the end, and treats a green run as proof even where the tests do not cover the change. |
| Weak | Never runs anything, or accepts the AI's statement that it works. |
Why it is here. Verification is one of the four competencies Meta reportedly grades in its AI enabled round. Canva asks whether candidates can find and fix problems in AI generated code, and Google's pilot reportedly evaluates output validation. Anthropic's classifier looks at what users ask Claude to verify. One caution: in the Microsoft study, manual testing on its own was not linked to higher success.
4 Correcting the AI
Noticing when the AI goes wrong, stopping it, and steering it back precisely, or doing the small fix by hand.
| Level | In the transcript |
|---|---|
| Strong | Catches a wrong turn quickly, such as an edit that changes a test to make it pass, and rejects or interrupts it. Says exactly what is wrong and why. Makes one line fixes by hand. |
| Middling | Catches problems late, or corrects by starting again rather than saying precisely what to change. |
| Weak | Lets wrong changes through, loops on try again, or asks the AI to fix a one character bug. |
Why it is here. Rootly reads how quickly candidates catch and correct the AI, and treats repeated fix it prompts as a warning sign. Shopify's head of engineering called prompting for a trivial fix a warning sign. In the Microsoft study, writing some code by hand went with 73% success against 36%.
5 Tradeoff reasoning
Naming the options, choosing one for a reason tied to this system, and saying what was given up.
| Level | In the transcript |
|---|---|
| Strong | Names at least two options for each open decision, picks one with a reason specific to the code and its users, and writes the decision down. Can say what they would change with more time and before shipping. |
| Middling | Makes reasonable choices but does not say why, or lists options without choosing. |
| Weak | Lets the AI decide silently and cannot explain the choices afterwards. |
Why it is here. Superhuman's interviewers ask why candidates did what they did. Tolan's debrief covers constraints, tradeoffs, what the candidate would improve with more time and what they would change before shipping. Canva looks for sound technical decisions.
Where do the five dimensions come from?
From what employers say they read, and from the little research there is. Rootly, the one company we found that asks for AI transcripts with a take home, says its reviewers look for problem decomposition, course correction, domain understanding and iteration quality. Meta, according to Hello Interview, scores its AI enabled round on problem solving, code quality, verification and communication. Canva lists judgment about when to use AI, breaking down ambiguous requirements, sound decisions and fixing problems in AI generated code.
The research that links working styles to results is one study of 19 Microsoft developers working 33 real issues with an agent. Working incrementally, supplying expert knowledge, writing some code by hand and asking for refinements each went with markedly higher success. It is descriptive, from one company and an early 2025 model, and its authors found no significant link between success and years of experience.
Why is this not a score?
Because nobody has shown that a transcript score measures engineering skill. In the only professional sample we found, 15 developers studied by researchers at Accenture, a prompt score from an AI judge correlated 0.11 with years of experience. In a study of 100 university students in Zurich, human rated prompt quality correlated 0.479 with the outcome of the same session but also 0.353 with the students' essay grades, so a prompt score partly measures writing. Anthropic's study of about 400,000 Claude Code sessions found a clear gap between sessions rated novice and the rest, but only a modest one between intermediate and expert.
A transcript also misses much of what experienced engineers do: plans in files, rules files, edits made by hand, checks run outside the chat. Read it as evidence for a conversation about how you worked, not as a number.
Sources
- Rootly blog: Quentin Rousseau on take homes reviewed through AI transcripts
- Canva Engineering Blog: Simon Newton on AI in Canva's interviews
- Kumar and others: 19 Microsoft developers working real issues with an agent (arXiv 2506.12347)
- Anthropic: about 400,000 Claude Code sessions and the returns to expertise
- Hello Interview: guide to Meta's AI enabled coding round
- Moneywise on Yahoo Finance, reporting Business Insider: Google's AI assisted interview pilot
- The Pragmatic Engineer podcast: Shopify's Farhan Thawar on AI and hiring
- LeadDev: Speak, Superhuman and Stack Overflow on hiring for judgment
- Tolan: Dan Federman on how Tolan hires engineers
- Mehra and others: prompt scores of 15 professional developers (arXiv 2607.06074)
- Thorgeirsson, Weidmann and Su: what predicts vibe coding proficiency, 100 students (CHI 2026)
Last checked: .