16 candidates. One cleared the bar.

An enterprise software company needed four engineers who had taken AI agents to production. The market was full of people who had read about it.

Get the full study ↓

The situation

A new practice, four seats.

Forward-deployed engineers sit with customers and ship agent systems that stay live. For five weeks we reviewed the sourcing and ran the first technical screen.

16 candidates evaluated in five weeks
8 of 11 resumes we read had the job description pasted back in
0 of 3 early candidates passed the hiring manager's round
1 cleared our technical screen and moved to the final round

The failure mode

Resumes started answering the questions back.

The first three candidates reached the hiring manager and were rejected there. Then our own job description started turning up in resumes, pasted into unrelated jobs.

Where 16 candidates ended up

One filled circle out of 16.

3 Rejected by the hiring manager
6 Rejected on resume + written answers
1 Rejected at our technical interview
2 On hold: claims not verifiable
3 Screen offered, outcome open
1 Cleared our screen, sent to final round

Every open circle cost a resume read, a call, or an interview slot.

Resume forensics, 11 resumes read line by line

8 of 11 resumes quoted our own job description back to us.

Job-description language pasted into unrelated roles 8 / 11
Tools or rules claimed before they existed 5 / 11
Bullets copied across roles or industries 4 / 11
Hedged verbs: “-style”, “transferable to”, “concepts” 3 / 11
Not a single number in the whole resume 2 / 11

Tells overlap: 7 of the 11 resumes showed two or more.

Claimed before it existed

The job ended. Then the technology shipped.

Redux Toolkit claimed in a front-end role, 2016–2018 · released Oct 2019
Next.js middleware claimed in a SaaS role, 2019–2020 · released Oct 2021
A 2024 industry regulation claimed in a public-sector role, 2020–Jan 2023 · rule final Jan 2024
MCP tool servers claimed in an insurer role, Oct 2021–Oct 2024 · released Nov 2024

Four examples from three different resumes. None needed a reference check to catch.

The bar

Ownership, not AI knowledge.

About a third of the final interview was about the data under the agent. Another third was about the evaluation set that guards it. Only someone who built the system end to end can answer both.

What the final interview actually asks

It tests ownership, not AI vocabulary.

≈⅓≈⅓≈⅓

The data under the agent

How many tables? Which one matters most?
Its keys and fields, and how the agent uses them
Which data comes from outside, which you generate

The evaluation set

How it was built, and by whom
How big, and what one case contains
Pass/fail rules, and why it's enough to ship

Agent, model, metrics, case study

How many agents are live, which model version and why
Precision, recall, F1: the formulas
Live: size a manual-review sample, set a monitoring cadence

Two-thirds of the questions can only be answered by someone who built the system end to end.

What we changed

A written questionnaire, then a ten-minute screen.

The screening questions had leaked, so we stopped asking them as written. We rebuilt the screen around the hiring manager’s own question list. A 24-question questionnaire goes straight to the candidate. A ten-minute live screen follows: five questions, scored against the final round.

The result

Sixteen reviewed. One recommended.

One candidate cleared our screen and went to the final round. That candidate had hands-on production agent work and no background in the client’s industry.