'Build It Ourselves': The Hidden Cost of DIY Assessment Tools

The visible cost is the one that makes it onto the slide: a few sprints of front end, a recruiter panel, a backend, an integration with the ATS. Doable. Your engineers really are good.

The trouble is that in-house build estimates have the emotional accuracy of a holiday weather forecast - sunny, confident, and usually wrong by a factor of three. But let's be generous and assume you land it on time. You have still only paid for the tip.

A platform with no content is an empty box with a login screen.

Someone has to write the questions, and not just any questions. For a test to measure anything at all you need hundreds of items per role, spread across difficulty levels, and calibrated well enough to separate a strong candidate from a confident one. In cybersecurity they have to come from a practitioner, not from someone who skimmed an OWASP page on Tuesday.

Then comes the part nobody budgets. This is not a one-time job. Languages ship new versions, fresh CVEs land every week, and a framework that was mandatory in 2023 is legacy by 2025. A question bank is a garden, not a monument. Stop weeding it for two quarters and it quietly goes stale - and a stale test is worse than no test, because it hands you false confidence with a number attached.

So ask the honest question: who inside your company writes, and then rewrites forever, expert-level questions on penetration testing or Kubernetes hardening? It is almost always the same senior engineer whose time is the scarcest thing you own. This, incidentally, is the single biggest reason vendors exist at all: a platform like cp.center ships a question bank with quarterly updates included, so that nobody on your team spends Friday afternoons pruning that garden.

This is the layer that should keep a CTO awake, and it is also the one most often miscounted - so let's be precise, because the sloppy version of this argument is easy to dismiss.

The hours your engineers spend interviewing candidates are not a cost of building. You pay those whether you build, buy, or do nothing. That is the problem you are trying to solve, not the price of solving it.

The real bill is what building adds on top. A credible internal platform is not a weekend project: two or three engineers for a quarter is roughly a thousand to fifteen hundred engineering hours before anyone logs in. Then comes the part with no end date - patching the code-execution sandbox, fixing scoring edge cases, keeping the thing upright during a hiring spike. Budget somewhere between ten and twenty percent of an engineer's year, every year, forever. Those are not exotic numbers. They are what any long-lived internal tool costs.

And here is the trap: you pay that engineering bill and, for the first year or two, you keep paying most of the interviewing bill too, because a half-finished internal tool rarely screens well enough for anyone to trust it. You have bought the cost without yet buying the relief.

There is a softer cost underneath the hard one. Engineers who joined to build your product do not stay motivated maintaining an internal side project that turns out to be endless plumbing with no glory. "You didn't hire me to babysit a quiz engine" is a sentence that gets said in a one-on-one shortly before someone refreshes their LinkedIn.

The most expensive line item never appears in any ledger. It is the release that did not ship, because three of your best people spent the quarter keeping the internal test tool alive.

Executing a candidate's code on your own infrastructure means deliberately inviting people you have never met to run arbitrary programs inside your perimeter. That is not a bug in the design; it is the entire feature. Which is exactly why the security problem is subtler than it first looks.

The risk is not "remote code execution" - you are providing remote code execution on purpose. The risk is what happens next: escaping the container, reaching something on the internal network that should never have been reachable, quietly mining cryptocurrency on your bill, or exfiltrating whatever the sandbox could see. Getting isolation, egress rules, and resource limits right is a security discipline in its own right, and it is the kind of thing that shows up in an incident report when it is almost right.

Then there is the arms race, and it deserves an honest sentence rather than a marketing one. Questions leak. Candidates trade answers in group chats. And no amount of question randomization fully solves an AI assistant open in a second window - anyone who tells you otherwise is selling something. What randomization, per-question time limits, and hands-on lab tasks actually do is shift the test toward things that are harder to outsource to a chatbot and easier to verify. That shift is continuous work. It is somebody's job, permanently. The only real question is whether it is your job or your vendor's.

Whoever built it is the only person who truly understands it. When that person switches teams, takes leave in the middle of hiring for thirty roles, or simply resigns, the platform becomes a black box nobody wants to touch.

Every internal tool begins life as a point of pride and ends as a line in the technical-debt register that nobody wants to open. And there is a subtler cost still: a home-grown test measures your candidates against the only benchmark it has, which is itself. You never find out whether your "senior" is a market senior or merely a senior by local standards.

That has a sharper edge than it first appears. A scoring threshold nobody has ever validated still rejects real people, and those decisions have to survive being questioned. When a candidate asks why they failed, "our internal tool scored them 61" is not an answer - and the burden of showing that the 61 meant anything at all sits with whoever built it.

If the argument only ever runs one way, it is marketing rather than analysis - so here is the other side, honestly.

Building makes sense when assessment is the product, or a real part of it: if you sell certification, run a training business, or your evaluation method is itself a competitive edge, outsourcing it would be strange. It makes sense when your process is genuinely idiosyncratic in a way that matters commercially, and no vendor covers it. It makes sense when you have constraints - data residency, air-gapped environments, a regulator with strong opinions - that no vendor will meet, and enough hiring volume to amortise the build. And it makes sense if you already own the hard part: a team whose actual job is producing and validating assessment content.

Buying is not free either, and pretending otherwise would be the same trick in reverse. You pay a licence. You pay for integration and, more painfully, for adoption - a tool your recruiters quietly stop using is a total loss regardless of price. You take on vendor risk, and you should ask about the exit before you ask about the discount.

The honest version of this decision is not "buy good, build bad." It is that both options have an iceberg, and only one of them routinely gets estimated with the underwater part left off.

In Enterprise Deals: Every One We Lost, a candid post-mortem of his own losses, Tomasz Karwatka gives salespeople one piece of advice: name your rival. The most dangerous rival, he notes, is rarely a competing vendor. It is "we'll build it ourselves" and "let's just leave things as they are."

Read that from the other side of the table. From the buyer's seat, building in-house looks like the safest and cheapest option available - precisely because it arrives without a price tag. That missing price tag is the whole trick. An option with no visible cost never gets challenged in the room, so it wins arguments it has no business winning, including the internal ones.

The absence of a price is not the absence of a cost. It is a cost that arrives later, denominated in your engineers' time.

Build what is genuinely your competitive advantage. For a bank, a staffing agency, or a scale-up, a technical assessment platform is almost never it. It is infrastructure, not product - and nobody builds a power station to turn the office lights on.

Whether you end up choosing a vendor like cp.center or a well-argued internal build, the discipline is the same and it takes an afternoon: put a number on the content, the build, the maintenance, the sandbox, and the person who becomes the single point of failure. Then compare it with a quote.

Before the next "we'll just build it ourselves," do that one thing. Count the part under the water. Most of the time, the number ends the debate on its own.

Source: Tomasz Karwatka, Enterprise Deals: Every One We Lost - a post-mortem of lost enterprise deals, in which the author argues such deals are lost in the first two meetings and advises naming the real rival: "build in-house," "big software," or "no change." Published on the author's Substack.