Should you redesign the technical hiring process to allow AI use?

Chinese companies are nearly twice as likely as their U.S. counterparts to allow AI in live technical interviews, and they've largely moved away from automated code tests and take-home projects in favor of in-person sessions that observe how candidates think and work. That's from Karat's 2026 AI Workforce Transformation Report, drawn from 400 engineering leaders across the U.S., India, and China. As Karat CEO Mohit Bhende explains it: "They're here to win the next 100 years, and build great products to do all of that." While U.S. organizations are still running internal permission debates, Chinese firms have moved on to the harder question: once AI is in the room, how do you actually redesign the interview to generate useful signals?

Allowing AI use in an assessment changes the surface of the interview while leaving the underlying problem intact if the questions, format, and interviewer behavior all stay the same.

Why is the traditional technical assessment losing signal?

The assessment formats that defined technical hiring for the past decade are producing weaker predictive signals, and the pace of degradation has accelerated sharply. Karat's survey of 400 engineering leaders found that 71% now say AI is making it harder to assess candidates' technical skills, up from roughly 20% to 30% just a few years ago. That jump reflects something structural.

Bhende frames it directly: "When we think about why it is so hard to assess talent today in this AI world, many popular theories are that it must be the cheating, or it must be X. It's actually not that. It's because the fundamental job has changed, but technical interviews, as we know them, have not."

The asynchronous code test was never a perfect proxy for on-the-job performance. It was standardized, scalable, and consistent, which made it useful as a filter even when its predictive validity was modest. A CoderPad 2025 hiring survey found that 54% of developers cite lack of relevance to actual job roles as their top complaint about coding assessments. The signal gap has always existed; AI has widened it to the point where the cost of inaction is now measurable in bad hires.

What has replaced the take-home test?

Asynchronous formats are declining across the board. Today, 63% of U.S. employers still use automated code tests and 45% still use take-home projects, according to Karat's data, and confidence in both is eroding fast. Return-to-office trends reduce the practical case for take-home tests. AI assistance makes unsupervised assessments difficult to interpret. And as Bhende puts it: "If the test is to test your ability to code and the fundamental skills of an engineer are changing, then you've got the wrong test."

What has gained ground is the live coding session. Kareem Osman, VP at Robert Half, describes the shift in New York's market plainly: "Instead, it's all live coding interviews with other engineers on the team who assess them." During these sessions, teams present real scenarios, respond in real time to what a candidate builds, and introduce new constraints mid-session that demand judgment, adaptability, and architectural thinking. The live format costs more time and more senior attention, and that investment is worth making because proximity to real working conditions is where predictive signal lives.

China's adoption of this format has been faster and more deliberate. Karat's data shows Chinese companies are significantly more likely to have shifted to live sessions and to allow AI tools within them, treating the live format as the default rather than the exception.

What does "allowing AI" actually require?

Simply permitting AI use in an interview doesn't produce better signal on its own. The companies getting this right have redesigned the entire evaluation framework, and the detail work matters as much as the policy change.

Three companies illustrate what the redesign looks like in practice, each with a different approach and the same underlying principle.

Canva published its methodology in full on its engineering blog in June 2025. The company replaced its CS fundamentals screening entirely after discovering that AI tools produced correct, well-documented solutions to its existing questions in seconds. The new format expects candidates to use Copilot, Cursor, or Claude, and deliberately presents problems that "can't be solved with a single prompt; they require iterative thinking, requirement clarification, and good decision-making." Interviewers stop after each AI-generated code block to ask what it does. Accepting output without understanding it is a disqualifying signal.

Meta took a narrower approach. Its AI-enabled coding round, launched in late 2025, uses a CoderPad environment with a built-in AI chat panel. The evaluation focuses on whether candidates can catch errors the model introduces, drive the solution toward correctness, and handle follow-up requirements that the AI can't anticipate. The round replaced one of two traditional coding rounds in the onsite loop.

Shopify goes furthest in replicating real work conditions. Candidates receive an empty GitHub repo, bring their own IDE and AI tools, and build from scratch over screen share, no starter code, no guardrails, no sandboxed environment. According to candidates who have gone through the process, the interviewer watches for design instincts, project organization, and whether the candidate is directing the AI.

Bhende's framing captures the requirement precisely: "If all a company does is allow AI to be used in technical interviews, they have not really changed anything. You have to create new interview questions and retrain interviewers." That means three things in practice: revisit which skills the role actually requires; redesign assessment questions around real-world simulation and multi-step problems that demand genuine judgment even with AI assistance; and retrain interviewers to work collaboratively with candidates, evaluating the reasoning process rather than scoring outputs from a distance.

"Allowing AI is a new process, and it requires a holistic redesign of your interview framework and approach to doing assessments," Bhende says. The companies treating it as a permission toggle will keep producing weak signals. The companies treating it as a redesign trigger are building something fundamentally more predictive.

What does this reveal about how organizations measure performance?

The live interview and the in-the-moment coaching conversation are solving the same problem. Real work produces real signals. Controlled environments optimized for consistency trade validity for comparability, and that tradeoff compounds across the talent lifecycle, from the first assessment to every development interaction after it.

Karat's report finds that the ROI of a strong engineer is expected to triple over the next three years as AI amplifies the productivity gap between high and average performers. Organizations that align their measurement infrastructure to actual work conditions will see that return.

‍