What DIY Can Teach Us About AI’s Real Business Skills — Not Just Chat

Imagine crafting a wooden project. You might have the best tools and a detailed plan, but the real test is whether you finish the job, stay honest, and handle surprises under pressure. The same applies to AI in business—it’s not just about how well it can chat; it’s whether it can get the work done when stakes are high.

AI Models Face the Same Business Crises, But Only Some Finish the Job

In a recent experiment by Firmulate, four of the latest frontier AI models were put through a simulated week of running a small software company. The goal? To see if these models could manage real-world crises, resist manipulation, and, crucially, close a deal worth €55,000 based on their own analysis.

Each model was given identical scenarios—customers with issues, potential scams, and conflicting demands—and the decision-making process was fully auditable and transparent. All four models could identify every crisis, and all refused every attempt at manipulation, such as fake CEO messages or reporter tricks. These are the skills that chat demos often showcase: understanding and resisting manipulation.

But here’s where the real business test begins. When it came to closing the deal, only two models succeeded in signing their own analysis and sealing the agreement. The other two, despite diagnosing the problems correctly and proposing the same solutions, failed to follow through and left the deal unexecuted. The gap? It wasn’t in knowledge, but in discipline and execution—a vital, invisible skill that chat-based demos rarely measure.

The Hidden Weakness in AI’s Business Performance

Digging deeper, the decisive difference lay in reading and acting on critical internal documents. The models that closed the deal had uncovered a key fact buried two documents deep in the company’s own files—information that, if ignored, cost them the full revenue potential, estimated at over €4,500 in monthly recurring revenue (MRR). The models that failed to act on this information left money on the table, despite no issues in crisis detection or manipulation resistance.

This underscores a fundamental lesson: assessing AI’s business readiness isn’t about how well it can generate convincing chat responses or handle simple queries. It’s about whether it can read context, prioritize information, and follow through on the insights—skills that require discipline and a focus on execution.

Beyond Chat: Measuring the True Business Value of AI

Many companies evaluate AI capabilities based on conversational demos or superficial benchmarks. But as the experiment shows, those tests miss the critical skills that determine real success—such as integrity under pressure, attention to detail, and persistence in closing deals.

The experiment’s results reveal that even the most thorough model—Opus 4.8, with over 80 learned rules and deep analysis—still left the closing on the table. Its discipline slipped in the heat of the moment, illustrating that thoroughness alone doesn’t guarantee execution. Meanwhile, the more disciplined Kimi K3 closed the deal cleanly, showing that consistency matters as much as knowledge.

The Lessons for Business Owners and AI Developers

• AI’s true business capability is measured by whether it can finish what it starts, not just generate convincing chat.
• Reading internal documents and acting on hidden information can be the difference-maker in closing deals and capturing value.
• Resisting manipulation is a baseline; execution discipline is the real skill that takes AI from a demo to a business partner.
• Testing AI in realistic, auditable scenarios reveals strengths and vulnerabilities that chat demos overlook.

For entrepreneurs and managers, this experiment underscores the importance of testing AI in a controlled environment that mimics real business pressures. Firmulate offers such simulations—running your company’s decision-making in a watchable, read-only wargame that reveals whether your AI workforce can handle your toughest days.

Visit firmulate.com to explore how this digital twin approach can help you identify and build the skills your AI tools need to succeed in real business settings.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI