Microsoft ThinkingBox tests what AI agents actually change, and whether they can repeat a success
Microsoft and Hugging Face present an open evaluation environment with 507 business tasks, each repeated 20 times. It checks final records and unwanted changes, separating occasional success from consistent completion.
Microsoft and Hugging Face published a joint article on October 3, or October 4 in Shanghai, introducing ThinkingBox through Hugging Face. Agents perform business tasks in isolated tool environments, then executable checks inspect final database states and side effects. One example closes an unresolved delivery ticket, despite a plausible sequence of tool calls.
ThinkingBox-Bench has 507 tasks across retail, auto insurance, travel, digital banking and consulting, each repeated 20 times from a clean state. Single-attempt success, at-least-once success and success on all 20 attempts measure different things. The reported Kimi-K3 results illustrate the gap: 93.89% of tasks succeeded at least once, but 13.41% passed every attempt. These benchmark results are not production success rates.
The open framework and benchmark let developers repeat tests with their own models. Public tasks reconstruct enterprise workflows synthetically rather than containing real customer records. They help teams inspect correct changes, unwanted effects and repeatability. This site has not independently rerun the evaluation.
Sources
- The Agent Said It Was Done. The Database Disagreed. — Microsoft / Hugging Face
- microsoft/thinkingbox — Microsoft / GitHub