Visa Just Graded the AI Shoppers: 22 Models Fail 42.8% of Runs and Still Buy Anyway
TL;DR
Visa’s new Agentic Commerce Benchmark finds that AI shopping agents passed just 57.2% of test runs across 22 models. More concerning, about 54% of failed runs still ended in a purchase. The result is not another agentic protocol or partnership story: it is a quality-control warning for operators. Before unsupervised agentic payments scale, businesses need to test reliability across repeated runs, improve machine-readable product information and keep delegated spending within clear limits.
What did Visa’s benchmark test, and what did it find?
Visa’s benchmark tests whether an AI agent can discover a shopper’s requirements through conversation and make the right buy-or-no-buy decision. Visa launched the Visa Agentic Commerce Benchmark on 5 November 2026, describing it as the first agentic commerce benchmark to require both.
The first round covered 22 AI models from 12 labs and 15 shopping challenges, including buying a TV, laptop or personal care appliance. Shoppers did not state every requirement upfront, so each agent had to ask questions, uncover relevant details and decide whether a suitable product existed. If nothing fit, the correct result was not to buy. The agent also had to uncover enough information to justify its decision, rather than succeed by luck.
Overall, models passed 57.2% of runs and failed 42.8%. The best model passed 82.2% of attempts; the worst passed 28.9%. Visa did not name the models in its release. Fortune reported that Google and Meta agents shared the top score at 82%, a Claude agent placed third at 76%, and an OpenAI agent ranked sixth at 64%.
That is the reliability story. Recent coverage has examined Walmart’s merchant control plane for agent shopping, Mastercard’s OpenAI and Tencent moves, and Block’s merchant-side agentic operations. This benchmark asks a different question: can AI shopping agents consistently understand what people want before they buy?
Why is “failed, but bought anyway” the critical finding?
Because an agent that misses a requirement and still completes a purchase can turn a discovery failure into a merchant’s fulfilment, returns and dispute problem.
About 54% of failed runs nevertheless ended in a purchase. In those cases, the agent either recommended something that did not meet the customer’s requirements or bought before uncovering all of them. That is not simply an unsuccessful conversation. It is a potentially wrong-fit order, made without the customer having clearly stated every constraint.
Visa’s “Superintelligence Sweep” example shows the challenge. A customer asks for help choosing a television, mentions a brand recommended by a relative and a third-party supplier membership, but leaves other requirements unsaid. The agent must draw out details including budget, format compatibility, HDR10+ support for family videos and gaming capabilities.
If the agent misses a requirement and buys anyway, the merchant may face a dissatisfied buyer, a return or a dispute. This does not make every wrong-fit order a payment dispute, but it does create a foreseeable source of friction and confusion about what the customer intended and what the agent was authorised to do.
That matters for dispute operations. Visa has recently expanded its Compelling Evidence 3.0 rules for qualifying card-absent fraud disputes. Agentic purchases introduce a distinct evidence question: what did the customer ask for, which requirements did the agent uncover, and what did the customer authorise? The agent is not the cardholder; operators will need clear records of the human’s intent and the agent’s actions.
What do the results say about consistency and product data?
The benchmark suggests reliability depends on both asking good questions and delivering repeatable results; neither can be assumed from a successful demo.
The five models that uncovered the most hidden buying factors passed 70.2% of runs. The five that uncovered the fewest passed 44.4%. That gap makes conversation quality a measurable part of agent performance. A capable shopping agent must interrogate the shopper when the brief is incomplete.
Merchants therefore have a role in making answers available. Product catalogues need structured, machine-readable attributes that can address questions about compatibility, specifications, restrictions, availability and other relevant details. If an agent cannot interpret a product page or retrieve a crucial attribute, that product may be harder to recommend, even if it is suitable. Mastercard has also said merchants are “slightly behind” on agent discoverability.
Consistency is a further weakness: all three attempts passed only 34.2% of the time across models and scenarios. Some models that passed a scenario once failed it when they repeated it. For an operator, a single impressive demonstration is not evidence of dependable performance.
Visa’s Trust Index, released in September 2026, provides a useful context: 72% of US consumers had used an AI assistant, but only 23% trusted AI to handle payment transactions on their behalf. Visa’s Rubail Birwadker, Global Head of Growth, said: “AI agents have the potential to power the next era of commerce, but trust is key to making it work.” He told Fortune, “We’re not there yet from a trust or technology perspective. But we’ll soon get into a world where consumers can trust and delegate.”
What should payments operators do before agentic checkout scales?
Operators should treat agentic commerce as a quality-control challenge as well as a payments-integration opportunity.
Visa plans to add shopping scenarios and product categories, then extend testing to checkout, payment and order placement. That next phase will bring agent reliability into direct contact with acceptance economics. For now, the benchmark offers a practical starting point for fintechs, POS companies, PayFacs, ISOs and issuers:
Test repeated runs, not one-off demos. If you are piloting agentic checkout, measure how often an agent succeeds across repeat attempts and realistic shopper conversations.
Prepare for wrong-purchase disputes. Review whether your evidence workflows can connect the customer’s requirements, agent decisions, authorisation and final order.
Audit your product data. Check whether structured attributes, stock information, compatibility details and restrictions can answer an agent’s clarifying questions.
Keep delegated spending gated. Use limits, scopes and agent profiles until reliability in your product category is meaningfully stronger. Walmart’s Agent Profiles approach is one example of the control-plane discussion.
Track the benchmark’s checkout extension. Payment completion is where reliability meets acceptance, authorisation and the commercial cost of failed attempts.
Cost adds another operational question. The five most expensive models per run passed 67.1% of attempts, compared with 50.2% for the five cheapest. That correlation does not guarantee a more expensive model will perform better in every use case. But for anyone subsidising agent shopping (an issuer, wallet or merchant) the relevant metric is cost per successful purchase, not simply cost per model run.
There is also a strategic signal in Visa publishing a public benchmark and inviting people to try it, watch replays and read the whitepaper on Visa Payments Frontier. Visa shared it with leading model providers for feedback, and says the research is complementary, not directly part of the Visa Intelligent Commerce product portfolio. This is a quality and trust initiative, not a product announcement. It echoes the network’s Know Your Agent (KYA) trust-layer work: KYA addresses who an agent is and what authority it has; this benchmark addresses whether it can act competently. By defining a public scoreboard, Visa is helping shape what “good” looks like, and giving consumers a way to judge the technology for themselves.
Frequently asked questions
What is the Visa Agentic Commerce Benchmark?
It is a public test of AI shopping reliability. Agents must discover customer requirements through conversation and then make the correct purchase or no-purchase decision.
How reliable are AI shopping agents right now?
In Visa’s first round, 22 models passed 57.2% of runs and failed 42.8%. All three attempts passed only 34.2% of the time across models and scenarios.
What happens when an AI agent buys the wrong thing?
The merchant may face a wrong-fit order, return or customer complaint. If the purchase leads to a payment dispute, records of the customer’s intent, the agent’s questions and the final order may become important evidence.
Can I try the benchmark myself?
Yes. Visa has made the benchmark public, with opportunities to try it, watch replays of AI attempts and read the whitepaper through Visa Payments Frontier.
Key takeaway
Visa’s benchmark shows that AI shopping agents are not yet consistently ready to buy unsupervised: they fail many runs, vary across repeated attempts and sometimes purchase despite failing. For payments operators, the priority is not to rush integration. It is to build measurable quality assurance, machine-readable product information, dispute-ready evidence and spending guardrails before delegated purchasing becomes routine.