How to Evaluate Ecommerce AI Vendors and Avoid Lock-In

Foundation · 7 min

What you will learn

Why a demo is not evidence, and what to ask for instead
Eight dimensions to evaluate, each with the question that finds the weakness
The costs that are not on the price list
What actually creates lock-in, and how to avoid it before signing

A demo is not evidence

Demos are built to work. The products, the questions, and the edge cases were chosen by someone who knew where the system performs well. That is not dishonest; it is what a demo is. It just tells you nothing about your catalog.

So bring your own material. Your real products, including the ones with thin attributes. Your actual policies. The questions your support team gets on a bad day. Define before you start what counts as correct, what counts as useful and what you would call unacceptable.

And if they will not let you, that is the finding. A vendor confident in their system wants it tested on real material. One who insists on their own set is telling you where it performs best, which is useful information arriving in an unhelpful form.

Before any of that, write the requirement down. “We need AI” is not testable. “Cut the time spent classifying support tickets without increasing routing errors” is, and it gives every vendor the same thing to be measured against.

Eight dimensions, and what to ask

Use the same list for every vendor. The third column is the one that does the work, because it asks about failure rather than capability.

Dimension	Evidence to ask for	The question that finds the weakness
Business fit	A run against your own scenarios	Does it beat what we do today?
Quality	Their evaluation results, and real error examples	What does it do when it is unsure?
Data and privacy	Data flow, retention, sub-processors, deletion	Can we get our data out and have it erased?
Security	Access model, audit logs, incident process	How would we know if something went wrong?
Integration	Interfaces, limits, versioning policy	What breaks when you change the interface?
Operations	Availability terms, support, change notices	How do we recover when you are degraded?
Cost	Pricing model plus your own ops estimate	What drives the bill at ten times this volume?
Portability	Export paths, standard interfaces	How hard is it to leave?

The quality row deserves particular attention. Ask to see real error examples rather than accuracy figures. A vendor who cannot show you what their system gets wrong either has not looked or would rather you did not.

The cost that is not on the price list

The usage charge is the visible part and often the smaller one. Budget for implementation and integration, the data preparation the feature depends on, monitoring, evaluation, the human review time it adds, support, and the ongoing maintenance of prompts and retrieval as your catalog moves.

A low per-call price can still produce an expensive feature. If every output need ten minutes of checking, that review time is the real cost and it scales with volume in a way the price list does not show. Ask what drives the bill at ten times your current usage, because the answer is often not the thing you were quoted on.

What actually creates lock-in

Rarely the contract. It is your authoritative product and customer data living only in their system, your prompts and business rules existing nowhere you control, and your workflows built directly against their specific interface. Each of those is a decision you make early, cheaply, and unmake later at considerable expense.

So keep source data in your own systems. Keep prompts, policies, evaluation cases and integration logic somewhere portable. Prefer documented interfaces with export paths. Then write down, honestly, which parts are genuinely proprietary and roughly what replacing them would take. That estimate is the real measure of your dependency.

Read the contract as an operating dependency

Check what happens when they change the service or substitute the model behind it, what rights they have over your data and over the output, how liability is allocated, what security they actually commit to, and the terms on termination, data return, deletion and notice periods.

Two patterns are worth naming. If the quality is good but the data terms fail your requirements, reject or renegotiate rather than proceeding and intending to revisit, because your leverage is highest now. And if a provider is strong, but replacement would be genuinely difficult, reduce the dependency before you scale the usage, not after.

Key takeaways

Write the requirement before you look at vendors. “We need AI” cannot be tested.
Test on your own products, policies and awkward questions, with the same tasks for each vendor.
Ask what the system does when it is unsure, and ask to see real errors.
The review time a feature adds is usually a bigger cost than its price list.
Lock-in comes from where your data, rules and workflows live. Decide that before signing.