Foundation · 6 min
What you will learn
- The six kinds of provider change and how each one looks from inside your store
- What to record so you can tell what you actually depend on
- How to build a regression set from merchant tasks rather than demos
- How to keep the cost of switching providers low before you have to
Your store can change while your code does not
It arrives without a deployment. There is no release, no commit, and no ticket. One week the assistant describes a product one way and the next week differently, and nothing in your own history explains it.
So the investigation starts in the wrong place. Teams check their own prompts, their catalog and their integrations first, because those are the things they control. Hours go by before anyone reads the provider’s change notes, which is usually where the answer was.
The fix is to treat it as a dependency, not a product you bought. You would not run a payment integration without knowing its version and watching its deprecation notices. A hosted model is the same kind of thing, and it changes more often.
What a provider change can actually alter
| What the provider changed | How it shows up in your store |
|---|---|
| The model behind the same name | Different recommendations, a different tone, answers that used to be right |
| Safety or refusal behavior | The assistant declines questions it used to answer, or answers ones it used to decline |
| A version is retired | A feature stops working on a date you were told about months ago |
| Pricing or usage limits | The monthly bill moves, or the feature throttles at your busiest hour |
| The tool or API interface | Actions fail, structured output no longer parses, integrations break |
| Context or input limits | Long conversations or large catalogs silently lose part of what was sent |
The first row is the hardest to detect, because the name stays the same while the behavior moves. The third row is the one with a deadline attached, and it is usually announced long before anyone in a small team reads the notice.
Know what you depend on
Write down the provider, the model and version where it is exposed, the interfaces you call, the retrieval setup, the prompts and instructions, what data flows where, your rate limits, what you assumed about pricing, and the support and contract terms. This sits with the AI register described under governance rather than in a separate document.
The rule that matters: the record must not live only inside one person’s account. If the person who set it up is unavailable, everything above has to still be knowable, because a deprecation deadline will not wait for them to come back.
Test the tasks that matter, not the demo
Keep a small regression set built from real merchant and shopper tasks: product questions, comparisons, policy answers, any action the feature takes, structured output your systems parse, and how it refuses or hands off. Twenty cases chosen well beats two hundred generated for coverage.
Where the provider lets you reach both versions, run the set against each and compare before you cut over. Where it does not, run the set immediately after and have the result in hand before anyone starts guessing. Refusal behavior deserves its own cases, because a model that newly declines a routine product question fails in a way no accuracy test catches.
Keep the business truth outside the model
Product facts, permissions, transaction validation and business rules belong in your systems, as every page in this series says for its own reasons. Here the reason is portability: the more of your commerce logic lives inside prompts, the more a model change is a redesign rather than a swap.
Keep the model behind a stable internal interface, so the rest of your stack does not know which provider is answering. That one decision is most of what separates a migration measured in weeks from one measured in quarters.
Plan the exit before you need one
Watch deprecation notices and put the dates somewhere a person will see them. Decide in advance what rollback looks like where it is technically possible, which alternate provider or non-AI path covers each important workflow, and who revalidates quality, cost, privacy and contract terms after a change.
When a provider stops meeting your requirements on cost, behavior, privacy, or support, the options are to narrow the feature, replace the provider, or remove the feature. Having decided which of those applies to each use case, in advance, is the whole of this page.
Key takeaways
- A hosted model is a dependency that changes on someone else’s schedule.
- Read the provider’s change notes before investigating your own code.
- Record the version and the setup somewhere that is not one person’s account.
- Regression-test real merchant tasks, including refusal and handoff behavior.
- Keep business rules outside the model, and keep a fallback that still works.