Almost every week right now, someone asks what decisions, creative tasks or reporting they should be handing over to AI. Given how much has changed in the last few years, it’s absolutely the right question. If you’re not looking to automate or optimise your ecommerce business with AI, you’ll be left behind.
Most people start from the upside: how many hours does this save me, what does it give me back. That’s a fair place to start and a terrible place to decide from, because the upside keeps you looking only at what you get. The thing that actually sorts one job from another is the downside: what happens when it’s wrong, and more to the point, who finds out first, you or your customer.
The Day Autopilot Went Wrong
Justin Bausch sold artwork on Facebook Marketplace while still working in CRM at Ryderwear. He bought a printer, hired his nephew, and turned it into Style My Wall: seven figures in year one, one staff member to 22 inside two years. He’s since built an AI platform, running in Claude Code for almost eleven months, that handles his ads, his warehouse and his customer service. He’s also one of the very few people who’ll tell you about the day it went wrong.
For months, the AI had been handling damaged-frame returns automatically: reading the photo, deciding if it could be repaired, generating the return label, redrafting the order in Shopify. Then it started misreading the variant titles.
I identified customers getting the complete wrong order and sending out emails where it was identifying that they’re getting unframed, like an unframed rolled up poster versus a framed. All these customers that spend five times more than a poster, they’re just like, is this a scam? So I turned off that autopilot.
Justin Bausch, Founder & CEO, Style My Wall
He turned autopilot off, had his customer service team check every response by hand for a couple of weeks, and only switched it back on once they said out loud they were confident it was accurate.
In this article, we cover three things ecommerce operators can take into their business:
- Know the worst case before you automate anything, not just the upside
- Tell the difference between a bumpy rollout and a genuine showstopper
- Give every automation an owner, because flipping it to autopilot isn’t the end point
Know the Worst Case
Have a think about the thing you’d most like to hand over this month, and instead of finishing the sentence “this would save me,” finish this one: “if this gets it wrong, the worst that happens is.” Most of us never write that second sentence down.
There’s a scoring method for this called ICE: score each job out of ten on Impact, Confidence and Ease, then average the three. Run your list through it as it stands and it will hand you the customer-facing jobs first every time, because those carry the big impact numbers. Score a fourth thing alongside it: the worst case, who finds out first when it’s wrong, and what it costs to undo.
Justin had two automations running in the same months. His inventory scanner logged a frame wrong and a number was off; he found out himself on Monday. His customer service automation told paying customers they’d bought a poster, same builder, a much worse case, because it hit the customer directly.
Abel Butler runs HealthPost, and he’s drawn that same line through the middle of his own customer service.
We do really value being able to give people really appropriate natural health product advice, and we need to automate aspects of that very carefully rather than rush in.
Abel Butler, CEO, HealthPost
He’s comfortable automating order status straight out of Shopify, because the customer gets the right answer immediately. He stops dead at product and natural health advice. It’s the same customer service system, but the line has nothing to do with how hard the job is. It’s a risk and customer-experience decision.
Bumpy or Showstopper
Every automation is rough for the first few weeks. The skill is telling which kind of rough you’re looking at.
Bumpy is friction you push through: the output needs editing, it keeps missing context, it’s slower than doing it yourself for now. That’s a learning period. A showstopper is wrong in ways you can’t predict, or wrong without telling you, or reaches a customer before it reaches you. None of those improve with familiarity. That’s a design problem, not a settling-in problem.
Three questions, and any yes means showstopper: can it be wrong without me finding out; does being wrong reach a customer before it reaches me; when it’s wrong, can I tell you why. Justin’s poster bug is the clean example. The fix took no time once he knew. The problem was question two, it reached a customer before he did.
James Johnson from Shopify has a way of finding your bumps on purpose, before anything is riding on it.
Start with, why not AI? It’ll be painful and bumpy and you’ll learn what’s really crappy.
James Johnson, Director, Shopify
He calls the discoveries nuggets, and he’s just as firm the other way: plenty of workflows have AI sprinkled into them that never move the business impact. What he’s describing is a bumpy fortnight you enter deliberately, with nothing at stake but your own afternoon. Before switching anything on for real, write the pass mark down, not “we’ll see how it goes.” Justin ran three to four weeks of recommendations he checked by hand before his team said they were confident.
Somebody Owns It
The day you flip something to autopilot isn’t the end point. It’s the day the automation becomes part of the operating system. In the old world, a project has an owner while it’s being built, then gets handed to someone who manages it from then on. The same should apply to AI.
Emily Elvey has spent years inside customer experience teams and has the best framing on this.
Your customer service manager should effectively own looking after it like it looks after every other team member that you have in your business. That one doesn’t talk back.
Emily Elvey, Founder, Emily Elvey Consulting
She runs a weekly check-in on the agent’s results against the rest of the team: is it handling returns better than the humans, is it doing something slightly out of policy, does it need retraining. For every automation running in your business, you need three answers: who owns it, when do they look at it, and who can switch it off if that person’s on leave. If any of those are blank, you haven’t finished rolling it out. You’ve just let it loose.
The Takeaway
There’s no formula for best practice AI in ecommerce right now. Nobody can hand you a list of what’s safe to automate, because it depends on your customers, your margins and how much of a mistake your brand can absorb.
What you can do is make the call deliberately instead of by default: know the worst case before you start, know the difference between a rough patch and a real problem, and make sure every automation has an owner.
Frequently Asked Questions
How do you decide what tasks to automate with AI? Don’t just weigh the upside, the hours it saves you. Ask what happens when it’s wrong and who finds out first, you or your customer. Score potential jobs on Impact, Confidence and Ease (ICE), then add a fourth factor: the worst case and what it costs to undo, since customer-facing jobs often carry the biggest risk alongside the biggest reward.
What’s the difference between a “bumpy” AI rollout and a “showstopper”? Bumpy is friction that improves with training: the output needs editing, it’s slower than doing it yourself at first. A showstopper is wrong in ways you can’t predict, wrong without telling you, or reaches a customer before it reaches you. None of those improve with familiarity, because it’s a design problem, not a settling-in one.
How long should you test an AI automation before switching it on fully? There’s no fixed number, but set the bar before you start rather than deciding as you go. Style My Wall’s Justin Bausch ran three to four weeks of AI-generated customer service recommendations, manually checked by his team, before switching autopilot on, and again after switching it off to fix an error.
Who should own an AI automation once it’s live? The same person who’d own the equivalent human role, such as a customer service manager treating an AI agent like another team member. For every automation, you need three clear answers: who owns it, how often they check it, and who can switch it off if that person is unavailable.
