Ep 668 · 20 min · Fri 2 Oct 2026

How to Decide What You Let AI Run

Justin Bausch

Almost every week someone asks what tasks they should hand to AI. The better question is what happens when it goes wrong, and who finds out first.

Play episode 668

In this episode

Almost every week right now, someone asks what decisions, creative tasks or reporting they should be handing over to AI. Given how much has changed in the last few years, it’s absolutely the right question. If you’re not looking to automate or optimise your ecommerce business with AI, you’ll be left behind.

Most people start from the upside: how many hours does this save me, what does it give me back. That’s a fair place to start and a terrible place to decide from, because the upside keeps you looking only at what you get. The thing that actually sorts one job from another is the downside: what happens when it’s wrong, and more to the point, who finds out first, you or your customer.


The Day Autopilot Went Wrong

Justin Bausch sold artwork on Facebook Marketplace while still working in CRM at Ryderwear. He bought a printer, hired his nephew, and turned it into Style My Wall: seven figures in year one, one staff member to 22 inside two years. He’s since built an AI platform, running in Claude Code for almost eleven months, that handles his ads, his warehouse and his customer service. He’s also one of the very few people who’ll tell you about the day it went wrong.

For months, the AI had been handling damaged-frame returns automatically: reading the photo, deciding if it could be repaired, generating the return label, redrafting the order in Shopify. Then it started misreading the variant titles.

Justin Bausch, Founder & CEO, Style My Wall

I identified customers getting the complete wrong order and sending out emails where it was identifying that they’re getting unframed, like an unframed rolled up poster versus a framed. All these customers that spend five times more than a poster, they’re just like, is this a scam? So I turned off that autopilot.

Justin Bausch, Founder & CEO, Style My Wall

He turned autopilot off, had his customer service team check every response by hand for a couple of weeks, and only switched it back on once they said out loud they were confident it was accurate.

In this article, we cover three things ecommerce operators can take into their business:

  • Know the worst case before you automate anything, not just the upside
  • Tell the difference between a bumpy rollout and a genuine showstopper
  • Give every automation an owner, because flipping it to autopilot isn’t the end point

Know the Worst Case

Have a think about the thing you’d most like to hand over this month, and instead of finishing the sentence “this would save me,” finish this one: “if this gets it wrong, the worst that happens is.” Most of us never write that second sentence down.

There’s a scoring method for this called ICE: score each job out of ten on Impact, Confidence and Ease, then average the three. Run your list through it as it stands and it will hand you the customer-facing jobs first every time, because those carry the big impact numbers. Score a fourth thing alongside it: the worst case, who finds out first when it’s wrong, and what it costs to undo.

Justin had two automations running in the same months. His inventory scanner logged a frame wrong and a number was off; he found out himself on Monday. His customer service automation told paying customers they’d bought a poster, same builder, a much worse case, because it hit the customer directly.

Abel Butler runs HealthPost, and he’s drawn that same line through the middle of his own customer service.

Abel Butler, CEO, HealthPost

We do really value being able to give people really appropriate natural health product advice, and we need to automate aspects of that very carefully rather than rush in.

Abel Butler, CEO, HealthPost

He’s comfortable automating order status straight out of Shopify, because the customer gets the right answer immediately. He stops dead at product and natural health advice. It’s the same customer service system, but the line has nothing to do with how hard the job is. It’s a risk and customer-experience decision.


Bumpy or Showstopper

Every automation is rough for the first few weeks. The skill is telling which kind of rough you’re looking at.

Bumpy is friction you push through: the output needs editing, it keeps missing context, it’s slower than doing it yourself for now. That’s a learning period. A showstopper is wrong in ways you can’t predict, or wrong without telling you, or reaches a customer before it reaches you. None of those improve with familiarity. That’s a design problem, not a settling-in problem.

Three questions, and any yes means showstopper: can it be wrong without me finding out; does being wrong reach a customer before it reaches me; when it’s wrong, can I tell you why. Justin’s poster bug is the clean example. The fix took no time once he knew. The problem was question two, it reached a customer before he did.

James Johnson from Shopify has a way of finding your bumps on purpose, before anything is riding on it.

James Johnson, Director, Shopify

Start with, why not AI? It’ll be painful and bumpy and you’ll learn what’s really crappy.

James Johnson, Director, Shopify

He calls the discoveries nuggets, and he’s just as firm the other way: plenty of workflows have AI sprinkled into them that never move the business impact. What he’s describing is a bumpy fortnight you enter deliberately, with nothing at stake but your own afternoon. Before switching anything on for real, write the pass mark down, not “we’ll see how it goes.” Justin ran three to four weeks of recommendations he checked by hand before his team said they were confident.


Somebody Owns It

The day you flip something to autopilot isn’t the end point. It’s the day the automation becomes part of the operating system. In the old world, a project has an owner while it’s being built, then gets handed to someone who manages it from then on. The same should apply to AI.

Emily Elvey has spent years inside customer experience teams and has the best framing on this.

Your customer service manager should effectively own looking after it like it looks after every other team member that you have in your business. That one doesn’t talk back.

Emily Elvey, Founder, Emily Elvey Consulting

She runs a weekly check-in on the agent’s results against the rest of the team: is it handling returns better than the humans, is it doing something slightly out of policy, does it need retraining. For every automation running in your business, you need three answers: who owns it, when do they look at it, and who can switch it off if that person’s on leave. If any of those are blank, you haven’t finished rolling it out. You’ve just let it loose.


The Takeaway

There’s no formula for best practice AI in ecommerce right now. Nobody can hand you a list of what’s safe to automate, because it depends on your customers, your margins and how much of a mistake your brand can absorb.

What you can do is make the call deliberately instead of by default: know the worst case before you start, know the difference between a rough patch and a real problem, and make sure every automation has an owner.


Frequently Asked Questions

How do you decide what tasks to automate with AI? Don’t just weigh the upside, the hours it saves you. Ask what happens when it’s wrong and who finds out first, you or your customer. Score potential jobs on Impact, Confidence and Ease (ICE), then add a fourth factor: the worst case and what it costs to undo, since customer-facing jobs often carry the biggest risk alongside the biggest reward.

What’s the difference between a “bumpy” AI rollout and a “showstopper”? Bumpy is friction that improves with training: the output needs editing, it’s slower than doing it yourself at first. A showstopper is wrong in ways you can’t predict, wrong without telling you, or reaches a customer before it reaches you. None of those improve with familiarity, because it’s a design problem, not a settling-in one.

How long should you test an AI automation before switching it on fully? There’s no fixed number, but set the bar before you start rather than deciding as you go. Style My Wall’s Justin Bausch ran three to four weeks of AI-generated customer service recommendations, manually checked by his team, before switching autopilot on, and again after switching it off to fix an error.

Who should own an AI automation once it’s live? The same person who’d own the equivalent human role, such as a customer service manager treating an AI agent like another team member. For every automation, you need three clear answers: who owns it, how often they check it, and who can switch it off if that person is unavailable.


Read the full transcript Auto-generated

[Ad]: Marketplaces in Australia aren't just emerging or a nice side piece to DTC anymore. They're established, they're complex and they are moving fast. Amazon continues to accelerate. TikTok shop is looming and for most brands, managing a marketplace properly has become a full time job. Forecasting, content, ads, logistics, compliance. It's a lot and it requires specialist knowledge. That's where pattern can help. Pattern don't just advise on marketplaces, they actually buy your product and then sell it on global marketplaces as your partner. That means they have real skin in the game. They only win if you win. Simple as that. Through their marketplace accelerator model, Patton handles everything from inventory forecasting and listing creation through to advertising, fulfillment and international expansion. So if marketplaces feel overwhelming or you know you should be doing more, but you just don't have the time, the team or the energy, contact Patton to help you make the most of the marketplace opportunity. Learn more at au.pattern.com. Almost every week at the moment, someone asks me what decisions, what creative tasks, what reporting they should be handing over to AI in their e commerce business. And given the huge changes that we've had over the last few years, it is absolutely the right question to ask. Because if you're not looking to automate or optimize your e commerce business with AI, you will be left behind. I think we all get that. However, most people start from the upside. We get lost in the romance of AI. What does this save me? How many hours does it give me back? That is a fair place to start, but it's also a terrible place to decide from because the upside is everywhere. We can see every job on our list can be improved by AI. Everything looks like it's worth automating when you're only looking at what you're getting back. The thing that actually sorts one priority from the next is what is the potential downside, what happens when when AI gets it wrong and more to the point, who finds out first, you or your customer? What are you giving up by handing something over to AI that could go wrong or that your competitors could easily do just as well and you lose your human edge. So today we are going to work through how to decide what you let AI run. And then once it's running, how do you tell the difference between what a normal AI learning or rough patch might look like and what's the sign that you should pull it straight back to how you were doing it before? Justin Bausch has been further down this path than most. He sold artwork on Facebook Marketplace while he was still the CRM manager at RideAware. He bought a printer, hired his nephew and turned it into Style My wall. It did seven figures in year one. It grew from one staff member to 20 inside two years. And underneath it all, he's built this business on an AI platform that now runs his ads, his warehouse, and his customer service. He built it all in Claude code over about 11 months and now he's offering it to other businesses if they want to use his system. Now, he is one of the few people who, while he's very excited about AI and the possibilities and he's shown what he can build, he's also very open to telling us where it can and where it has gone wrong. So let's hear it first from Justin. It's almost functioning as your ERP for small business.

Justin Bausch: Absolutely, yeah. And I'm tailoring it for like I had another business that says, hey, Justin, I want my courier performance rates. I want you to integrate with how that's tracking and have it provide recommendations. What am I paying too much for? Where can I? And I did it. You know, at the moment, it's doing customer automation for customer service. It's going into all my tickets integrated with Gorgeous and it's replying to customers where it knows it can reply. So that's another beautiful thing about it. Before, if a customer had a frame that was damaged before, they would have to take a photo, send it to my customer service team. They would have to reply and say, when are you free for us to pick up the damage return when can do a resend. And now that handles it all by itself. It will analyze the image of the broken frame. It will understand that it's, we can't repair it. So we say, you know, it will recommend just throw it out. Or if we can repair it, it'll say, yep, here's a portal for you to click for returns from a courier by the label. They pick in the time and date. It will publish that, that will create a draft new order in Shopify, tag my dispatch team to say Janet from blah blah blah, got this frame, it was broken. Please resend it. Or it does it all automatically, you know, so the, the possibilities are just endless.

[Ad]: That's where it gets interesting, I think when you start putting into the process chain, when it's not just one off executions or one off activities, but it's actually part of a process.

Justin Bausch: Yep, yep. And to be honest, there was teething issues, but it's gotten very, very confident now that it, it does it accurately.

[Ad]: I mean, yeah, you couldn't push it to the level that it's at without teething issues. What's been the biggest mistake? Let us in on. On where it's gone rogue.

Justin Bausch: Probably it identified customers getting the complete wrong order and sending out emails where it was identifying they were getting unframed. Like an unframed rolled up poster versus a framed. It just had something to do with identifying the variant titles and it just spit back, oh, you actually didn't buy frame, you just bought a poster. And all these customers that spend five times more than a poster, they're just like, what the hell is this a scam? Blah, blah, blah. So I turned off that autopilot and then I got my customer service team, hey, let me know. It would send. It would create then draft responses instead of confirmed ones. So then my customer service team, after a couple of weeks, they said, yep, I'm confident now that it's replying accurately. Let's now turn it back on. So that was a big tip.

[Ad]: That big pain point that you had around the print on demand. And when you go through those peak periods like Black Friday and being able to keep up with customer service and customer communication, how have you used your platform with that? Because that is a problem that is probably fairly unique to your business.

Justin Bausch: 100%. And I this is a great question. It's the best case study that I've done with my app because we were in Bali and I this is during a big sale that we had and I was being hounded every day by customers with updates and I'm like, what can I do to fix this? So I got my app to look at a rolling day 7 average of when a customer places the order and then when it is marked as delivered to their door. So then I'm flagged every single day. If that window is increased by one day or decreases by one day, if it starts blowing out over a threshold, it pings. Me and my operations manager, we need more staff for overtime and we need weekend work to catch up. So immediately you identify that. On top of that, if it understands that it's blown out the dispatch window, it'll then go into my Shopify update at checkout. Your the shipping date window. Instead of saying 8 to 12 business days, it'll automatically say 8 to 15 business days. It will then go into Clavio when I made that change and the email that a customer is sent after they place an order, it will update that in the flow. Hey, just letting you know, this is the update window. And then I got the app to build a tracking portal. So then the cut and then that tracking portal estimated window will notify that customer. It's now actually gone out to 8 to 15 business days, not 8 to 12. And then the progress bar in that thingy will slightly lengthen and it will just let customers know so they are updated at every single possible stage before, during, after. And look, to be honest, we still get complaints, people still don't read. It's unfortunate but that's a, that's a big, big example of what I did to turn that around because we need to over communicate and that's something now that the app does and I don't have to look at it anymore, it just does it automatically for me.

[Ad]: Now the problem there isn't that the AI broke. If you're trying anything with AI at the moment, you need to expect that it is going to break or that it's not going to work perfectly. Right now that's the cost of change. The problem was that the first people to find out was were customers who ended up paying five times the poster price. That's not just the system collapsing, but also doing massive damage to your brand. So we're going to dive into a few lessons from the archives now to draw on what can go wrong with AI and how to work out where you automate your business. So the first lesson is knowing your worst case. Where do you start? Have a think about what you would most like to hand over to AI to this month. And instead of finishing the sentence, this would save me finish this one. If I get this wrong with AI. The worst that happens is most of us never go down that path when we're assessing what we are going to hand over. But it is a very useful lens to look at your list to understand where the safe wins are and where you'll need to do more due diligence. There's a scoring methodology for this and you might already know it. It's called ICE I C E. You score each priority out of 10 on impact, confidence and ease and then you average out the three. It's a really good way of helping prioritize when you've got so many things that you could be doing. Run your potential AI automations through it as it stands and it will hand you back where some of the biggest risks and the easiest wins are. However, you might also want to score a fourth thing along it the worst case, who finds out first if it does go wrong and then what does it cost you to undo? Justin had two automations running in the Same month, the inventory scanner logs a frame as each staff member assembles it and deducts the stock automatically. If that goes wrong and a number is off in his stock inventory, he's going to find out about it pretty soon and it's not going to be the end of the world. However, if an automated AI customer service agent replies telling customers that they've bought a poster, that's a very different scenario. Both are using AI and both are automating business processes, but the customer facing one is a much worst case scenario because it directly impacts the customer. Abel Butler runs Health Post and he's drawn a clear line through the middle of his customer service on what he will continue to own and what he's happy to hand over to AI agents.

Abel Butler: We need some, some really solid ethical guidance and guardrails around that because yeah, we, we do really value being able to give people appropriate natural health product advice. And yeah, we need to automate aspects of that very carefully rather than rush in.

[Ad]: Abel is comfortable automating order related inquiries and points of what Gorgis is building where a customer gets an order status straight out of Shopify. Because the customer gets the right answer immediately. That makes a lot of sense to automate. However, he stops dead at product and natural health advice. As you heard, he said, we really value being able to give people appropriate natural health product advice and need to be really careful with that because one, it's people's health, but it's also what they're known for. They need to really stick to ethical guidance and guardrails first. It's all the same customer service system here where you've got AI agents and humans operating it. But there is a clear delineation between what needs a machine and what needs a human touch. And notice that the line that he drew there had nothing to do with how hard the job was. It was more around the risk and the customer experience in working out what he will hand over to AI. That brings us to the second lesson and it's working out whether in those early stages that automation is bumpy or it's a showstopper. Because once you send an automation live, every automation is going to be rough and have its moments. For the first hours, days or weeks, the skill is working out what kind of rough you are looking at. Bumpy is the kind of friction that you can push through. That might be where the output needs editing, it might keep missing some context that you have to feed it, or it might just not fully understand your brand. All of that is probably a Learning period. And like a team member, you need to keep training it to get it up to the point where it can work by itself. A showstopper is different in kind. It is wrong in ways that you cannot predict, or it is wrong without telling you it is wrong. Or it is wrong and reaches a customer before it reaches you. None of us want to be in that situation and none of these improvements with how familiar the bot gets with your business. That is a design problem, not a settling in problem. So three questions and NES here probably means that you're looking at a showstopper. One, can it be wrong without me finding out? Two, does being wrong reach a customer before it reaches me? And three when it is wrong, can I tell you why the agent or the bot or the automation was wrong? Justin's poster bug is a really clean example and the fix took him no time at all. Once he knew where it was, the problem was. Question 2 it got to a customer without him knowing about it. We had a great conversation with James Johnson from Shopify and he has a way of finding your bumps on purpose before anything major is riding on it.

Abel Butler: That will be painful and bumpy and you'll learn what's really hard and what's really crappy. That's a good mindset to have and you will have a few nuggets in your workflow to go. Everything is AI scribbled into it, but that's irrelevant if you're not getting the business impact.

[Ad]: James gives us all a task to do there where he says go AI first on everything for one to two weeks. Before you send an email, write a report or do any task, reach for the tool before you do it yourself. Start with why not AI? He's honest about what that feels like, that it feels painful and bumpy and a little bit crappy. But he calls the discoveries nuggets. He is just as firm though the other way, where everything has AI sprinkled into it. But it's irrelevant if you're not going to get the business impact and worse if you get a negative impact. What James is describing is maybe a bumpy fortnight where you enter into it deliberately with nothing in stake but a few hours of your work and you'll probably get some learning along the way. That's a really cheap place and probably really beneficial place as you play around and you work out what is and what isn't possible. But before you switch anything on for real, you need to write the pass mark down, not as you're doing it. What good looks like before you actually automate anything, what is the job that you're trying to achieve and what does good look like? If you set that as you're working through it, that will change and you'll probably find yourself letting AI get away

Abel Butler: with more and more.

[Ad]: Justin ran three to four weeks of recommendations that he checked by hand. His customer service team had to say out loud that they were confident before autopilot went back on. You will not set an honest bar after you've started automating because by then you'll just want the thing to pass. All right, lesson three. Someone needs to own the automations and I think this is the one that we get wrong the most. The day that you flip an automation to autopilot cannot be the endpoint. It's the day that the automation becomes part of your operating system in your business. Have a think about how this usually goes out into the old world of project management. When we do a project, the project manager usually has a sponsor who looks over things while they built. But it's generally the project manager's responsibility. Once it's stood up, it works, the project closes and that project gets handed over to the owner or the sponsor or who will manage it from there on in. That same approach should be for new AI automations and additions that are integral to how your team and your business operates. Emily Elvy has spent years inside customer experience teams and she has the best framing that I've heard on this.

Abel Butler: Your customer service manager should effectively own looking after it like it looks after every other team member that you have in your business. The next thing I'd say is it can't do everything and don't expect it to and nor should it. There's things it's going to be really good at, but there's absolutely moments where you need a human and human interaction.

[Ad]: I love that idea of Emily running a weekly check in on the agent's results against the rest of the team. And I love that it goes both ways. Is it handling better? Is it handling returns better than the humans? And what can the team learn from it? Is the team doing something slightly out of policy and does the agent need retraining to match? She is also clear that an AI automation cannot do everything and nor should it that there are moments where you need human and a human interaction. Because here em's not talking about describing supervision, she's describing a working relationship where both sides get better and that is a continual learning system that makes both the AI and the humans better over time. Can you believe we're saying all this, but it's so true. That's the world we're in right now. She reckons running these properly is a real job rather than something bolted onto someone's afternoon, and that an AI agent manager is going to become its own role, which is very much worth thinking about as you look at your own team structures. Because at the moment those people are either scattered in your business or perhaps it's likely to be you knowing where all those automations are. So for every automation running in your business right now, 3 Answers who owns it, when do they look at it and who can switch it off if that person is on leave? If any of those are blank, you have not finished rolling out your automation, you've just let it loose now. There's no formula in saying all of this. There is no formula for best practice AI in E commerce at the moment, nobody can hand you a list of what is safe to automate and what is not, because it depends on your customers, on your margins and how much of a mistake your brand can absorb. What you can do is make the call deliberately instead of just trying to automate everything. Know the worst case scenario before you start. Know the difference between a rough patch and a real problem, and make sure AI has an owner in your business for each project system or automation. Now we cover how to decide what to take on and what to leave as humans in module 1 of the E Commerce Foundations course, which we are just about to release and it brings together lessons from over 600 interviews that I've done with Australia's best E commerce leaders. And if you're trying to work out what you would hand over first as an automation or or you want to compare notes with other leaders on how long they let something prove itself for before they trust it. Those conversations are happening right now in the Add to Cart community and every day you can join for free over on add to cart.com that's the playbook for this week. Go write the doomsday scenario for your AI automations and see where you land. See you next Friday.

Add To Cart is brought to you by
Gold partners of Australia's ecommerce podcast
Primary episodes · Mondays and Wednesdays

Other episodes you might like