My Jeff Bezos Story: How to Survive a Bad Failure
7 steps to overcome a bad failure and earn back trust
Welcome to this week’s free article of Level Up: Your source for executive insights, high performance habits, and specific career growth actions.
How do you recover when you personally disappoint a CEO like Amazon’s Jeff Bezos?
In 2011, for the Amazon Appstore launch, Jeff Bezos was planning to put a personal letter on the Amazon homepage introducing his favorite feature to customers.
I owned that feature.
And it failed under load at launch.
Jeff was never able to publish his letter and he knew who owned the failed code.
Me.
To make matters worse, the bugs that brought down the launch were not complicated.
The simple truth was that we missed some things.
Thus, I had no reasonable explanation for the problems aside from failure to audit deeply enough.
I share this painful and embarrassing story because I survived this incident.
In fact, I strengthened my relationship with Jeff through it and was ultimately promoted from Director to VP.
If I could get through failing to deliver a feature in a public launch for the CEO of the company, then you can get through your problems when they happen.
Here’s how:
I owned it. From the minute the launch failed, I owned it. I did not blame others and I was the main communicator about the problem.
We worked around the clock to fix it. Some will say that we should not have had to do that. I would say we wanted to. We were embarrassed and wanted it fixed quickly.
I communicated thoroughly and regularly. The secret to crisis status is frequent updates that say what you are doing and when you will update leadership again. As long as leaders know they will get more information in an hour or two, they will usually give you space.
We accepted help from experts. In the end, it was the Amazon Principal Engineering community that fixed our problems, not my team alone.
We showed we were not totally incompetent (because at first, it looked like we were). When interrogated and audited, we showed that overall our engineering was solid despite the rookie mistakes.
I went and faced Jeff Bezos. I went to a small meeting with him, sat next to him where it was inevitable we would speak, and discussed the issue. Hiding or running would have been doom, as it is easy to flame someone in an email. In person, I got a calm hearing and some empathy.
I endured a long period of distrust in my execution and judgment. I had to prove that what had happened was an aberration, not a pattern. That took two years.
I still have regrets about the incident, but they are not the point.
The point is, work for long enough and you will screw something up.
What you do next determines how that works out for you.
Important lessons as a young leader.
I did not get out of this problem without a supportive team and a boss who stood by me for the long term.
I also regret to say that others took the problem harder than I did. I had a personal feeling of failure and some shame for having missed obvious things.
One of the engineers on the team, the young man who had actually written the code that failed under load, quit shortly after. He was not forced to quit nor reprimanded, but I believe he could see the pressure on all of us and felt bad about it.
I was a younger leader and while we avoided blaming him, this was not enough.
If I could go back and change one thing about how we handled the incident I would change the support we gave him to go from "no negatives" to actual positive support. We failed him, not the other way around, and I do not think I made that clear enough to him at the time.
It's always a team effort.
I have described my leadership actions so that other leaders can learn from them. But without a team that could recover from the problems, I could not have either.
My termination was considered.
My SVP told me (to my face) he thought about it.
However, the combination of how I handled the problem and my track record of 6 years of high performance worked in my favor.
I wrote a follow-up post on caring for your team in a crisis.
And if you are new to management, I co-wrote 10 hard truths about tech leadership with a former Twitch colleague.
Audience Insights
I consolidated additional ideas worth considering from my LinkedIn audience, including:
What seems obvious as a leader about root causes (e.g. team did not build mechanisms to prevent this failure) may not be obvious to the team/individuals in the situation. As a leader, provide the external help to see that your miss was only one of many problems. Help your team see the forest from the trees.
Majority of comments cite #7 as the most difficult to endure due to the long time frame (in my case, two years). Trust can only be built slowly, but it can be easily and quickly lost. You regain trust by time, consistent performance, and not being defensive about the initial distrust.
Be vocally self-critical and own the communication for your mistake. Do not blame others and do not let someone else communicate for you.
Become A Trusted Decision Maker
I said it took me two years to overcome distrust in my execution and judgment.
In other words, I had to prove I was a trusted decision maker.
My course, Stuck at Senior Manager - How to Break Through to Executive covers the performance standards that make an executive and how to optimize your promotion process. Part of the performance standards includes what it means to be a trusted decision maker and a science-based process on how to make good decisions.
Below is a video clip on this topic.
I offer this course in two formats:
Live course online (17.5+ total hours of content, 4.7/5 rating, 800+ alumni, named Maven’s Top Leadership & Career course, recommended by Lenny Rachitsky)
On-demand course (3 hour course)
Share this Article
If my failure story helps you either overcome your own failure or helps prepare you for future failures, share it inside your workplace and with others who might get value.
Connect With Ethan & Jason
Follow Ethan on LinkedIn.
Get Ethan’s career advice on YouTube.
Connect with Jason (Ethan’s operating partner) on LinkedIn.
Learn more about Ethan’s live online courses and on-demand courses.
Contact us for corporate training, speaking, podcast appearances, and more.
Many subscribers expense this newsletter to their Learning & Development budget, use this email template to send to your manager.