How long does it take for Amazon to deploy a bug fix in production?
Short Ans: ~2 days
Firstly, I don’t work at Amazon, and I haven’t worked at Amazon (as of 2019).
You can rightly ask me, how do I know the time taken for Amazon to deploy a bug fix in production? Well, I found out a bug in the Amazon.in and reported to the Amazon security team (CCed Jeff Bezos, I know ;) ) and then calculated the time it took Amazon to fix the bug in production.
Bug in itself was a rookie mistake. All the subtitle links from “Subscribe and Save” were redirecting to pre-prod servers (Maybe an intern deployed the code in production with a pre-prod configuration file, well we will never know).
|The pre-prod domain of Amazon!|
As soon as I found out, I wrote a quick mail to the Security team at Amazon, and I thought it would be a significant incident and stuff. To my surprise it was not until the next day, I got a call from Amazon executive and was asked to submit more information, and it was treated as a Customer support case.
|The Original Mail reporting the bug|
|Response from the Amazon Executive|
Then, I did a screen recording of the bug by reproducing the issue(which was clicking on those links which were redirecting to the pre-prod servers) and forwarded the video to the Amazon Executive.
|The screen recording of the bug!|
The issue was resolved after two days. The most interesting thing about this entire incident was fixing a bug in production took almost two days for even a company like Amazon. This illustrates however big you are, robust your testing is, talented the people working on this project are, if there is a software, bugs are bound to be there, and it will take hours to fix it (In this case it was days).
Since the incident was treated as a customer issue, I received the last mail from Amazon after two weeks as shown below.
|A generic response from Amazon team!|
I think I deserved a cap or $15 credit in my AWS Account!