How We Broke Our Site for a Week

Today we’re going to tear down the tech-curtain and look at something we don’t often talk about: how our site works and what happens when it doesn’t. This past week a few of you may have received errors while trying to place orders. We only learned of it after a few angry customers called after their twentieth check-out attempt Monday afternoon.

 

What Happened?

Sometime over the weekend of the 21st, a glitch surfaced on our site which caused products not to show. This literally happened while I was sitting on my roof to patch a little hole from some storm damage. It’s a fun time trying to diagnose a broken website from 20′ in the open air. My biography one day will probably be titled “You wouldn’t believe me if I told you.”

We concluded in the short term that a very important file got deleted by mistake and the solution was to just restore a back up of the site. Back-up done, site online, we decided to make some updates to the site while we were at it to prevent any future repeats of the same problem. In principle, our updates shouldn’t have broken anything. In principle curling stones shouldn’t curve to the side either, but they do.

Some of our updated code referenced a new error logging system, which conflicted with the system used by our payment processing system, and things really fell apart from there. Our poor lead sales guy, Scott, couldn’t get off the phone with people complaining the site wouldn’t let them process an order. We pop onto the store, punch in a company card, load up an order, and… we can’t even buy stuff from ourselves. The order doesn’t show up in our sales panel, we don’t get any notifications, the orders just never happened (and luckily no one was charged).

 

What Now?

We fixed the bug with some internal adjustments and we’re going to be more stringent in testing updates in the future. We deploy them to a test server, but we never actually test whether or not we can check out on that server. It’s always been a foregone conclusion that the check out process will work because we rarely change it. Even this last update didn’t change it, it changed a harmless error logging system that… had an error.

To our customers who called to complain: THANK YOU. We appreciate your feedback, especially when things break. It makes us a better company and helps improve your shopping experience. If there’s anything you think we can do to better serve you (like having a checkout page that works), don’t hesitate to call or email us. We’re here to serve you.

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s