Measuring your Way to Profit

Great teams I was on didn't just measure points and rebounds, they measured deflections.

Two-time NBA Champion Kenny Smith.

May 4, 2013

Wufoo.com was wildly successful. Chris, Kevin and Ryan took $100,000 and turned it into a company that sold for $35,000,000 to SurveyMonkey.com. Survey Monkey was last valued at 1.35 billion dollars.

I worked at both companies.

What was the biggest advantage Survey Monkey had over Wufoo? First, Survey Monkey has an oatmeal machine (yes a machine dedicated to making oatmeal). Second, Survey Monkey measures everything.

Every marketing campaign is measured for effectiveness, every page is measured for conversion rate, every feature is A/B tested, every ad is measured for click throughs, every price is tested for results. The biggest difference between the two companies is how much is known about the impact of every change.

There is a dark side to measurements. Measurements in the wrong hands can hinder progress or raise unfounded concerns. Sometimes they can be a real bitch to capture. However this is not an excuse for ignorance, no matter how small your organization.

Great companies measure what makes them great.

Fix Processes Not People

Despite what it seems in the moment, most quality problems are not caused by people slacking off or acting maliciously. In reality, most quality problems are systemic in nature.

Eric Ries, Writer of The Lean Startup

On Wednesday October 17th CodePen.io went down hard. Chris Coyier, the lead huckle bucker himself and site designer, made a mistake that brought down the entire site during peak hours. He accidentally merged a development branch into our mainline development branch. The development branch had a database update that unbeknownst to Chris would run automatically when he deployed to production.

The next 25 minutes were harrowing. Waiting for servers to come back up is painful. Users are annoyed, new users and potential customers are turned away. There is a lot of angst and frustration during this wait time, but years in software development have taught me where to direct my anger. At the software itself.

It's easy to blame Chris for this downtime but it's stupid. Blaming Chris is a classic example of the fundamental attribution error.

The push process failed us all. We failed to setup a safe push process.

3 major mistakes we made

  • Merging a development branch
  • Running database updates automatically
  • Deploying to production without running smoke tests

The fixes we now need to make

  • Stop database updates from running automatically
  • Setup continuous integration server
  • Run smoke tests prior to push to production

The moment after a major mistake is crucial. It's easy to forget the pain of a mistake after things start running. The team needs to conduct an error autopsy. What went wrong? Why? What can we do to stop this from happening again?

Good people can make great teams if they combine their strengths and cover their weaknesses. A great team doesn't look for who to blame. Instead great teams look for what to fix.