The BigDoor Blog | August, 2010

For companies looking to gamify their services, BigDoor’s Badge-O-Matic provides the path to a quick implementation

SEATTLE — August 31, 2010 — BigDoor, a startup that powers game-like mechanics for website publishers, announced today the launch of a new tool for easy gamification called the Badge-O-Matic.

BigDoor is a Seattle startup that helps companies build game-like mechanics and loyalty programs into their site or app by enabling points, badges, levels, leaderboards, virtual currency and virtual goods. BigDoor provides a platform that allows any website or mobile application to create and manage their own virtual economy and game mechanics through an open and accessible API.

“We’re continuing our work to make gamification easy for site owners,” said Keith Smith, co-founder and CEO of BigDoor. “The Badge-O-Matic takes away some of the complexity so that setup and implementation of badges is painless.”

The Badge-O-Matic is a tool that lets the virtual economy administrator quickly create a series of badges to reward users for performing actions. As they level up, the users automatically receive the badges. With the growth in services that feature badges such as Foursquare, the implementation of badge achievements in websites is becoming much more pervasive, and BigDoor is launching Badge-O-Matic to help provide quick badge customization for site owners.

“We heard from our customers that the badge creation process was a bit time consuming,” said Jeff Malek, co-founder and CTO of BigDoor. “So we expect the Badge-O-Matic to be just the first in a series of tools to make customizing their economies quick and easy.”

About BigDoor

BigDoor provides a platform that helps companies build game-like mechanics and loyalty programs into their site or app through points, badges, levels, leaderboards, virtual currency and virtual goods. The BigDoor platform is extensible in order to allow a wide variety of apps to be built on top of it and is enabling an ecosystem where publishers, developers, content providers and advertisers can all interact and realize benefits well beyond what can be gained by building these features in-house.

So you’ve already decided you want to reward your users for performing certain actions by giving them points and badges. Level up! But you’re stuck on the next step: “Now how do I make my stinkin’ badges?”

In our system, awarding the progressive badges as users hit different thresholds of activity used to require a little bit of tinkering. You would create URL objects for the badge images, and then you had to tie them to the different Level objects in a Level Collection, which was tied to a Currency, etc. It worked great and kept track of everything for you, but simply wasn’t very user friendly to set up.

With that difficult experience in mind, we’ve introduced the Badge-O-Matic to our account tools. This new quick start tool ties together the different objects in our system so that you can easily create and edit Badge levels without having to jump around in your Economy settings. It’s even got a few template badge level tracks that you can use directly in your own gamification experience, or that you can just feel free to play with to learn how things all work together.

Even if we don’t have a premade template that matches exactly what you want to do, remember that Badges in our system are entirely customizable with your own creative since you can always point URLs to your own image resources and define them any way you want. Want to create a badge set that awards 25 different levels of custom Unicorn Horn badges for users who add more and more virtual Glitter to their profile page? Badge-O-Matic can do that. But it may feel a little bit embarrassed and not talk about it at home.

Check it out now to get yourself some stinkin’ badges. Or watch the video below for more details.

- roy

The small running back leading the large offensive line onto the field

My 10 year old son has begun playing tackle football this year, and I honestly can’t tell which of us is more excited about this fact.  We now find ourselves spending a fair amount of time talking football strategy.  He plays running back and is easily the smallest kid on the team.  As a small, nimble, ball-carrying back, my son has quickly developed an appreciation for the strategic importance of his offensive line.  Without them blocking for him, he would quickly be overwhelmed by the 11 defenders eager to tackle the kid with the ball.

I often compare the football offensive line to a startup company’s tech operations; the guys who keep our servers and systems up and running.  As a startup that prides itself in being small, nimble and quick – with weekly sprints and product releases – the analogy to 5^th grade tackle football feels somehow appropriate.

As an offensive lineman your primary job is to protect the guy with the ball.  If you play a perfect game, the credit you’ll get is that nobody will ever call your name.  If you screw up just once, a defender will sack your quarterback or tackle your running back for a loss and you’ll look like a buffoon and likely kill the drive your entire team is working on.  Same goes for the operations of a small tech startup; being perfect 90% of the time means that 10% of the time you are down which will almost guarantee the failure of the entire organization – no matter how good you are in every other position.  In other words, 90% perfection isn’t even close to being good enough as an offensive lineman or a sysadmin.

Startups are a team sport where the true “skill” positions are most often behind the scenes, technical and usually very underappreciated.  In honor of football season kicking off across the U.S., I want to point out how important the tech ops folks are and say a great big thank you to every one of them.

–Keith

We have a great product, awesome, supportive customers, and an incredible team.   As difficult as the last week has been, these things still hold.  In fact, as a result of last week, our product offering has become more bullet-proof, we’ve discovered how incredibly supportive and empathetic our customers are, and our small team of developers has joined a tiny worldwide population of geeks well-versed in MySQL InnoDB data recovery.

Admittedly, over the last few months we’ve moved too quickly, accepting higher risk in exchange for greater innovation and forward strides with our product.  We’ve worked hard to push the maximum amount of changes and value for as many customers possible.   In fact, our release manifests show that over the last four months, we’ve pushed out 20 releases of our product totaling close to 400 discrete enhancements or fixes.  Those changes were performed by a team of two feature developers and a single developer solely dedicated to automated system testing – and they’ve done an incredible job of it.

But increasing velocity can increase risk, and conversely reducing risk often involves slowing down.   Slowing down means spending more precious startup time, which there isn’t an endless supply of.  On one end of the continuum there’s a failed mess rushed to market, and on the other a near-perfect half-baked nada.   We’re there in the middle, trying to balance the risks and benefits.

One way to move quickly with a small team is to give it full access to all server resources. We didn’t do that – one person has been responsible for building out and maintaining the back-end systems,  47 servers at the moment.  Complete access to the production systems is limited, servers are firewalled, and accounts are well locked-down. But like any small and agile IT team, every person in the group still has enough access to wreak havoc.  That’s what happened last week.

We’ve developed a suite of automated system tests that amongst other things verify the integrity of our API to ensure that we don’t make any inadvertent backwards-incompatible changes, measure performance by entry-point, and detect unexpected increases in SQL query counts.  These tests require removal of pre-existing data from the database, which is scripted and part of the entire process.  There are two hosts that we run these tests against, but a new employee inadvertently ran them against our production database last Monday night (ironically, apparently national Roller Coaster Day), causing all data to disappear on both the primary and backup slave server.  In addition, through a strange sequence of events our only good backup would have left us with around 8 days of missing data.  Once reality set in, I went through a series of emotions and reactions that can’t really be described and I’d prefer never to experience again.

After realizing the severity we began calling on external expertise for help.  Many sleepless hours and days later, we successfully recovered the truncated data with the help of two other external teams, one including the author of the only tool that can actually do this kind of arcane recovery.

Yes, this employee is still with us, and here’s why: when exceptions like this occur, what’s important is how we react to the crisis, accountability, and how hard we drive to quickly resolve things in the best way possible for our customers.   I’m incredibly impressed with how this individual reacted throughout, and my theory is that they’ll become one of our legendary stars in years to come.

I was even more impressed with how the entire tech team became hell-bent on fixing things.   We couldn’t have slept less.  This sentiment was later echoed by a number of the other residents at our open floor-plan office (Founder’s Co-Op headquarters).   Both Keith and I heard remarks about what a great team we have, many coming from other co-founders.

With that, there are a number of folks that deserve thanks and recognition for going above and beyond to help us through this colossal recovery last week:

• The BigDoor team
  • Collin Watson, diver-in extraordinaire and speaker of tongues
  • Lee McFadden,  API entry-point and big fan of NoSQL technologies
  • Brian Oldfield,  insomniac automator
  • Harley Holt, unflappable fire-eater
  • Keith Smith, megaphone and seer of forests
  • Ring Nishioka, Roy Schmidt and the rest of the BigDoor team
• The BlueGecko team : Sarah Novotny, Mike Hamrick, Patrick Galbraith and Jonathan Nicol, thank you for your on-call DBA and operational expertise
• The Percona team :   Aleksandr Kuzminsky and Baron Schwartz, thank you for your on-call data recovery expertise
• Geoffrey Nuval from DevHub/EvoMedia, thank you for being a great customer, partner and for your continued confidence and support
• Jim Banister from SpectrumDNA, thanks for being a great customer and for your confidence and words of support
• Elvis, for reasons that are self-evident.

I hope that this small bit of color helps to convey how seriously we take our customers and our uptime here at BigDoor.

There are a number of things we’re doing to prevent this from happening again.  It should have never happened to begin with, and as the CTO it’s my responsibility to make sure we take the time to implement proper safeguards.   For that failure, my most sincere apologies go out to our customers and theirs, as well as to our board and investors.   We’ll make it up to you, in the coming releases of our kick-ass platform.  In fact, the same day that we called the recovery complete, the team worked late into the night in order to do a feature release (the Badge-O-matic , amongst other things) – so we’re already delivering on that promise.

–Jeff

Games Beat: Website builder DevHub gets users hooked by “gamifying” its service

We’ve just recovered from an extremely painful and embarrassing outage of our API.  Because we love to embrace transparency, I figured I’d post the email that I sent to our customers earlier today.

——————

Update: BigDoor API is up!  Please resume your usage.

Please note that we are currently missing historical transaction detail and leaderboards.  All of the data has been restored, but due to its volume it will take a while for it to be loaded back into our production system.  Transaction detail will be loaded over the next couple days and leaderboards should become available during that same period of time.

All currency balances, endusers, levels, awards, goods, and all configuration data (defined transactions, currencies, etc.) are working and fully updated as of the time of the outage.  Please let us know right away if you see anything that doesn’t seem right – but we have thoroughly tested and scrutinized the database and believe all is good.

I want to take this chance to once again offer up the sincerest of apologies for this unbelievably painful outage.  There is absolutely no excuse for this kind of disruption in service, but I think we owe each of you an explanation of the root cause.

The outage was caused by a test script that was meant to refresh a test database when it was inadvertently run against our production systems.  This script truncated 79 gigabytes of data in a matter of seconds.  Truncate actions on InnoDB databases are meant to be final so they are incredibly difficult to be undone.  We have a master/slave database configuration but unfortunately our slave database dutifully truncated all of its data as well.  Then through an incredibly odd series of events, it turned out that our last good database snapshot was from 7/21 and through another strange series of events we were missing 16 days of binary logs – meaning we weren’t able to just reload our old snapshot and replay the bin logs.

This left us in a situation where we could either attempt to recover a database with missing data, or take on the nearly impossible task of recovering a 79 gb database with 65 highly relational tables post truncate.  We quickly began pursuing multiple tracks each holding varying degrees of badness.  We engaged Percona and Blue Gecko (two of the leading experts in data recovery) to assist us and large numbers of each of their teams have been working non-stop alongside our BigDoor team to recover the lost data.

The resulting chaos is encapsulated in the series of email updates below where I continued to provide overly optimistic timeline estimates.  My apologies for sucking so bad at that – each time we made a guess at a timeframe it was done with sincerity and what we believed was a conservative approach.  Obviously I was wrong in almost every case.

We will be diving into the postmortem of this outage immediately, but for now suffice it to say that we sucked badly and we feel like fucking idiots because of it.  The good news is that we have some of the smartest and most dedicated people I’ve ever had the pleasure of knowing here at BigDoor, so we will take the appropriate steps to be sure this doesn’t happen again.  I can’t guarantee that we won’t ever screw up again, but I can promise that we will have a much more effective net in place going forward to mitigate issues such as this.  We’ll get better, I promise.

Thank you for your patience and support through this ordeal.  Please contact me if you have any questions.

Keith

In a technology startup speed is very often the enemy of quality.  I tend to prefer speed and thankfully our CTO tends to prefer quality.  He is usually right.  He has a strongly held belief that you can speed up by slowing down.  There are a fair number of variations on this that I heard often as a child, “slow down and do it right the first time” or the oft-quoted Benjamin Franklin favorite, “an ounce of prevention is worth a pound of cure.”

Tonight we are experiencing by far the worst outage we’ve ever experienced in our company’s history as a direct result of us trying to go fast and operating for a short period of time without a proper net.  Due to a unique set of circumstances we’ve ended up with a major database outage and will likely be down for at least 12 hours.  This is maddening, embarrassing and frustrating as hell.  Unfortunately I can trace most of the blame directly back to a variety of my decisions that were supposed to make us go faster.  Startups have an amazing ability to continually provide painful lessons.

This outage is having a huge impact on our customers and we hate that more than just about anything in this world.  Our team is continuing to work through to the night to mitigate the outage and get our systems back online.  We regret the outage and will be taking steps to make sure it doesn’t happen again – just as soon as we triage the situation and get everyone back online.

Since our announcement of our public beta and fundraising almost two months ago, we’ve been blown away at the reaction by the market.  There is clearly a large and growing market interest in gamification and virtual economies.  In a startup there are no shortage of challenges, but we’ve been blessed to have been able to address the issues of capitalization and market desire in our first year (although these two are sure to come back into play again before too long) and are now very focused on what we call, “finding the love.”

What finding the love means to us is that we are on a mission to make our product both necessary and loved by a large number of customers.  We’ve already been hearing from a variety of our partners that the BigDoor platform has allowed them to launch game mechanics significantly faster and with more flexibility, functionality and insights than they could have accomplished without our platform.  We are encouraged and happy to hear those things.  At the same time, we recognize that in its current form you have to be a developer (or have one working for you) in order to use our platform.  That will quickly be changing.  Within the month we will be launching new functionality that will allow a non-developer to quickly and easily add game mechanics to their site.  We are confident that our first pass at this will need some work, but we’ll continue to iterate by listening to our customers, analyzing tons of data and asking lots of questions to prospective partners.

Finding the love is a journey wrapped in a process, and one that we are having a blast with.