Easier Gamification with Badge-O-Matic

Posted in: Badges, BigDoor news, Blog, Gamification, UI on August 31, 2010 | No Comments

So you’ve already decided you want to reward your users for performing certain actions by giving them points and badges. Level up! But you’re stuck on the next step: “Now how do I make my stinkin’ badges?”

In our system, awarding the progressive badges as users hit different thresholds of activity used to require a little bit of tinkering. You would create URL objects for the badge images, and then you had to tie them to the different Level objects in a Level Collection, which was tied to a Currency, etc. It worked great and kept track of everything for you, but simply wasn’t very user friendly to set up.

With that difficult experience in mind, we’ve introduced the Badge-O-Matic to our account tools. This new quick start tool ties together the different objects in our system so that you can easily create and edit Badge levels without having to jump around in your Economy settings. It’s even got a few template badge level tracks that you can use directly in your own gamification experience, or that you can just feel free to play with to learn how things all work together.

Even if we don’t have a premade template that matches exactly what you want to do, remember that Badges in our system are entirely customizable with your own creative since you can always point URLs to your own image resources and define them any way you want. Want to create a badge set that awards 25 different levels of custom Unicorn Horn badges for users who add more and more virtual Glitter to their profile page? Badge-O-Matic can do that. But it may feel a little bit embarrassed and not talk about it at home.

Check it out now to get yourself some stinkin’ badges. Or watch the video below for more details.

- roy

The Strategic Importance of the Offensive Line

Posted in: Blog, Development, Startups, Technology on August 29, 2010 | 1 Comment

The small running back leading the large offensive line onto the field

My 10 year old son has begun playing tackle football this year, and I honestly can’t tell which of us is more excited about this fact.  We now find ourselves spending a fair amount of time talking football strategy.  He plays running back and is easily the smallest kid on the team.  As a small, nimble, ball-carrying back, my son has quickly developed an appreciation for the strategic importance of his offensive line.  Without them blocking for him, he would quickly be overwhelmed by the 11 defenders eager to tackle the kid with the ball.

I often compare the football offensive line to a startup company’s tech operations; the guys who keep our servers and systems up and running.  As a startup that prides itself in being small, nimble and quick – with weekly sprints and product releases – the analogy to 5^th grade tackle football feels somehow appropriate.

As an offensive lineman your primary job is to protect the guy with the ball.  If you play a perfect game, the credit you’ll get is that nobody will ever call your name.  If you screw up just once, a defender will sack your quarterback or tackle your running back for a loss and you’ll look like a buffoon and likely kill the drive your entire team is working on.  Same goes for the operations of a small tech startup; being perfect 90% of the time means that 10% of the time you are down which will almost guarantee the failure of the entire organization – no matter how good you are in every other position.  In other words, 90% perfection isn’t even close to being good enough as an offensive lineman or a sysadmin.

Startups are a team sport where the true “skill” positions are most often behind the scenes, technical and usually very underappreciated.  In honor of football season kicking off across the U.S., I want to point out how important the tech ops folks are and say a great big thank you to every one of them.

–Keith

Recovery Retrospect

Posted in: API, Blog, Development on August 26, 2010 | 2 Comments

We have a great product, awesome, supportive customers, and an incredible team.   As difficult as the last week has been, these things still hold.  In fact, as a result of last week, our product offering has become more bullet-proof, we’ve discovered how incredibly supportive and empathetic our customers are, and our small team of developers has joined a tiny worldwide population of geeks well-versed in MySQL InnoDB data recovery.

Admittedly, over the last few months we’ve moved too quickly, accepting higher risk in exchange for greater innovation and forward strides with our product.  We’ve worked hard to push the maximum amount of changes and value for as many customers possible.   In fact, our release manifests show that over the last four months, we’ve pushed out 20 releases of our product totaling close to 400 discrete enhancements or fixes.  Those changes were performed by a team of two feature developers and a single developer solely dedicated to automated system testing – and they’ve done an incredible job of it.

But increasing velocity can increase risk, and conversely reducing risk often involves slowing down.   Slowing down means spending more precious startup time, which there isn’t an endless supply of.  On one end of the continuum there’s a failed mess rushed to market, and on the other a near-perfect half-baked nada.   We’re there in the middle, trying to balance the risks and benefits.

One way to move quickly with a small team is to give it full access to all server resources. We didn’t do that – one person has been responsible for building out and maintaining the back-end systems,  47 servers at the moment.  Complete access to the production systems is limited, servers are firewalled, and accounts are well locked-down. But like any small and agile IT team, every person in the group still has enough access to wreak havoc.  That’s what happened last week.

We’ve developed a suite of automated system tests that amongst other things verify the integrity of our API to ensure that we don’t make any inadvertent backwards-incompatible changes, measure performance by entry-point, and detect unexpected increases in SQL query counts.  These tests require removal of pre-existing data from the database, which is scripted and part of the entire process.  There are two hosts that we run these tests against, but a new employee inadvertently ran them against our production database last Monday night (ironically, apparently national Roller Coaster Day), causing all data to disappear on both the primary and backup slave server.  In addition, through a strange sequence of events our only good backup would have left us with around 8 days of missing data.  Once reality set in, I went through a series of emotions and reactions that can’t really be described and I’d prefer never to experience again.

After realizing the severity we began calling on external expertise for help.  Many sleepless hours and days later, we successfully recovered the truncated data with the help of two other external teams, one including the author of the only tool that can actually do this kind of arcane recovery.

Yes, this employee is still with us, and here’s why: when exceptions like this occur, what’s important is how we react to the crisis, accountability, and how hard we drive to quickly resolve things in the best way possible for our customers.   I’m incredibly impressed with how this individual reacted throughout, and my theory is that they’ll become one of our legendary stars in years to come.

I was even more impressed with how the entire tech team became hell-bent on fixing things.   We couldn’t have slept less.  This sentiment was later echoed by a number of the other residents at our open floor-plan office (Founder’s Co-Op headquarters).   Both Keith and I heard remarks about what a great team we have, many coming from other co-founders.

With that, there are a number of folks that deserve thanks and recognition for going above and beyond to help us through this colossal recovery last week:

• The BigDoor team
  • Collin Watson, diver-in extraordinaire and speaker of tongues
  • Lee McFadden,  API entry-point and big fan of NoSQL technologies
  • Brian Oldfield,  insomniac automator
  • Harley Holt, unflappable fire-eater
  • Keith Smith, megaphone and seer of forests
  • Ring Nishioka, Roy Schmidt and the rest of the BigDoor team
• The BlueGecko team : Sarah Novotny, Mike Hamrick, Patrick Galbraith and Jonathan Nicol, thank you for your on-call DBA and operational expertise
• The Percona team :   Aleksandr Kuzminsky and Baron Schwartz, thank you for your on-call data recovery expertise
• Geoffrey Nuval from DevHub/EvoMedia, thank you for being a great customer, partner and for your continued confidence and support
• Jim Banister from SpectrumDNA, thanks for being a great customer and for your confidence and words of support
• Elvis, for reasons that are self-evident.

I hope that this small bit of color helps to convey how seriously we take our customers and our uptime here at BigDoor.

There are a number of things we’re doing to prevent this from happening again.  It should have never happened to begin with, and as the CTO it’s my responsibility to make sure we take the time to implement proper safeguards.   For that failure, my most sincere apologies go out to our customers and theirs, as well as to our board and investors.   We’ll make it up to you, in the coming releases of our kick-ass platform.  In fact, the same day that we called the recovery complete, the team worked late into the night in order to do a feature release (the Badge-O-matic , amongst other things) – so we’re already delivering on that promise.

–Jeff

BigDoor API service has been restored

Posted in: API, Blog on August 19, 2010 | No Comments

We’ve just recovered from an extremely painful and embarrassing outage of our API.  Because we love to embrace transparency, I figured I’d post the email that I sent to our customers earlier today.

——————

Update: BigDoor API is up!  Please resume your usage.

Please note that we are currently missing historical transaction detail and leaderboards.  All of the data has been restored, but due to its volume it will take a while for it to be loaded back into our production system.  Transaction detail will be loaded over the next couple days and leaderboards should become available during that same period of time.

All currency balances, endusers, levels, awards, goods, and all configuration data (defined transactions, currencies, etc.) are working and fully updated as of the time of the outage.  Please let us know right away if you see anything that doesn’t seem right – but we have thoroughly tested and scrutinized the database and believe all is good.

I want to take this chance to once again offer up the sincerest of apologies for this unbelievably painful outage.  There is absolutely no excuse for this kind of disruption in service, but I think we owe each of you an explanation of the root cause.

The outage was caused by a test script that was meant to refresh a test database when it was inadvertently run against our production systems.  This script truncated 79 gigabytes of data in a matter of seconds.  Truncate actions on InnoDB databases are meant to be final so they are incredibly difficult to be undone.  We have a master/slave database configuration but unfortunately our slave database dutifully truncated all of its data as well.  Then through an incredibly odd series of events, it turned out that our last good database snapshot was from 7/21 and through another strange series of events we were missing 16 days of binary logs – meaning we weren’t able to just reload our old snapshot and replay the bin logs.

This left us in a situation where we could either attempt to recover a database with missing data, or take on the nearly impossible task of recovering a 79 gb database with 65 highly relational tables post truncate.  We quickly began pursuing multiple tracks each holding varying degrees of badness.  We engaged Percona and Blue Gecko (two of the leading experts in data recovery) to assist us and large numbers of each of their teams have been working non-stop alongside our BigDoor team to recover the lost data.

The resulting chaos is encapsulated in the series of email updates below where I continued to provide overly optimistic timeline estimates.  My apologies for sucking so bad at that – each time we made a guess at a timeframe it was done with sincerity and what we believed was a conservative approach.  Obviously I was wrong in almost every case.

We will be diving into the postmortem of this outage immediately, but for now suffice it to say that we sucked badly and we feel like fucking idiots because of it.  The good news is that we have some of the smartest and most dedicated people I’ve ever had the pleasure of knowing here at BigDoor, so we will take the appropriate steps to be sure this doesn’t happen again.  I can’t guarantee that we won’t ever screw up again, but I can promise that we will have a much more effective net in place going forward to mitigate issues such as this.  We’ll get better, I promise.

Thank you for your patience and support through this ordeal.  Please contact me if you have any questions.

Keith

Speed versus Quality

Posted in: API, Blog on August 17, 2010 | No Comments

In a technology startup speed is very often the enemy of quality.  I tend to prefer speed and thankfully our CTO tends to prefer quality.  He is usually right.  He has a strongly held belief that you can speed up by slowing down.  There are a fair number of variations on this that I heard often as a child, “slow down and do it right the first time” or the oft-quoted Benjamin Franklin favorite, “an ounce of prevention is worth a pound of cure.”

Tonight we are experiencing by far the worst outage we’ve ever experienced in our company’s history as a direct result of us trying to go fast and operating for a short period of time without a proper net.  Due to a unique set of circumstances we’ve ended up with a major database outage and will likely be down for at least 12 hours.  This is maddening, embarrassing and frustrating as hell.  Unfortunately I can trace most of the blame directly back to a variety of my decisions that were supposed to make us go faster.  Startups have an amazing ability to continually provide painful lessons.

This outage is having a huge impact on our customers and we hate that more than just about anything in this world.  Our team is continuing to work through to the night to mitigate the outage and get our systems back online.  We regret the outage and will be taking steps to make sure it doesn’t happen again – just as soon as we triage the situation and get everyone back online.

Finding the Product Love

Posted in: API, Blog, Game Mechanics, Improvements, Startups on August 8, 2010 | No Comments

Since our announcement of our public beta and fundraising almost two months ago, we’ve been blown away at the reaction by the market.  There is clearly a large and growing market interest in gamification and virtual economies.  In a startup there are no shortage of challenges, but we’ve been blessed to have been able to address the issues of capitalization and market desire in our first year (although these two are sure to come back into play again before too long) and are now very focused on what we call, “finding the love.”

What finding the love means to us is that we are on a mission to make our product both necessary and loved by a large number of customers.  We’ve already been hearing from a variety of our partners that the BigDoor platform has allowed them to launch game mechanics significantly faster and with more flexibility, functionality and insights than they could have accomplished without our platform.  We are encouraged and happy to hear those things.  At the same time, we recognize that in its current form you have to be a developer (or have one working for you) in order to use our platform.  That will quickly be changing.  Within the month we will be launching new functionality that will allow a non-developer to quickly and easily add game mechanics to their site.  We are confident that our first pass at this will need some work, but we’ll continue to iterate by listening to our customers, analyzing tons of data and asking lots of questions to prospective partners.

Finding the love is a journey wrapped in a process, and one that we are having a blast with.

Holiday? What Holiday?

Posted in: Blog, Development, Startups, Success on July 5, 2010 | No Comments

As a kid I recall reading a story about Thomas Edison being so wrapped up in his work that he would often times forget to eat.  I remember thinking that it would be absolutely wonderful to someday find something that I care that passionately about that my focus on my passion would drive me to forget the basic essentials of life.  When I started my first company (over 15 years ago), I quickly realized what that wonderfully passionate focus felt like.

And it’s still there with company #4, but not just with me – with everyone else on our team as well.  Last Friday as we were working through a few issues, I had a thought that maybe we should have a day off given that it’s America’s birthday and all.  I mentioned something about it to our development team and all I got were a bunch of stunned looks.  You would have thought I had just suggested we swap our dev environment to .NET or something.  After a brief pause, they each explained to me a few of the long list of things they planned to complete over the weekend and rollout on Monday and then we moved on and the conversation was forgotten.

This morning as we were all cranking away in the office, our wonderful (and apparently more considerate than me) HR person reminded me that today is an official (and documented on our Holiday Calendar) company holiday.  That’s when it struck me how amazing our team is.  Not because they’ll work themselves to the bone – that doesn’t help anything – but because they are so incredibly passionate about what it is we are doing that other details seem to suddenly matter less.  That’s when you know you are onto something cool, and when you know you have the right team to execute on the plan.

And just so you don’t think that I’m some sort of a-hole, slave driver (I am, but I don’t want you to think that); I had payroll add a day of accrued vacation for everyone who “forgot” and showed up for work today.  Now I’ve got to get back to testing…the dev guys have more stuff to rollout.

5 Things We Suck At (for now)

Posted in: API, Blog, Partners, Startups on June 27, 2010 | No Comments

As a startup with limited resources, we are keenly aware of the fact that we can’t do everything we’d like to.  Most of this will be fixed with time, but as of right now we want everything to have been done yesterday.

Over the last two weeks I’ve had the opportunity to personally walk 47 companies through our platform for the first time.  For the most part the response I get to a new demo is, “We were just getting ready to build this ourselves, I’m glad we found you.”  Most companies find that our platform is flexible, can accomplish what they need, and will be a considerably better option than building everything from the ground up internally (even the hard-core “not invented here” CTOs quickly come to this conclusion).  A few come up with use-cases that will require some added functionality for our API – which we love to find.  Some file the concept away for when they have a bit more time, most seem to dive in immediately.

I’ve been “selling” stuff in one form or another for the last 20 years and I can honestly say that I’ve never before experienced this much interest in anything I’ve been involved in.  That is very cool, but it also means we need to deliver – and “find the customer love” as we like to call it around here.  One of the best ways I know of to do that is to listen to our customers and then constantly improve on the things that are important to them.  We value transparency, so I figured we’d make a list of some of the stuff we suck at so you’ll know what it is that we are working on and getting better at:

1. We don’t have game mechanics on our own site. Lame, I know.  What’s the old saying, “The cobbler’s children have no game console” (or something like that).  The best place to demo game mechanics for a non-game site is on our own site.  We’ll get there, but for now the cobbler’s children will be forced to play with a stick and an old tire in the middle of the street.
2. Real-time analytics. Up until a week ago we had real-time aggregated analytics, with cool charts and graphs.  We built our systems from the ground up to provide deep analytics, but the process we were using for aggregating and then presenting that data was meant to be an intermediate fix.  We knew it wouldn’t scale to huge proportions, but we thought it would last for a while.  We ended up hitting our critical mass late last week, and the hourly updates to our analytics ground to a halt.  Turns out that providing deep analytics on 200 million API calls per month is tough for a startup to do.  We are cranking on a very cool new infrastructure for our analytics that will provide real-time analytics at scale and will give our platform considerably more access to analytics.  (Worth noting: All transactions and data in our API are accessible real-time, it’s just our aggregated analytics that temporarily outgrew themselves.)
3. User Interface. We’ve spent the vast majority of our time and energy making the back-end of our platform work really well, but our “front-end” tools need some work.  They may be highly functional, but they aren’t overly pretty and more importantly the design and flow isn’t always overly intuitive.  This comes from a deeply held belief that the API is more important than the GUI we slap on it, since anyone can put their own GUI on our API.  But the reality is that our GUI should set the example for the rest that are to come so we will be working to improve it.
4. We aren’t quite sure yet how to talk about our platform. Is it game mechanics, or is it a virtual economy?  Are we the scorekeepers, the accountants or are we the game builders?  Do we power virtual currency or do we enable points and reward systems? Not sure?  Neither are we.  We do all of the above, but we are still figuring out how to communicate what exactly it is that we do.  With an emerging concept comes an emerging industry and with it an emerging lexicon.  Some words mean one thing to one company and something entirely different to another.  I love discussing what it is we do and seeing how people react – even when it is painfully obvious that they are giving me the “What in the f___ did this guy just say?” look.
5. Our platform isn’t accessible enough. Jeff likes to say that we’ve built silly putty; “It can do everything, but by itself it doesn’t do anything.”  This is the way many platforms start out, but it doesn’t change the fact that it introduces the requirement that for now programmers/developers are required to implement our platform.  Our early customers love that, because they want to configure and customize their own game mechanics and virtual economy and all have great engineers to implement what their product and marketing folks design.  But we are working with the developer community to build an “apps layer” on top of our platform that will make it much easier for non-developers to add game mechanics to their site or app.  Just think how cool it will be when as a blogger, in under 10 minutes you can point and click your way to adding game mechanics and a virtual economy to your site.

Like any startup, there is a very long list of the stuff we suck at.  These are a few of ours that I’ve been focusing on and that we are working furiously to improve upon.  So tell us, what did we miss that’s bugging you and what do we need to make better?

I Can’t Accept Not Trying

Posted in: Blog, Game Mechanics, Success on June 20, 2010 | 1 Comment

In 1997 a friend gave me a hard cover copy of Michael Jordan’s book “I Can’t Accept Not Trying: Michael Jordan on the Pursuit of Excellence“.  I had always been a Jordan fan, but frankly was much more interested in watching him than reading what he had to say.  The book sat on my coffee table for about a year before I picked it up one day and read it.  Turns out that Michael didn’t have much to say, but what he did say was pretty profound and his message stayed with me.

Brad Feld’s recent post reminded me of this book, which is nicely summarized in this 30 second clip.  The second lesson to learn from this video is that you don’t have to be long winded to be profound and impactful.

A Blessed Father

Posted in: Blog, Family on | No Comments

I absolutely love being an entrepreneur.  Starting a company consumes me and fuels my passion for creating something where nothing existed before.  Yet as much as I love what I do, it simply doesn’t hold a candle to what it is like being a Father.  Since I was 7 years old I’ve held the dream of one day owning my own business, as well as being a Dad.  I consider myself among the most blessed that I get to do both.

Yesterday my kid’s and I spent much of the day at my brother and sister-in-law’s house visiting with them and doting over their recently arrived third child.  Days like that tend to make you slow down and count your blessings, and I did exactly that.

Two months ago my son got his hands on some Mariner’s tickets for today’s game – so he is taking me to the baseball game for Father’s Day.  It should be a wonderful Father-Son kind of day, and I can’t imagine anything I’d rather be doing.