The Strategic Importance of the Offensive Line

Posted in: Blog, Development, Startups, Technology on August 29, 2010 | 1 Comment

The small running back leading the large offensive line onto the field

My 10 year old son has begun playing tackle football this year, and I honestly can’t tell which of us is more excited about this fact.  We now find ourselves spending a fair amount of time talking football strategy.  He plays running back and is easily the smallest kid on the team.  As a small, nimble, ball-carrying back, my son has quickly developed an appreciation for the strategic importance of his offensive line.  Without them blocking for him, he would quickly be overwhelmed by the 11 defenders eager to tackle the kid with the ball.

I often compare the football offensive line to a startup company’s tech operations; the guys who keep our servers and systems up and running.  As a startup that prides itself in being small, nimble and quick – with weekly sprints and product releases – the analogy to 5th grade tackle football feels somehow appropriate.

As an offensive lineman your primary job is to protect the guy with the ball.  If you play a perfect game, the credit you’ll get is that nobody will ever call your name.  If you screw up just once, a defender will sack your quarterback or tackle your running back for a loss and you’ll look like a buffoon and likely kill the drive your entire team is working on.  Same goes for the operations of a small tech startup; being perfect 90% of the time means that 10% of the time you are down which will almost guarantee the failure of the entire organization – no matter how good you are in every other position.  In other words, 90% perfection isn’t even close to being good enough as an offensive lineman or a sysadmin.

Startups are a team sport where the true “skill” positions are most often behind the scenes, technical and usually very underappreciated.  In honor of football season kicking off across the U.S., I want to point out how important the tech ops folks are and say a great big thank you to every one of them.

–Keith

Recovery Retrospect

Posted in: API, Blog, Development on August 26, 2010 | 2 Comments

We have a great product, awesome, supportive customers, and an incredible team.   As difficult as the last week has been, these things still hold.  In fact, as a result of last week, our product offering has become more bullet-proof, we’ve discovered how incredibly supportive and empathetic our customers are, and our small team of developers has joined a tiny worldwide population of geeks well-versed in MySQL InnoDB data recovery.

Admittedly, over the last few months we’ve moved too quickly, accepting higher risk in exchange for greater innovation and forward strides with our product.  We’ve worked hard to push the maximum amount of changes and value for as many customers possible.   In fact, our release manifests show that over the last four months, we’ve pushed out 20 releases of our product totaling close to 400 discrete enhancements or fixes.  Those changes were performed by a team of two feature developers and a single developer solely dedicated to automated system testing – and they’ve done an incredible job of it.

But increasing velocity can increase risk, and conversely reducing risk often involves slowing down.   Slowing down means spending more precious startup time, which there isn’t an endless supply of.  On one end of the continuum there’s a failed mess rushed to market, and on the other a near-perfect half-baked nada.   We’re there in the middle, trying to balance the risks and benefits.

One way to move quickly with a small team is to give it full access to all server resources. We didn’t do that – one person has been responsible for building out and maintaining the back-end systems,  47 servers at the moment.  Complete access to the production systems is limited, servers are firewalled, and accounts are well locked-down. But like any small and agile IT team, every person in the group still has enough access to wreak havoc.  That’s what happened last week.

We’ve developed a suite of automated system tests that amongst other things verify the integrity of our API to ensure that we don’t make any inadvertent backwards-incompatible changes, measure performance by entry-point, and detect unexpected increases in SQL query counts.  These tests require removal of pre-existing data from the database, which is scripted and part of the entire process.  There are two hosts that we run these tests against, but a new employee inadvertently ran them against our production database last Monday night (ironically, apparently national Roller Coaster Day), causing all data to disappear on both the primary and backup slave server.  In addition, through a strange sequence of events our only good backup would have left us with around 8 days of missing data.  Once reality set in, I went through a series of emotions and reactions that can’t really be described and I’d prefer never to experience again.

After realizing the severity we began calling on external expertise for help.  Many sleepless hours and days later, we successfully recovered the truncated data with the help of two other external teams, one including the author of the only tool that can actually do this kind of arcane recovery.

Yes, this employee is still with us, and here’s why: when exceptions like this occur, what’s important is how we react to the crisis, accountability, and how hard we drive to quickly resolve things in the best way possible for our customers.   I’m incredibly impressed with how this individual reacted throughout, and my theory is that they’ll become one of our legendary stars in years to come.

I was even more impressed with how the entire tech team became hell-bent on fixing things.   We couldn’t have slept less.  This sentiment was later echoed by a number of the other residents at our open floor-plan office (Founder’s Co-Op headquarters).   Both Keith and I heard remarks about what a great team we have, many coming from other co-founders.

With that, there are a number of folks that deserve thanks and recognition for going above and beyond to help us through this colossal recovery last week:

  • The BigDoor team
    • Collin Watson, diver-in extraordinaire and speaker of tongues
    • Lee McFadden,  API entry-point and big fan of NoSQL technologies
    • Brian Oldfield,  insomniac automator
    • Harley Holt, unflappable fire-eater
    • Keith Smith, megaphone and seer of forests
    • Ring Nishioka, Roy Schmidt and the rest of the BigDoor team
  • The BlueGecko team : Sarah Novotny, Mike Hamrick, Patrick Galbraith and Jonathan Nicol, thank you for your on-call DBA and operational expertise
  • The Percona team :   Aleksandr Kuzminsky and Baron Schwartz, thank you for your on-call data recovery expertise
  • Geoffrey Nuval from DevHub/EvoMedia, thank you for being a great customer, partner and for your continued confidence and support
  • Jim Banister from SpectrumDNA, thanks for being a great customer and for your confidence and words of support
  • Elvis, for reasons that are self-evident.

I hope that this small bit of color helps to convey how seriously we take our customers and our uptime here at BigDoor.

There are a number of things we’re doing to prevent this from happening again.  It should have never happened to begin with, and as the CTO it’s my responsibility to make sure we take the time to implement proper safeguards.   For that failure, my most sincere apologies go out to our customers and theirs, as well as to our board and investors.   We’ll make it up to you, in the coming releases of our kick-ass platform.  In fact, the same day that we called the recovery complete, the team worked late into the night in order to do a feature release (the Badge-O-matic , amongst other things) – so we’re already delivering on that promise.

–Jeff

Holiday? What Holiday?

Posted in: Blog, Development, Startups, Success on July 5, 2010 | No Comments

As a kid I recall reading a story about Thomas Edison being so wrapped up in his work that he would often times forget to eat.  I remember thinking that it would be absolutely wonderful to someday find something that I care that passionately about that my focus on my passion would drive me to forget the basic essentials of life.  When I started my first company (over 15 years ago), I quickly realized what that wonderfully passionate focus felt like.

And it’s still there with company #4, but not just with me – with everyone else on our team as well.  Last Friday as we were working through a few issues, I had a thought that maybe we should have a day off given that it’s America’s birthday and all.  I mentioned something about it to our development team and all I got were a bunch of stunned looks.  You would have thought I had just suggested we swap our dev environment to .NET or something.  After a brief pause, they each explained to me a few of the long list of things they planned to complete over the weekend and rollout on Monday and then we moved on and the conversation was forgotten.

This morning as we were all cranking away in the office, our wonderful (and apparently more considerate than me) HR person reminded me that today is an official (and documented on our Holiday Calendar) company holiday.  That’s when it struck me how amazing our team is.  Not because they’ll work themselves to the bone – that doesn’t help anything – but because they are so incredibly passionate about what it is we are doing that other details seem to suddenly matter less.  That’s when you know you are onto something cool, and when you know you have the right team to execute on the plan.

And just so you don’t think that I’m some sort of a-hole, slave driver (I am, but I don’t want you to think that); I had payroll add a day of accrued vacation for everyone who “forgot” and showed up for work today.  Now I’ve got to get back to testing…the dev guys have more stuff to rollout.

Everything I need to know about APIs I learned from my Mom

Posted in: API, Blog, Development, Game Mechanics, Our Thesis on May 31, 2010 | 2 Comments

The early API instructor.

While I may be a bit biased, I happen to think that my Mother is one of the most amazing people who has ever walked the face of this earth.  She’s also a bit technically challenged (my 13 year old daughter regularly teaches her “new” things about Grandma’s iPhone).  Yet despite some periodic technical ineptitude, it’s safe to say that everything I ever needed to know about an API I learned from her at a very young age.

  1. Hurry up. Mom used to tell me, “You are slower than the seven year itch.”  I never quite understood what that meant, but I knew that her saying it needed to result in me significantly increasing my speed.
    Modern day API application: A slow API is a crappy API.  Make sure it goes fast (and then make it go faster).
  2. Keep track of where you put stuff. If I lost track of my toys, they were quickly rounded up and placed in the “Saturday Box.”  This meant that I couldn’t play with them again until Saturday.  Amazing how the dozens of other toys in the house would suddenly become very uninteresting as I sat on the edge of the Saturday Box and counted down the days until I could again play with that once forgotten toy.
    Modern day API application: Document your API.  If the code is written but not documented, it doesn’t exist.
  3. Be consistent. When Mom counted to three, she didn’t stutter and didn’t slow down.  If she told me she was going to spank me when she got to “3” – you can bet your ass that mine was going to be sore.
    Modern day API application: Do what you say and say what you do.  Anything else leaves external developers scratching their heads and wondering if you are really serious or not.
  4. Don’t talk to strangers. Mom warned of this so often that by the time I headed off to first grade I assumed everyone who drove a windowless van was loaded with candy and missing a puppy.
    Modern day API application: Because our API relies heavily on security, every PUT, POST or DELETE call has to be securely signed (and you can decide if you want your GETs to be signed as well).
  5. Communicate clearly. Mom used to say, “Use your words.”  Apparently she thought that my series of whines and grunts wasn’t going to be an effective communication strategy for me as I got older.
    Modern day API application: Using consistent and industry standard languages, formats, response codes and protocols is critical for a friendly API.
  6. Play nice with others. Mom required that I stop doing things like hair pulling and biting at a fairly young age.
    Modern day API application: Make sure you can plug into other systems and apps without causing harm.
  7. Always make your bed. Mom was a stickler for neatness, and an unmade bed represented all that was evil in the world.  She is the only person I’ve ever known that makes her bed before checking out of a hotel.
    Modern day API application: An API needs to be neat and tidy, and the documentation needs to match.
  8. Don’t bring more than you need. As a kid we had a little, green, Datsun station wagon – with little being the key word.  I would show up ready for a trip with an armful of toys, and Mom would dutifully go through each one and make me defend my reason for bringing it.  The two or three toys that made the cut would fit easily in our little car.
    Modern day API application: Adding features for the sake of features just clutters your API and makes it more difficult for external developers to understand.  Reduce the clutter, and perfect those features that you can make a real case for.
  9. Take naps. Mom had a strongly held belief that naps could cure just about anything that ailed me.  She was generally right.
    Modern day API application: Be RESTful.
  10. Go outside and play. Mom’s rule was that the TV was never on if it was light outside.  “Play a game”, “have fun”, “go outside” – these were common refrains in our household.
    Modern day API application: Just about everything you need to know about your business resides outside of your walls.  An API is a great way to “get outside” and as for playing games and having fun – that’s the entire purpose of our API.
Beginning UI Design

Posted in: Blog, Development, Technology, UI on December 17, 2009 | 2 Comments

And now another word from the developer side of the house:

With many years of experience in the Microsoft static development world with all their fancy programs and business models I have found it quite difficult to make the initial pivot into the dynamic world of Python and Javascript. That being said, it has been quite exciting to not have everything handed to you on a silver platter. This post describes the initial design and structure of the code behind a flexible UI.

Javascript

In the Javascript world, one of the big points I keep hearing is that Javascript needs to be written from the prototypical mindset. As I have learned in past experiences, it almost always less of a headache to not fight the design of the paradigm but instead learn it’s intricacies and use it to your advantage. Hence, I have slowly learned how to incorporate duck typing to accommodate a style of coding that enables other developers to plug-in to the same objects being passed around and extend the current system.

With this belief in mind, I have come to a point where I believe the best course of action is to use a classic OOP design. The hardest part of implementing inheritance is that I constantly wonder if there is a way to be just as efficient in the dynamic/prototype paradigm. For most developers this is quite hard to put aside due to curiosity and performance improvements but my first lesson of operating in a startup, has been to focus on glaring, rather than possible problems. A difficult task but there are more interesting problems to solve at this point.

Events and State

One of my favorite parts of Javascript and Python is the ability to pass functions around as variables. Growing up in the Microsoft world, this was quite difficult to do with type restrictions on delegates. It was still possible but not nearly as easy. (It will be much easier with their upcoming dynamic keyword.) Seeing as how we need a way to link the state of a control to a particular action which occurs on the page (not necessarily attached to the state of the dom), I needed a way to call multiple functions when one action occurred. Basically, I needed Javascript to support event-driven design.

At this point all the event manager does is use an associative array to store function signatures in an array and then sequentially call those functions when the event is raised.


var onrender1 = function() { alert('Render receiver 1'); }
var onrender2 = function() { alert('Render receiver 2'); }
var manager = {
events: {},
    trigger: function(event) {
        for(e in this.events[event]) { this.events[event][e](); }
    }
}
manager.events.rendered = [onrender1];
manager.events.rendered.push(onrender2);
manager.trigger('rendered');
// Render receiver 1
// Render receiver 2
 
Download [zip]

Finally, the manager could be designed as a class to enable instances and attached to objects, allowing objects to enable events using duck typing if necessary.

This is only a basic implementation but works for are requirements quite well. It might be better to implement in the prototype model at a later time…but that’s for another time.

The entire purpose of enabling events is to call functions as soon as something happens This is extremely important when there are multiple controls/components whose visual state depends on a particular action (user initiated or not). A model similar to this is the UI state manager in Silverlight which handles events as well as supports dynamic transitions.

Personally, this is an exciting and interesting beginning to the UI development cycle.

- Collin