Back to Basics

We live in a very demanding world with so much focus on performance and results and value sometimes it’s easy to get distracted from what really matters and spend too much time chasing those ‘value add’ activities.  It helps to remember the reason that they’re called ‘value add’ is that they add onto something – and that something is the basics.

The basics are what businesses succeed or fail on; cash flow, recruiting good people, cost control, legislature – rinse and repeat.  The basics are what all sports teams succeed or fail on [look at rugby]; contact time, passing, tackling – rinse and repeat.  There are basics to every organization, every team and every role.  To be successful at anything you need to know what your basics are, then focus on them to the exclusion of all else until you have them running like clockwork and then you’ve got a really solid foundation to build on.   Then you’re in a position to ‘value add’ through whatever other opportunities you spot.

So what are the basics of building webscale distributed systems?  I believe there are six things you can never opt out of:

1. Scalability

2. Maintainability

3. Quality

4. Security

5. Availability

6. User Experience

Always do these things well and you will succeed.  I have never seen an engineering team fail because they did a weak presentation at a conference or spent too little time on R&D but I have seen plenty of fail because their software couldn’t scale or suffered massive security flaws.

I’ve decided to do a series of posts exploring these six principles, looking at simple ways you can challenge yourself every day to make sure you’ve always done the most appropriate thing for each.

See you next week.

Screen Scraping

I recently read Dare’s post about screen scraping (hey call me out of date).  Essentially he’s taking issue with privacy and the responsible use of data/credentials from one site that you supply to another.  Seems obvious especially if you’ve come a cropper through something like the Quechup thing but it was an angle I haven’t thought about enough.  You see we struggle with screen scraping for a whole other reason…

Not being in the social networking business we never have a need to ask our customers about accounts they may have with other sites so screen scraping has only really hurt me in capacity.  We generate a lot of data on price and opinion and this is typically what people want from us.  That data is accessible through a website and, as the intention is human interaction, data viewed drives other features in the experience.  Looking at certain markets changes what’s displayed on the rest of the real estate (marketing donchaknow) and of course taking a position triggers a lot more back end activity validating orders, updating exposure and modifying accounts.  Multiply this by the superhuman number of hits only screen scraping robots are capable of and it adds up to a lot of load.

Hence an API.  Making this available programmatically helped (although not with what Dare’s worried about); without the requirement for an engaging user experience and personalization/cross-sell etc the same data and functionality can be delivered with a significantly lower system impact.

The Anti-Spec

Delivering a feature is about so much more than just coding up the functionality – you’ve got to consider how it scales, how you’ll test it, monitor it, how you’ll release it (hot!) and maintain it after that (hot!) and how it might be misused.  You’ve got to consider the impact on other areas of the system – if your feature gets really popular (or hangs) what will it do to any common data or network it shares with other features?

These are requirements that are [almost] never documented but [almost] always expected.  If you’re reading this then chances are you’re used to building software from user stories – your product owner expresses their desired results by describing a user’s journey through the system.  What won’t be in their spec is a journey they don’t want the user to take.  Users are tricky little rascals to predict – what if they enter letters instead of numbers?  What if they arrive at your page via here not there?  What if they hit your feature 1000 times/second instead of 1?  What if they clear their cookies after logging in but before pulling some prices?

I think we should call this the anti-spec, the reverse of the spec – what goes unsaid but is implied but what is.

But that’s all about how you put together software.  I’ve recently seen a post with some deft definitions of “done” that encompass this philosophy (unit tests done, code simplified) and go right through deployment and beyond (released, generating revenue).  That extra stretch is always good to focus on and it keeps you thinking about why you’re building software in the first place!

Many Happy Returns!

Sometimes I wonder what the etymology of that phrase is – what’s getting returned happily?  Why was it missing in the first place?  While I’m sure a quick Google search would yield an answer I think it’s kind of low on my list of priorities and besides, sometimes a little mystery in life is nice.  Anyway, I haven’t posted for a while as I’ve been too busy doing stuff to write about stuff.  Perhaps I should resolve to do something about that in the new year?

Speaking of resolutions I have made a few; without boring you with the compulsory [unachievable] ones like eating better and getting more exercise I’ll run over the 2 key professional commitments I’m making for next year.  They came up after I recently spent some time looking over our fantastic results from 2007 and, to stave off complacency, thought about what behaviors I could emphasize to make sure we build on this in 2008:

1. Communicating plans and strategy

2. Spending more time with individuals

The first one is important because if we know more about the direction we can make better decisions – so that as we deliver what we need today we’re also laying the groundwork for what we want tomorrow.  To me this also covers better communication of what our engineering philosophies (rules) are and more frequent contact about how we’re doing against our KPIs.  All engineers get a buzz out of knowing what the plans for their products are, they feel trusted and confident in the future of the organization.

The second one is important because it completes the feedback loop into plans and strategy – we need to know what’s working to continually adjust our strategy; after all, only half of it is about how we exploit our markets and the other half is about how we best equip our guys to be successful.  You should rely on other communication channels but beware of being trapped in lofty heights and disconnected from reality – it is good for teams to be able to tell you what they think first hand and you’d be amazed what you learn!

Merry Christmas and see you in 2008.

Shark got your leg? Try some of this SOA!

SOA is some kind of miracle cure where I work – you wouldn’t believe what it will fix; unplanned downtime, project durations, rollback times, blunt force trauma… some of this may even be true but even technologies’ second coming arrives fresh from the heavens with some downsides.  None of which are insurmountable and in fact many are preferable to current issues (like an acute case of shark for example) but it’s important expectation setting to know what you’ll be up against.  Oh and maybe you can be a little bit prepared too.

If, like us, you’re going from a centralized to a decoupled system then the first things you’ll come up against are data obfuscation and difficulty in testing.  Taking data first, because you don’t present data via a service so much as a particular use of that data you’ve got a bit of a challenge on your hands when you want to come up with another use for that same data.  You also might not have a uniform data model so unless you’re the team owning the service in question you will have a hard time even determining what else could be done with that data.  Dealing next with testing, whether it’s easier or harder along the distributed/monolithic continuum depends on what you find the most difficult.  There are certainly aspects that are much easier, for example you have significantly reduced regression test requirements and you can also take foregranted changes to your data schema provided your never change your API.  This comes with a few penalties though, for example when trying to orchestrate particular test conditions (such as what happens when I have taken a symbol at a certain price but need to fund before I can make the trade?) you need to ‘preset’ the responses of a whole lot of other services to create that exact scenario – and guess what – you don’t have direct access to that data…

Decoupling your system is a bit like selecting an SDLC; ultimately there is no silver bullet and you’re simply choosing between different benefits.  Seems obvious but it’s one of those ‘if nail use hammer’ basics we tend to loose sight of – if you want to build a distributed system solution first make sure you’re looking at a distributed system problem.

Internal Systems

Life is full of strange coincidences; just after a recent post about strategy in general I was asked to consult on internal systems strategy in our organization.  I’m not sure what they wanted was strategy per se but they did need one (and got it) anyway; the question was how do we take control of, de-duplicate and consolidate all the internal tools and systems we’ve got?  My answer was are sure you want to?

It’s probably a good idea to start by defining what we mean by internal systems as it’s a different pony for different riders.  To us, internal systems are the software and hardware that supports the business of the business – we throw accounting, legal and HR tools into this category by the scruff of the neck.  Given what we call internal systems you can deduce we’re basically supporting pods of professionals.  One thing all professionals have in common is they require a seriously narrow band of functionality and this is catered to by a small number of vertical LOB software providers.  Another thing they have in common is they are, more or less, all members of industry bodies and associations dedicated to their profession at which they inevitably discuss which accounting package is best, the new features of the dental patient booking system or the benefits of such-and-such a legal document filer.

To put the above paragraph succinctly they work in unique, highly specialized fields and they are best placed to select the software that supports their activities.

So why do we want to interfere with this natural course of events?  I don’t know, call it our basic instincts for centralization, standardization and uniformity.  So here is what I say, instead of trying to guide/control their individual solutions focus instead on integration, access and self-service; you’ll have an easier life, happier internal customers and more appropriate solutions.

Integration is key because that’s how you get engaged in the first place – yes these guys know their functionality best but they [generally] have no idea what kind of server it needs, how much disk space it will consume or who will be able to support it.  If you concentrate on providing this expertise in a fast and supportive manner they’ll always come to you and you won’t have to worry about rogue solutions popping up all over the show.

Access is important because, like integration, you are coming to the party as an enabler – for this one think single sign on, fast client side deployment and quick support turnaround.

Self-service is vital in my opinion because you want these guys to be as self sufficient as possible, after all time not spent looking after a payroll system is time spent on revenue generating products.  Another reason I believe in self-service is the power shift we’ve seen where technology is now very driven by user demand; we’ve come a long way from the ‘take it or leave it’ attitude of presenting simple applications and hoping users can achieve something with them.  This is a huge topic in itself so I’ll just mention how that manifests itself in our organization – people use excel and guess what, we don’t like to store data in spreadsheets.  Instead of trying to force them into adopting some kind of specially-engineered data querying tool why not just provide them a simple ODBC interface into the data they need to work with?  They’ll end up with the presentation they’re most proficient with and you get to keep your data in a nice database.  Everyone is happy.

If only our customer facing products were so easy!

The “S” Word

I get involved with (sucked into?) strategy quite a lot and one thing that’s become clear to me over the years is that strategy can be as big a job as you want it to be.  You need a strategy because you need a decision making framework that will help you prioritize what your resources tackle, choose from proposed options and spot opportunities.  Some companies have whole teams dedicated to it, other companies spend vast quantities of time and money on external consultants and offsite sessions but at the most basic level you only really need to know two things; what you [personally] actually do and what your organization really does.

What you actually do

Knowing what you are there to do is perhaps the most commonly ignored aspect of how individuals interpret strategy in a decision making process.  The answer might seem very obvious; for example you might say you’re here to manage a team of engineers or project managers or support technicians.  I’d tell you that’s just your job, it isn’t why you’re here.  Why you’re here might be to grow margin, expand the business, reduce operating costs, launch a series of new product suits etc and of course anyone at any level contributes to those goals in their own unique way.  If you know what they are you can start delivering on them but if you’re just “here to manage a team” then you are kind of just living for the moment.

What your company really does

Now we come to what most people traditionally associate with strategy; what your organization does.  This is a larger topic but also one of those ‘90% of the benefit is in 10% of the work’ things – you really need to state the obvious first and foremost.  “We make widgets” is a nice start but you can give people a lot more focus by talking about who you make widgets for (and who you chose not to), where you’ll sell these widgets and, quite importantly, why.  Why is a good one as it let’s you evaluate new opportunities; if a potential plan meets all the same justifications used for a current product [and there is ROI in it] then why not take it forward?

Here’s how you’d reach a practical decision by blending these two perspectives; let’s say why you’re here is to grow margin and what your organization really does is supply price data to city trading floors in the UK.  You could easily determine that manufacturing cars is unlikely to meet widespread approval at board level but perhaps extending the supply of price to Chinese traders is where it’s at; or maybe with the same resources you could package your feeds for the consumer market in the UK?  Given you supply UK traders because the market is mature and well regulated, the language barrier is low (hey they’re traders so it’s still there!) and you’re capitalized by some common investors then the Chinese option looks like it’s starting with a handicap.  Add to this the fact that you’re there to grow the margin – reusing an existing asset is efficient, you could get something to the local consumer market quite quickly, deliver it with the same infrastructure and support it with the same operational staff.  Looks like productising for private users is a winner – but what would you have done if you were there to help expand the business into foreign markets?  Or de-risk the organizations cash flow by diversifying the product set?

Clearly there are many more inputs to decisions like this but we’re only really talking about the role strategy plays.  The above scenario is also predicated on the assumption you’ve determined there was ROI in both options – strategy is about making the choice most relevant to your organization; I don’t think anyone seriously expects you to pursue ideas that never pay back…

You can pursue strategy as relentlessly as you have the time for.  The more time you spend on it the more you’re likely to be talking about how you execute your strategy rather than defining the strategy itself – that’s OK though; you need to work that out eventually but as a decision making reference you can never beat knowing what you and your company are really there for.  Everyone in an effective organization should know these basics.

When Enterprise Is Too Small

Enterprise technologies are fantastic – applications and toolsets you can combine to quickly create functional solutions.  Kind of like Jamie Oliver does technology.  The other good news about enterprise technologies is that they’re designed with big corporates in mind; so we’re talking about scaling to hundreds of thousands of users, sorting millions of lines items, distributing across cities and storing terabytes of data.  Awesome, but what if your problem is bigger than that?

We’re one of an expanding circle of online organizations whose business model and popularity creates demands far in excess of what can be delivered by enterprise; now we’re talking about tens of millions of users, billions of line items, global distribution and (not quite yet) petabytes of data.  That’s scary enough by itself but when you throw on top round-the-clock availability, multi-billion hit per day performance and some complicated technical minutiae such as massive concurrence and persistence you really have a tough challenge.  Perhaps we need a new term?  How about this: Megaprise.

One way to tell if you’re a Megaprise or not is how much of your system you’re able to take off the shelf and how much you are compelled to build for yourself.  Note that when I say compelled I mean forced to do so because it’s the only way to get the job done, not out of technical vanity or a misguided attempt to create some kind of redundant IPR.  Typically we all build our own front ends and business logic but if you find yourself augmenting this with an in-house network protocol, virtual machine or even file system then perhaps you’re there.

To me this seems to be a fast growing sector, it used to just be the ‘usual suspects’ Amazon.com, eBay and Google but now we’re joined by more and more new entrants like facebook, Flickr and YouTube – all struggling to keep pace with demand while still keeping features fresh.  I wonder how long it will be before the market for Megaprise solutions is big enough for vendors to take notice and start giving us something we can use?

Leave Me Alone – I’m Agile!

Moving an organization from a structure rooted in waterfall to a structure rooted in agile isn’t about gaining a whole lot of positive benefits and losing nothing; think of it more like a trade – you’re exchanging the benefits of waterfall for the benefits of agile.  They both come with their own limitations but if you’re making this change then chances are the principles of agile have become more advantageous to your company than those of waterfall-esque processes.  One way to make this kind of transformation as painless as possible is to accept that you’re losing some of the benefits you used to have – but that should be OK – after all; you’re making a conscious, informed decision when selecting one set of benefits over another aren’t you?

A good example of the sort of weigh-up I’m talking about (and inspiration for the title of this post) is when you move from waterfall to agile you trade away some certainty about the future for the ability to make it whatever you want it to be.

Fantastic if your business needs to be highly responsive but not so great if forecasting activity 3-5 years ahead is your biggest problem (and you are lucky enough to be in an industry conducive to that).  Even if making this trade is the right decision for you it’s unrealistic to believe anyone can get away with no forward planning at all.  In our case, the manifestation of this is project delivery planning – we kind of know the features we want to roll next and we have an intuitive hunch on how big each one is, but how do we know when each team will be finished what they’re working on now?  Timing can be important for us, features always have to drop when the marketing goes out and the operational readiness is up and of course we’re essentially a seasonal business too – that means many projects have to line up with external events we don’t control.

So how do we do that?  I am not convinced we have the best answer yet, but step one is to accept that we’ve made a tradeoff (or revise that decision); we’re saying flexibility is now more important to us than long-term predictability.  I think we’re there.  The best we can do now is develop techniques to buy back some of the predictability we traded away – as long as we can do so without impacting the flexibility we needed in the first place.  What’s working for us now is estimating user stories 2-ish sprints in advance and averaging the size of our work.  If we know, on average, we get 3 user stories per man per sprint then we can take a swift look at the backlog and the size of a team and throw a dart at a calendar for a shipping date.

Clearly that’s never going to be accurate but remember the ability to change direction and drop in last minute changes is more important to us – so actually, it’s accurate enough.

SCRUM Master Job Description

As we become more agile I’ve been going over job descriptions in the engineering organization, making sure all the duties are optimized as we change how we deliver software – one of the key roles being SCRUM master.  There are a whole lot of basics that go into this specification; such as facilitating the daily stand-ups and maintaining the burn down chart – easy – but how do you capture the essence of the role?  I took a look around to see what other people are hiring and came across this on Agile in Action:

“A SCRUM Master is like a conductor coordinating the efforts of musicians, helping them to play together. Some teams are like jazz bands, so they need a leader who encourages improvisation. Some teams are like symphony orchestras, so they need a leader who keeps everyone on the same sheet of music. Conductors have to be deeply familiar with each instrument and with the music, yet they don’t play in the band or tell the musicians what to do. They let the music provide detailed guidance; their job is to bring out the best in the musicians, both individually and as a group.”

We currently share this kind of flexible, adaptive leadership between SCRUM masters and the team’s manager (loaded to the management) but it does make sense to push more of this closer to the people that make our product happen.