Skip to content
July 2, 2010 / cohodo

Velocity 2010

Two Planes, an IT-related Sitcom, and a Shuttle-related Ruckus (a Shuckus?)

After setting off from Digbeth coach station some 25+ hours earlier, 2 tired Talisians (Matt and me) finally arrived at the Hyatt Regency, Santa Clara, California for O’Reilly’s Velocity 2010 conference for Web Performance and Operations. We first flew from Heathrow to Dallas Fort Worth on a surprisingly cramped American Airlines flight, then onto San Francisco; by the time we hit SFO we were looking forward to getting to the hotel to freshen up and sleep. Night had fallen as our plane approached SFO which made a nice change after nothing but daylight since waking up in Birmingham well over a day ago. Not a journey for vampires, I can tells ya.

On the flight from Dallas to San Francisco I sat next to Patrick Wilson, CEO of http://www.vitalsignstechnology.com/ who was on his way back from a conference in Florida. The Valley and Bay Area are chock full of people involved in one area or another of the Tech industry; you can’t swing a cat without hitting a developer, an ops guy and a couple of venture capitalists. And probably doing some damage to the cat.

IT Crowd's "Moss"

Patrick has a pretty extensive knowledge of the area and served as a high altitude tour guide of sorts as we made our approach over the lights of San Francisco (“There’s the Googleplex. That complex there is Sun” and so on) . He also revealed a recently acquired taste for the Channel 4 sitcom “The I.T. Crowd”, even going as far as to fire up his laptop and share an episode.  I was more than happy to help him with his query as to the “exchange rate between pounds and quid?”, given his obvious delight at one of our most surprising cultural exports.

We knew a cab from SFO to Santa Clara would run us well over 100 dollars; Irish genes kicked in and we started looking for a cheaper alternative, despite our by now zombie-like state of fatigue. At this point we discovered the magic of SFO “Shuttles”. These are basically big truck-like minivans that provide a (relatively) cheap way to get from the Airport to anywhere in the general Bay Area. When enough people want to go in roughly the same direction, you jump in and off you go.  Cheaper than a cab and organised to cover broad geographical regions e.g. South Bay area, there is an element of pot luck controlling how long it will take you to reach your destination depending on the route the driver chooses in order to encompass all stops.

SFO Shuttle

SFO employees are on hand to ensure fair play from Shuttle drivers. The lady we spoke to attempted to find a way to tactfully express the fact (“…it depends on…err… the… size of the people…”) that a family of very large Americans waiting expectantly inside the shuttle we needed meant there should really be 2 spare seats, but realistically there was now only 1 seat with access to a seatbelt. This was causing a problem.  Presumably potential litigation in the case of an accident was a worry. A brief Mexican (or at least Californian) stand-off ensued. SFO lady went to find a supervisor.

The driver attempted to slyly squeeze us in anyway now SFO lady’s back was turned. Oversized family took immediate exception to this blatant disregard for their safety (if only they showed the same concern for their health when passing a Dunkin’ Donuts). Dummies were spat, rattles were thrown, they retrieved their baggage and stormed (actually more like waddled) off in a huff (“You ****** up, Buddy! **** you!”). Too tired to find this episode as amusing as we really should have, we slid into the space vacated by their ample collective bulk and an hour or so later we were in Santa Clara, via Oakland. The driver was relying heavily on his sat nav, which caused some concern when it failed spectacularly. He switched it for an identical one that worked, prompting lame, tired jokes from us about redundancy and switching to a warm standby. First performance lesson of the trip: geek humour degrades dramatically after 26 hours without sleep.

A Serendipitous Meeting between Scribes

On Sunday we had dinner with fellow Velocity attendees Patrick Debois, and Torben Graversen. Matt met Patrick at Puppet Camp a few weeks ago. Patrick is the originator of the term “Devops”; his blog is highly recommended. As we chatted in the bar afterwards, Sean Power approached and asked if we were here for Velocity. He introduced himself and his friend Tracy Lee.

We chewed the fat for a while and talked lean start-ups, performance monitoring, Silicon Valley, and Sean’s upcoming talk at Velocity. Sean mentioned that he had contributed to an O’Reilly book that was due out any day now; Patrick asked which one. It turned out to be “Web Operations”; this book contains a chapter on Monitoring written by non other than one Mr. Patrick Debois! You could have knocked us over with an O’Reilly “In a Nutshell” book, such was the strength of the minor coincidence.

Lack of nerve scuppers a tour of Twitter (or possibly just a beating from security)

Since the conference didn’t start until Tuesday, we hired a car on Monday and drove to San Francisco. Whilst there we met with a couple of friends of Patrick, one of whom was something of a veteran of the Bay Area Tech scene. He told us he had calculated that there were at least 400 tech companies in a 3 block area of San Francisco; pretty mind blowing when you think of it. He also informed us the Twitter offices were just around the corner and suggested we should go in, ask at reception for his friend John Adams, tell him we were here for Velocity, (“Mike told us to ask for you”) and see if he would give us a tour.

We were all pretty surprised to find we were able to stroll into the Twitter building unannounced, take the lift up to the 6th floor, wander into reception and hang around without anybody once challenging us; I kept expecting to be thrown out any second. Ultimately, we were all far too European and reserved to ask for someone we didn’t know, tell him we were friends of somebody else we didn’t really know (“Does anybody know that guy Mike’s surname?”), and cheekily ask for a tour of Twitter, so we just hung around for a bit looking goofy and then left. So much for the meek inheriting the Earth; we couldn’t even blag a tour of Twitter.

Down to Business

Velocity traditionally covers two broad areas:

  • performance of Web applications
  • operations

At first glance some folks may not see the connection between these two topics, but they are increasingly intertwined as engineers seek to build highly available, scalable and fast applications that operate at Internet scale. Here in Platform Engineering & Operations, our development and operations functions work together closely in the same team, so it made perfect sense to us that these tracks had been combined into a single conference looking at performance in a holistic way. It also seemed fitting for us to send one developer and one ops person.

The conference was sold out, with over 1200 people in attendance, and up to 3 tracks at once at various times. Between the 2 of us, we tried to arrange our schedules to cover as many of the presentations and sessions as we could. Some of the sessions were billed as “workshops”, but in reality they were way too big to be anything other than long presentations; 400 people is far too many for anything “workshoppy”. Nevertheless, the content was generally of a very high standard; informative and well presented.

DO go chasing waterfalls…

Quite a number of sessions focussed on optimisation of applications that are delivered to the browser. Although not a problem we face directly in delivering the Platform (which is an API); this is an area that has come on in leaps and bounds over the last couple of years and it was interesting to see the current state-of-the-art. Annie Sullivan of Google gave a very good presentation covering many of the techniques engineers turn to when tuning performance of their web pages from the point after server-side processing is complete.

Waterfall charts are a common tool for analysing performance; during the course of the conference we saw many variations created by assorted tools including Webpagetest, Google Page Speed, DynaTrace, WebKit, Gomez, and a host of others. In fact, Steve Souders mentioned that there are twice as many of these type of tools as there were this time last year, which underlines the growth of this area of performance tuning.  Performance is arguably more important now than ever before, even Google page ranking is now partially dependent on the speed of your site.

Techniques mentioned by Annie and others included various ways to optimise and minify JavaScript, CSS & HTML, including getting JavaScript into a build system to help you identify dead code, code that can be modularised, and code that could be loaded asynchronously. Asynchrony, along with progressive rendering techniques to ensure the most important parts of the page load first also featured heavily in an exploration of how Facebook made their site twice as fast.

Engineering for the win!

One of my favourite presentations came from Theo Schlossnagle. Theo’s “Scalable Internet Architectures” was 90 minutes of wisdom covering a vast array of material, from analysing network packet size, to choosing between SQL and NoSQL databases, to version control, caching, monitoring, service decoupling, mastering tools and the importance of engineering maths. A truly wide-ranging and ambitious presentation, skilfully delivered. Unfortunately there appears to be no video, so you can’t really appreciate the moments when Theo worked himself into a righteous engineering rage as he dismissed various bone-headed architectural decisions. However, the slides are still well worth a look.

How do they do that?

Undoubtedly the most over-subscribed session of the week was “A Day in the Life of Facebook Operations” by Tom Cook; I literally had to watch this one while standing in the doorway to the lecture theatre. The room was full to bursting point and Tom did not disappoint. The sheer scale of the job at Facebook is daunting; more than 400 million users, 10s of thousands of servers, 300+ TB of data served from RAM alone via Memcached, and multiple software releases and configuration changes every single day across this gigantic stack. A great example of operating on a massive scale and yet still moving quickly and keeping risk small.

Similarly popular and insightful were John Adams’ “In the Belly of the Whale: Operations at Twitter“, John Allspaw’s “Ops Meta-Metrics: The Currency You Use to Pay For Change” and Paul Hammond’s “Always Ship Trunk: Managing Change In Complex Websites”. All of these presenters have real, in-the-trenches experience of managing development and operations in very large, very fast moving Web applications, servicing mind-boggling numbers of users via staggering amounts of code and infrastructure. Much can be learned from them.

Dev what now?

I have become increasingly aware of the Devops movement over the last few months. I believe this kind of thinking has the potential to change the face of Operations the way agile approaches have changed Software Development over the last 10 years, so it was good to see it well represented at Velocity. I particularly enjoyed Andrew Shafer’s “Change Management: A Scientific Classification”, which sounds almost like it could be espousing a very buttoned-down, paperwork and process heavy approach to managing change, but in fact stresses the importance of agile thinking (high-bandwidth communication, version control, small changes deployed regularly and monitored heavily, automation and configuration management tools) in safely managing change. Adam Jacob also touched on Devops during his innovative “Choose Your Own Adventure” session.

There is no spoon

Wedged in amongst all the good stuff on performance in the browser there were a couple of sessions that took different approaches to looking at performance. Firstly, there was Yahoo Search’s Stoyan Stefanov with “The Psychology of Performance”, offering fascinating insights into how humans perceive the duration of various things and what that means for web applications.

Secondly, Neil Gunther and Shanti Subramanyam used performance testing analysis of Memcached in “Hidden Scalability Gotchas in Memcached and Friends” to introduce Neil’s Universal Scalability Law and explain what mathematical modelling can do to help performance tuning in the Brave New World of multi core machines. This was truly eye-opening stuff; the material was accessible enough to pull you in, but deep enough that I will be digesting bits of it and delving into this further for a long time to come. It was also good to see that server-side performance was being addressed at Velocity, albeit on a much smaller scale than the browser-side.

A recurring theme for me was the additional material Velocity has pointed me towards; the performance-related blogs of Neil and Shanti being great examples.

What has all that got to do with the Platform?

Common high level threads amongst all these cool kids on the Tech block were being process light but review heavy and making frequent small changes with enough testing, automation and monitoring around them to keep the risk of change minimal, yet keep the pace punchy. I found it encouraging to see how much of this stuff we already do in Platform Engineering & Operations, e.g.:

  • version controlling everything (Subversion and Git)
  • always shipping trunk
  • using configuration management tools (Puppet)
  • stressing peer review
  • extensive automated testing (J-Unit, Grinder)
  • monitoring and alerting (Ganglia, Nagios, Cacti, etc.)
  • Continuous Integration (Hudson)
  • dark deployment
  • service decoupling
  • using switches in code to enable/disable features
  • frequent small releases
  • appropriate use of asynchrony
  • judicious use of cloud technologies (EC2, S3 and various other bits of AWS)
  • having ops and devs work closely together.

We don’t yet face the problems of scale that have led Facebook and Twitter to turn to BitTorrent as a means to roll out software quickly to thousands of servers (that would be one of those “nice” problems to have, given what it would represent in terms of take-up of the Platform), and we have some way to go before we can truly say we deploy continuously. However, I left feeling confident in the way we work, primed with new areas for us to explore, and inspired at having gained an insight into how some of the leading lights of Internet-scale engineering make it all hang together.

Advertisements

2 Comments

Leave a Comment
  1. Michael Fitzmaurice / Jul 8 2010 3:06 pm

    Nice summary of some of the content of velocity focussing on performance graphs and charts of various kinds: http://www.webperformancetoday.com/2010/07/01/the-best-graphs-of-velocity/

Trackbacks

  1. n² » Blog Archive » ScaleCamp 2010

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: