Running Netflix in the (public #aws) cloud

my talk at #qconsf in Nov http://qconsf.com/sf2010/presentation/Running+Netflix+in+the+Cloud

Can OpenStack catch up with AWS? Looks unlikely to me.

There has been a lot of chatter about the new OpenStack standard in the last few days, and how it could become a third option in the cloud infrastructure market alongside Amazon AWS and VMWare vCloud. For example Randy Bias/Cloudscaling has a good overview of the players.

The real question is whether OpenStack can catch up and become viable. Amazon is always described as being years ahead of the competition and having the lions share of the market. Estimates of the size of their lead depends on who you talk to, but I don't see anyone disputing their lead. On top of that, Amazon is investing heavily, has big customers like Zynga and Netflix stretching and hardening their systems (along with a huge number of small customers), and has already added many features beyond the basic compute (EC2) and storage (S3) that have been copied by others including OpenStack.

So for OpenStack to catch up, means that they have to move faster than Amazon, and leverage the slipstream effect where Amazon has had to educate the market and figure out what works, so competitors can copy their successes. However when I look at the details of the OpenStack specification they appear to be copying some of Amazon's problems as well. In particular the account and authentication model, which does not scale for enterprise use.

At the most basic level, it is infeasible to change the security model of a platform architecture, it's one of the most fundamental starting points that conditions the layers above. Changes to account and authentication management break all the layers of applications and tools that are built on the platform. One of the first problems that Netflix had with Amazon, was the lack of sub-accounts and role based access control (RBAC), and while beating up Amazon on this point for the last two years, we have built our own platform layers and tools to compensate. As a result, we find it impossible to use any of the web consoles or tools produced by Amazon or the many cloud vendors, which assume that there is a single account owner who can do anything. We hope that Amazon will eventually support sub-accounts, and when they do, I expect it will break everyone's tools.

At this point OpenStack is just the base level of the platform, it doesn't really have layers of tools on top yet, but its account and authentication model appears to be exactly the same as Amazon, so they will end up with layers of tooling that don't meet the needs of enterprise customers.

What's the difference between a startup and an enterprise? In a startup, everyone in IT knows the root password to every machine in their infrastructure. In an enterprise root passwords are carefully controlled, they change when someone in-the-know leaves, and specialist groups manage different parts of the system (Network ops can only mess with the switches, DBA's can only mess with the database etc.). The problem with a single account for the cloud is that everyone who needs to do anything to that account can do everything to it, and the common tools are oriented to a single user, managing several accounts, rather than a hierarchy of users managing parts of one account. All the systems in the account need an authentication key to access cloud services, and changing the password and key means you have to re-key every system. Get this wrong and your cloud will evaporate in an instant.

So in my opinion, a necessary but not sufficient condition for OpenStack to eventually catch up with AWS is that they need to build sub-accounts and RBAC into their spec from the start. However it seems much more likely that Amazon will just disappear into the distance from what I've seen so far.

iPhone Tethering Review

Along with iOS4 and the new ATT data plans they now also support tethering. I setup my wife’s personal iPhone 3GS for tethering, and it worked fine on her work laptop with Windows XP over the USB cable (she could work from the passenger seat while we were driving freeways). You just need to have iTunes installed on the XP machine, even if that isn’t the machine that sync’s the iPhone.

I also got Netflix to switch on tethering on my company iPhone 3GS, and I’m using it via bluetooth to my MacBook Air, easy setup and it also worked well for me.

The downside is that you lose the unlimited data plan, but for a few dollars a month, it can be very useful.

Netflix in the cloud and HTML5

One of the fun things about working at Netflix is that we are always "leaning forward" and looking for new technologies to leverage. As we roll out our website to run on the Amazon cloud, we are also re-architecting the code base, adding internationalization support and bringing in the latest technologies. One of these is HTML5, which is raising the bar for cross browser support for advanced user interface features, and is now supported by a large and rapidly growing percentage of the visitors to netflix.com. In addition many TV based devices now embed webkit, which is the HTML5 compatible technology that underpins the Safari and Chrome browsers. The user interface engineering team is looking to hire the best and brightest to work on these cutting edge technologies. Here's a write-up for one of the open positions:

Are you passionate about building great website experiences used by millions of visitors each day? Come to Netflix where we are using HTML5 based web technologies to move ecommerce directly onto televisions in our customers’ living rooms. As part of our Customer Acquisition team, you will lead the way to our internationalized television user interface designed to help new customers find Netflix and start streaming movies in seconds. This new experience will be deployed to HTML5 capable embedded browsers and served from our cutting edge cloud based backend service.

---------
Update - Hacking Netflix and TechCrunch picked up on this posting. MG Siegler at TechCrunch decided that I was talking about streaming video and Silverlight, which I wasn't. I was thinking of HTML5 features that let us build very cool user interfaces with drag-and-drop, canvas transforms etc. for the web site, and for embedded TV devices specifically. The Silverlight player is used for PC/Mac playback only, and the basic HTML5 Video doesn't have a viable DRM solution at this point. I'm the Cloud Architect for Netflix, so my involvement is to architect robust and scalable support in the cloud for these new user interfaces.

Garage Structure Completed

After lots of stops and starts, all that is left to do is some exterior paint and to fit the main doors. Work has already started on the electrical installation and water for fire sprinklers. Since the walls got so wet, they warped and parts of the siding had to be replaced. For some reason Tuffshed did other jobs on days where the weather was nice, and turned up to work on this one whenever it rained. We had what was probably the last rain of the winter on Sunday and Monday, but the roof was already tiled by then. The last video was the trusses being installed, here is the rest of the sequence.











Moving the viewpoint to catch trench digging around the back of the garage.

Cloud Architect

my role at Netflix just changed, from managing development of personalization in the cloud, to over-all cloud architect for Netflix. New challenges and a much broader scope. One challenge we have as we migrate to the cloud is that we need to build a new way to log and analyze what the systems in the cloud are doing, and we are basing this on Chukwa, Hadoop and Hive, all running in AWS. So we are looking to hire a Hadoop developer, and are looking at the research going on at Berkeley's RAD lab with interest.

AT&T 3G Microcell doesn't support corporate account phones

I just got an ATT 3G Microcell for home, since our service is intermittent, sometimes so bad I can't send an SMS message successfully (although at times it works fine). We live in the mountains, and have several different poor signals to bond to, so that's probably why it is variable.

In the store, they checked my phone number and said I couldn't activate a Microcell on my account, since it's a Netflix corporate account iPhone. However, my wife has an iPhone on a personal account, so I used that as the activation account.

However, when I tried to add my own phone number to the Microcell's whitelist of up to 10 numbers, I got this error message

"FTC309: invalid subscriber information found" - which got no hits on Google, however after this blog post it should get one..

So the website told me to call the generic ATT support number. After a few short waits, a very polite first level support guy (who had probably never heard of the Microcell before I called) found a well informed business support specialist who was able to confirm that the error code was due to the Microcell only supporting consumer accounts. She agreed that it didn't make a lot of sense, found that they had been running a trial of corporate accounts a few months ago, but didn't know when/if it might change. She forwarded me an internal support document that discussed the limitations of the trial, including the lack of corporate support. Although it does say that it supports all ATT 3G devices.

I installed it anyway, it takes a few hours to set itself up the first time. We will see how well it works for other iPhones, meanwhile I will just have to use Skype to make calls and send SMS... If it doesn't work out I will take it back.

------- Update -------
I contacted the corporate account support team via work, and they over-rode the system to add me to the Microcell. So my corporate account iPhone is now working fine with the Microcell. They are investigating further to see if they can make this a more general fix for other people, but if you are in the same situation as I was, you should be able to escalate to support to get it fixed.

Hiring a cloud platform engineer at Netflix

If you are interested in the CAP theorem NoSQL systems like Voldemort, Cassandra and SimpleDB, scalability and expect to be at the Velocity Conference in June, then you are the kind of engineer I'm interested in hiring. Netflix is aggressively moving to new technology, hiring only the very best and most experienced engineers and our growth is accelerating.

Garage Trusses Video

On Tuesday a big truck had some difficulty delivering a 40ft trailer containing the roof framing trusses, but it was raining so that is all that happened.

On Wednesday 21st April it started out dry, rained for a while then cleared up, but they did manage to put up all the trusses, and I made another stop-motion video of the work.

Tuffshed aren't great at letting us know what is going to happen next, we have to call them or ask the workers. I think the next step is to put plywood sheeting on the roof, then get an inspector up to check that the nailing pattern is correct, then they can put the doors and windows in, tile the roof add the external trim and paint it. It should be much warmer and sunny for the next few days.

garage - roof and rain on Tuesday

the roof trusses will be delivered tomorrow and rain is forecast for Tuesday and Wednesday - so we will see how they deal with that... it sucks that we don't have a roof before the rain.

[Posted with iBlogger from my iPhone]

Garage Framing Walkaround

The walls are up, but the trusses for the roof aren't ready, so it will be finished off next week. Luckily the weather looks good for the next week or so.

Garage Framing - first day of construction

Tuffshed turned up with their pre-built sections, assembled them and added some siding and the main door frame. Another stop-motion video on YouTube. It was wet over the weekend, and the ground is very soft, but we have a few dry and warm days in a row forecast, so I'm hoping it will dry out now.

Experiences using an iPad as my primary work laptop

I spent the last week carrying my iPad with me to meetings and using it as my main system at home an in my cube. My laptop was used for a few things, like the performance monitoring tool AppDynamics - which needs a bigger screen and is flash based, but just about everything else was usable on the iPad.

I have the apple case, which I think is essential, as it holds the iPad up at the right angle to use on a desk or lap, as well as protecting it and making it less slippery to carry around. I got the Video out cable but haven't had a chance to use it yet. After intensive use, the battery is still at 60% or more each evening.

I like the speed at which apps like mail and calendar come up, and they are both very nice to use, I prefer them to entourage on my Mac laptop. I spend a lot of my day reading writing and deleting email. Safari is also fast and worked well for gmail. I imported a PowerPoint presentation to keynote on the iPad and it needed adjustments to font sizes in diagrams, and lost some color coding, but seems like a usable tool. I will switch to keynote on my Mac as well and hopefully it will be compatible.

I have a VNC app, but moving the mouse cursor around is a a pain so it's only useful for emergency access to my laptop.

The other app I use a lot is NewsRack, which tracks about 100 random rss feeds for me by keeping in sync with google reader. I get most of my news and keep up with friends blogs this way. Compared to the iPhone version it seems to update much faster and the extra screen space is well used. The BBC app provides a more visual way to track stories, and the Guardian Eyewitness photography app contains stunning images, it's one of the most popular free apps at the moment.

Typing on the iPad is better than expected with two caveats. I keep hitting the b or n keys when I mean to hitbspace (like that), and the auto correction system is a bit over-active so I have to proofread what I write more than usual.

Of course I have been watching movies with the Netflix app, and have used the eBay and Pandora iPad apps, which look good. I'm still using the iPhone version of echofon as my Twitter reader, but I will try out the official twitter app when it comes out.

Garage - Pouring and Finishing the Foundation

Last Friday we poured concrete, and then it poured rain. Here's a stop-motion video of the work. Six concrete trucks came, we are high on the mountain, so they may not have been carrying a full load each time. The foundation is very deep on the downhill side, and the slab is 6" deep rather than the 4" minimum used for the walkway down the side and the parking apron.

The first few seconds of each video look messy, this was introduced when youtube encoded it, my original video is fine.



On Saturday, the weather was better, and the team came back to finish the surface and tidy up.



Final bill was about $25K, and I would recommend Amaya Concrete of Morgan Hill, they did a good job.

This week, we let the concrete cure, wait for the ground to dry out and do some basic grading of the ground levels and slopes. Next week the building will be delivered and installed.

IPad makes it hard to embed YouTube in blog

I'm trying to compose a blog post that embeds a YouTube video, but YouTube doesn't want to give me the HTML fragment to embed the video. If you browse to YouTube on an iPad it doesn't show any options to pick up the embed because it renders in "touch" mode. At the bottom of the page I found an option to switch to desktop mode and get the full page. This renders an embed option but doesn't let me select it to copy here.

So I need to use my MacBook to compose posts.

As the iPad moves the boundaries between content consumption and content generation, people need to rethink the functionality of the mobile and iPhone optimized web sites.

Garage - Inspected and delayed

We passed inspection by the county this morning, but the weather has turned too bad to pour concrete tomorrow, so we are postponing until after Wednesday's storm has passed, and will resume work on Friday. Today the electrician finished his work on the pipe that carries the power, and the Tuffshed foreman Mike visited to inspect the site and check and adjust the position of the hold-down bolts that are set into the foundation.

My CMG paper on Crunching Data In the Cloud is published in MeasureIT

The slides are also available at http://www.slideshare.net/adrianco/crunch-your-data-in-the-cloud-with-elastic-map-reduce-amazon-emr-hadoop - and MeasureIT has a PDF of the whole paper at http://www.cmg.org/measureit/

Garage - Working on Saturday

The concrete team were back today, putting in more rebar and tidying everything up. Chris Ramoutar the Electrician was also here putting in a 2-inch pipe to carry power to the garage. I'm having a high spec 220V and 200Amps dedicated circuit configured so that it's ready to recharge electric cars in the future. I got a work-out clearing up the felled fir tree and chipping the small branches, and collecting large rocks that had been dug up, and arranging them around the pond.

Here's a view from the deck, you can see the apron on the right, and the garage foundation on the left, and Chris working in the trench that takes the power line.



Here's a view from where the driveway will be as it joins the apron, you can see the walkway to the left of the garage.



Later that evening some very confused deer wandered over the foundations, since it has been built over their usual path from the forest out to the front yard.

Garage Site - rebar ready

Another stop motion video at 30s intervals shows most of the day's work. The tree that was falling over was pushed back the other way and cut up, and they added the parking "apron" in front of the garage.

There is a 4ft wide walkway along the side of the garage by the house, so the main pad is 36 by 36ft. It slopes to the right by three inches over that width so it will drain. The walkway slopes away from the garage by one inch. The apron is 26ft deep, 36ft wide and slopes by 12 inches (4%). The driveway will connect to the side of the apron that is nearest the house.

Garage Site - digging the foundations

Today the new leach-field pipe was inspected, and the foundations were dug, I did a stop-motion video using iStopMotion on an old MacOS laptop looking down from the house, and most of the day's work is compressed into one minute.



Tomorrow they will do the detailed preparations for the pour on Monday of the garage slab itself (6" thick), perimeter foundation (at least 18" deep, more on the downhill side where it is on in-fill), a walkway down the side of the garage, an "apron" in front of the garage to park on, and pads for the propane tank and generator.