Netflix for iPhone in the cloud and HTML5

I posted on this subject a while ago, and generated a lot of confused comments, who read things into my post that weren't there. Today I hope it's a bit clearer what I was talking about, because we released the Netflix iPhone app, which is based on HTML5 for its user interface elements, back ends into the Amazon cloud, and uses conventional, non-HTML5 video playback and DRM. The playback mechanism is the same as the iPad, but to support the scale of expected usage we needed more capacity (it's currently the top free app in the app store as I write this) and we got that by rebuilding our API tier and personalized movie choosing backend to run on the Amazon cloud. The user interface runs in a webkit based browser window in the app, just like on iPad, but the entire UI is built using Javascript with advanced CSS and HTML5 animations to get it to feel like a native iPhone app.

Returning to the theme of my last post, Netflix is hiring engineers to work on cloud tools, platforms and performance, and advanced user interfaces. I think we are breaking new ground in both areas and its an exciting place to be. We have very high standards and are looking for the best people in the industry to come and help us...

Eventual Consistency of Cloud?

Lori MacVittie wrote that eventually cloud standards will converge around the private cloud standards (RT @swardley). I disagree, and there are several examples I can think of that point to the opposite conclusion, that public cloud will be the standard, and AWS will continue to dominate. The original article is here.

The first point of disagreement is the claim that public clouds aren't getting the input they need from customers to mature their services, because real enterprise customers aren't running in the public cloud. Well, here at Netflix, we are giving Amazon exactly that kind of input. We use every feature they have, we are driving them hard and Amazon is taking the input and improving their product rapidly in ways that benefit all the users of AWS.

My main disagreement is the claim that lots of individual IT departments will converge on a single standard that will win out over public cloud standards. I find this highly implausible, there are a host of vendors feeding technology to enterprise clouds, and they will all do the usual vendor thing of looking for ways to lock in the customer, even if they base on the same standards their implementations will be different. Here's an example, the "private" Enterprise Unix variants Solaris, AIX, HP-UX, IRIX, OSF/1 etc. are all based on the same Unix standards, however the "public" alternatives are Linux and BSD, and Linux has won the mind share in this space. I see Linux as an analogy for the public cloud in the sense that there is a very low barrier to adoption. For Linux, you can download it to run on any old computer for nothing, learn to use it, then build very low cost solutions out of it. For AWS, for a few dollars on the existing Amazon account you use to buy books etc, you can explore all the features and learn to build very powerful systems in a few hours. This has produced a large population of very productive engineers, who know how to use AWS, Linux and other open source tools to solve problems rapidly at low cost using the same tools. In contrast, if every enterprise cloud moves ahead by solving their problems independently they will produce a variety of architectures, each optimized to their own problem, and with their own tooling, and a very small number of people who know how to run each variant. You will also find that every company also uses the public cloud to get stuff done more quickly and cheaply than the IT department, so that will become the common standard.

Part of the thinking behind Netflix' move to the cloud is that large public cloud providers like Amazon will have far more engineers working on making their infrastructure robust, scalable, well automated and secure than any individual enterprise could afford. By using AWS we are leveraging a huge investment made by Amazon, and paying a small amount for it on a month by month basis. We also get to efficiently allocate resources, for example how much does it cost to provision a large cage of equipment in a new datacenter and how long does it take from deciding to do it, to having it running reliably in production? Let's say $10M and many months. Instead we could spend $10M on licensing more movies and TV shows to stream, and grow incrementally in the cloud. In a few months time we have more customers than if we spent the money up front on buying compute capacity and we just keep re-provisioning new instances in the cloud, so we never end up with a datacenter full of inappropriately sized or obsolete equipment. At present, Netflix' growth is accelerating, so it is difficult to guess in advance how much capacity to invest in, but we have already flat-lined our datacenter capacity, and have all incremental capacity happening on AWS. For example, we can just fire up a few thousand instances for a week to encode all the new movies we just bought the rights to, then stop paying for them until another big deal closes. Likewise on The Oscars Awards night, there is a big spike in web traffic, and we can grow on the day and shrink afterwards as needed without planning it and buying hardware a long time in advance.

While the other public and private cloud vendors are competing to come up with standards, we are finding that resumes from the kind of engineers we want to hire already reference their experience with AWS as a de-facto cloud standard. It's also easier to attract the best people if they will learn transferable skills and work on the very latest technologies.

That might sound like a lock-in, but a well designed architecture is layered, and the actual AWS dependencies are very localized in our code base. The bet on the end game is that in coming years, other cloud vendors produce large scale AWS compatible offerings (full featured, not just EC2 and S3), and a very large scale multi-vendor low cost public cloud market is created. Then even a large and fast growing enterprise like Netflix will be an insignificant and ever smaller proportion of the cloud. By definition, you can't be an insignificant proportion of your own private cloud....

Reducing TCP retransmit timeout?

Cloud networks are lossy and low latency, reducing TCP_RTO_MIN and TCP_DELACK_MIN looks like a good idea, but it looks as if this needs a linux kernel recompile. Anyone else looked at this?
Here is a relevant paper “Safe and Effective Fine-grained TCP Retransmissions for Datacenter Communication“
http://www.cs.cmu.edu/~vrv/papers/sigcomm147-vasudevan.pdf

Open letter to my Sun friends at Oracle

I recently heard about Illumos via a tweet from Alec Muffett, and responded with my own tweet "I predict that #illumos will be just as irrelevant as Solaris has been for the last few years. Legacy." - personally I haven't logged into a Solaris or SPARC machine for about four years now. There are none at Netflix.

I have also been talking to a few friends who stayed at Sun and are now at Oracle, and there is a common thread that I decided to put out there in this blog post.

This week I presented at a local Computer Measurement Group meeting, talking about how easy it is to use the Amazon cloud to run Hadoop jobs to process terabytes of data for a few bucks [slideshare]. I followed a talk on optimizing your Mainframe software licensing costs by tweaking workload manager limits. There are still a lot of people working away on IBM Mainframes, but it's not where interesting new business models go to take over the world.

The way I see the Oracle/Sun merger is that Oracle wanted to compete more directly with IBM, and they will invest in the bits of Sun that help them do that. Oracle has a very strong focus on high margin sales, so they will most likely succeed in making good money with help from Solaris and SPARC to compete with AIX, z/OS and P-series, selling to late-adopter industries like Banking, Insurance etc. Just look where the Mainframes are still being used. Sun could never focus on just the profitable business on its own, because it had a long history of leading edge innovation that is disruptive and low margin. However, what was innovative once is now a legacy technology base of Solaris and SPARC, and it's not even a topic of discussion in the leading edge of disruptive innovators, who are running on x64 in the cloud on Linux and a free open source stack. There is no prospect of revenue for Oracle in this space, so they are right to ignore it.

That is what I meant when I tweeted that Illumos is as irrelevant as Solaris, and it is legacy computing. I don't mean Solaris will go away, I'm sure it will be the basis of a profitable business for a long time, but the interesting things are happening elsewhere, specifically in public cloud and "infrastructure as code".

You might point to Joyent, who use Solaris, and now have Bryan Cantrill on board, but they are a tiny bit-player in cloud computing and Amazon are running away with the cloud market, and creating a set of de-facto standard APIs that make it hard to differentiate and compete. You might point to enterprise or private clouds, but as @scottsanchez tweeted: "Define: Private Cloud ... 1/2 the features of a public cloud, for 4x the cost", that's not where the interesting things are happening.

So to my Sun friends at Oracle, if you want to work for a profitable company and build up your retirement fund Oracle is an excellent place to be. However, there are a lot of people who joined Sun when it was re-defining the computer industry, changing the rules, disrupting the competition. If you want some of that you need to re-tool your skill set a bit and look for stepping stones that can take you there.

When Sun shut down our HPC team in 2004 I deliberately left the Enterprise Computing market, I didn't want to work for a company that sold technology to other companies, I wanted to sell web services to end consumers, and I had contacts at eBay who took me on. In 2007 I joined Netflix, and it's the best place I've ever worked, but I needed that time at eBay to orient myself to a consumer driven business model and re-tool my skill set, I couldn't have joined Netflix directly.

There are two slideshare presentations on the Netflix web site, one is on the company culture, the other on the business model. It is expected that anyone who is looking for a job has read and inwardly digested them both (its basically an interview fail if you haven't). These aren't aspirational puff pieces written by HR, along with everyone else in Netflix management (literally, at a series of large offsites), I was part of the discussion that helped our CEO Reed Hastings write and edit them both.

What can you do to "escape"? The tools are right there, you don't need to invest significant money, you just need to carve out some spare time to use them. Everything is either free open source, or available for a few cents or dollars on the Amazon cloud. The best two things you can have on your resume are hands on experience with the Amazon Web Services tool set, and links to open source projects that you have contributed to. There isn't much demand for C or C++ programmers, but ObjectiveC is an obvious next step, it's quite fun to code in and you can develop user interfaces for iPhone/iPad in a few lines of code, that back-end into cloud services. Java code (for app servers like Tomcat) on Android phones, Ruby-on-Rails, and Python are the core languages that are being used to build innovative new businesses nowadays. If you are into data or algorithms, then you need to figure out how to use Hadoop, which as I describe in one of my slideshare decks is trivially available from Amazon. You can even get an HPC cluster on a 10Gbit ethernet interconnect from Amazon now. There is hadoop based open source algorithm project called Mahout that is always looking for contributors.

To find the jobs themselves, spend time on LinkedIn. I use it to link to anyone I think might be interesting to hire or work with. Your connections have value since it is always good to hire people that know other good people. Keep your own listing current and join groups that you find interesting, like Java Architecture or Cloud Computing, and Sun Alumni. At this point LinkedIn is the main tool used by recruiters and managers to find people.

Good luck, and keep in touch (you can find me on LinkedIn or twitter @adrianco :-)

Running Netflix in the (public #aws) cloud

my talk at #qconsf in Nov http://qconsf.com/sf2010/presentation/Running+Netflix+in+the+Cloud

Can OpenStack catch up with AWS? Looks unlikely to me.

There has been a lot of chatter about the new OpenStack standard in the last few days, and how it could become a third option in the cloud infrastructure market alongside Amazon AWS and VMWare vCloud. For example Randy Bias/Cloudscaling has a good overview of the players.

The real question is whether OpenStack can catch up and become viable. Amazon is always described as being years ahead of the competition and having the lions share of the market. Estimates of the size of their lead depends on who you talk to, but I don't see anyone disputing their lead. On top of that, Amazon is investing heavily, has big customers like Zynga and Netflix stretching and hardening their systems (along with a huge number of small customers), and has already added many features beyond the basic compute (EC2) and storage (S3) that have been copied by others including OpenStack.

So for OpenStack to catch up, means that they have to move faster than Amazon, and leverage the slipstream effect where Amazon has had to educate the market and figure out what works, so competitors can copy their successes. However when I look at the details of the OpenStack specification they appear to be copying some of Amazon's problems as well. In particular the account and authentication model, which does not scale for enterprise use.

At the most basic level, it is infeasible to change the security model of a platform architecture, it's one of the most fundamental starting points that conditions the layers above. Changes to account and authentication management break all the layers of applications and tools that are built on the platform. One of the first problems that Netflix had with Amazon, was the lack of sub-accounts and role based access control (RBAC), and while beating up Amazon on this point for the last two years, we have built our own platform layers and tools to compensate. As a result, we find it impossible to use any of the web consoles or tools produced by Amazon or the many cloud vendors, which assume that there is a single account owner who can do anything. We hope that Amazon will eventually support sub-accounts, and when they do, I expect it will break everyone's tools.

At this point OpenStack is just the base level of the platform, it doesn't really have layers of tools on top yet, but its account and authentication model appears to be exactly the same as Amazon, so they will end up with layers of tooling that don't meet the needs of enterprise customers.

What's the difference between a startup and an enterprise? In a startup, everyone in IT knows the root password to every machine in their infrastructure. In an enterprise root passwords are carefully controlled, they change when someone in-the-know leaves, and specialist groups manage different parts of the system (Network ops can only mess with the switches, DBA's can only mess with the database etc.). The problem with a single account for the cloud is that everyone who needs to do anything to that account can do everything to it, and the common tools are oriented to a single user, managing several accounts, rather than a hierarchy of users managing parts of one account. All the systems in the account need an authentication key to access cloud services, and changing the password and key means you have to re-key every system. Get this wrong and your cloud will evaporate in an instant.

So in my opinion, a necessary but not sufficient condition for OpenStack to eventually catch up with AWS is that they need to build sub-accounts and RBAC into their spec from the start. However it seems much more likely that Amazon will just disappear into the distance from what I've seen so far.

iPhone Tethering Review

Along with iOS4 and the new ATT data plans they now also support tethering. I setup my wife’s personal iPhone 3GS for tethering, and it worked fine on her work laptop with Windows XP over the USB cable (she could work from the passenger seat while we were driving freeways). You just need to have iTunes installed on the XP machine, even if that isn’t the machine that sync’s the iPhone.

I also got Netflix to switch on tethering on my company iPhone 3GS, and I’m using it via bluetooth to my MacBook Air, easy setup and it also worked well for me.

The downside is that you lose the unlimited data plan, but for a few dollars a month, it can be very useful.

Netflix in the cloud and HTML5

One of the fun things about working at Netflix is that we are always "leaning forward" and looking for new technologies to leverage. As we roll out our website to run on the Amazon cloud, we are also re-architecting the code base, adding internationalization support and bringing in the latest technologies. One of these is HTML5, which is raising the bar for cross browser support for advanced user interface features, and is now supported by a large and rapidly growing percentage of the visitors to netflix.com. In addition many TV based devices now embed webkit, which is the HTML5 compatible technology that underpins the Safari and Chrome browsers. The user interface engineering team is looking to hire the best and brightest to work on these cutting edge technologies. Here's a write-up for one of the open positions:

Are you passionate about building great website experiences used by millions of visitors each day? Come to Netflix where we are using HTML5 based web technologies to move ecommerce directly onto televisions in our customers’ living rooms. As part of our Customer Acquisition team, you will lead the way to our internationalized television user interface designed to help new customers find Netflix and start streaming movies in seconds. This new experience will be deployed to HTML5 capable embedded browsers and served from our cutting edge cloud based backend service.

---------
Update - Hacking Netflix and TechCrunch picked up on this posting. MG Siegler at TechCrunch decided that I was talking about streaming video and Silverlight, which I wasn't. I was thinking of HTML5 features that let us build very cool user interfaces with drag-and-drop, canvas transforms etc. for the web site, and for embedded TV devices specifically. The Silverlight player is used for PC/Mac playback only, and the basic HTML5 Video doesn't have a viable DRM solution at this point. I'm the Cloud Architect for Netflix, so my involvement is to architect robust and scalable support in the cloud for these new user interfaces.

Garage Structure Completed

After lots of stops and starts, all that is left to do is some exterior paint and to fit the main doors. Work has already started on the electrical installation and water for fire sprinklers. Since the walls got so wet, they warped and parts of the siding had to be replaced. For some reason Tuffshed did other jobs on days where the weather was nice, and turned up to work on this one whenever it rained. We had what was probably the last rain of the winter on Sunday and Monday, but the roof was already tiled by then. The last video was the trusses being installed, here is the rest of the sequence.











Moving the viewpoint to catch trench digging around the back of the garage.

Cloud Architect

my role at Netflix just changed, from managing development of personalization in the cloud, to over-all cloud architect for Netflix. New challenges and a much broader scope. One challenge we have as we migrate to the cloud is that we need to build a new way to log and analyze what the systems in the cloud are doing, and we are basing this on Chukwa, Hadoop and Hive, all running in AWS. So we are looking to hire a Hadoop developer, and are looking at the research going on at Berkeley's RAD lab with interest.

AT&T 3G Microcell doesn't support corporate account phones

I just got an ATT 3G Microcell for home, since our service is intermittent, sometimes so bad I can't send an SMS message successfully (although at times it works fine). We live in the mountains, and have several different poor signals to bond to, so that's probably why it is variable.

In the store, they checked my phone number and said I couldn't activate a Microcell on my account, since it's a Netflix corporate account iPhone. However, my wife has an iPhone on a personal account, so I used that as the activation account.

However, when I tried to add my own phone number to the Microcell's whitelist of up to 10 numbers, I got this error message

"FTC309: invalid subscriber information found" - which got no hits on Google, however after this blog post it should get one..

So the website told me to call the generic ATT support number. After a few short waits, a very polite first level support guy (who had probably never heard of the Microcell before I called) found a well informed business support specialist who was able to confirm that the error code was due to the Microcell only supporting consumer accounts. She agreed that it didn't make a lot of sense, found that they had been running a trial of corporate accounts a few months ago, but didn't know when/if it might change. She forwarded me an internal support document that discussed the limitations of the trial, including the lack of corporate support. Although it does say that it supports all ATT 3G devices.

I installed it anyway, it takes a few hours to set itself up the first time. We will see how well it works for other iPhones, meanwhile I will just have to use Skype to make calls and send SMS... If it doesn't work out I will take it back.

------- Update -------
I contacted the corporate account support team via work, and they over-rode the system to add me to the Microcell. So my corporate account iPhone is now working fine with the Microcell. They are investigating further to see if they can make this a more general fix for other people, but if you are in the same situation as I was, you should be able to escalate to support to get it fixed.

Hiring a cloud platform engineer at Netflix

If you are interested in the CAP theorem NoSQL systems like Voldemort, Cassandra and SimpleDB, scalability and expect to be at the Velocity Conference in June, then you are the kind of engineer I'm interested in hiring. Netflix is aggressively moving to new technology, hiring only the very best and most experienced engineers and our growth is accelerating.

Garage Trusses Video

On Tuesday a big truck had some difficulty delivering a 40ft trailer containing the roof framing trusses, but it was raining so that is all that happened.

On Wednesday 21st April it started out dry, rained for a while then cleared up, but they did manage to put up all the trusses, and I made another stop-motion video of the work.

Tuffshed aren't great at letting us know what is going to happen next, we have to call them or ask the workers. I think the next step is to put plywood sheeting on the roof, then get an inspector up to check that the nailing pattern is correct, then they can put the doors and windows in, tile the roof add the external trim and paint it. It should be much warmer and sunny for the next few days.

garage - roof and rain on Tuesday

the roof trusses will be delivered tomorrow and rain is forecast for Tuesday and Wednesday - so we will see how they deal with that... it sucks that we don't have a roof before the rain.

[Posted with iBlogger from my iPhone]

Garage Framing Walkaround

The walls are up, but the trusses for the roof aren't ready, so it will be finished off next week. Luckily the weather looks good for the next week or so.

Garage Framing - first day of construction

Tuffshed turned up with their pre-built sections, assembled them and added some siding and the main door frame. Another stop-motion video on YouTube. It was wet over the weekend, and the ground is very soft, but we have a few dry and warm days in a row forecast, so I'm hoping it will dry out now.

Experiences using an iPad as my primary work laptop

I spent the last week carrying my iPad with me to meetings and using it as my main system at home an in my cube. My laptop was used for a few things, like the performance monitoring tool AppDynamics - which needs a bigger screen and is flash based, but just about everything else was usable on the iPad.

I have the apple case, which I think is essential, as it holds the iPad up at the right angle to use on a desk or lap, as well as protecting it and making it less slippery to carry around. I got the Video out cable but haven't had a chance to use it yet. After intensive use, the battery is still at 60% or more each evening.

I like the speed at which apps like mail and calendar come up, and they are both very nice to use, I prefer them to entourage on my Mac laptop. I spend a lot of my day reading writing and deleting email. Safari is also fast and worked well for gmail. I imported a PowerPoint presentation to keynote on the iPad and it needed adjustments to font sizes in diagrams, and lost some color coding, but seems like a usable tool. I will switch to keynote on my Mac as well and hopefully it will be compatible.

I have a VNC app, but moving the mouse cursor around is a a pain so it's only useful for emergency access to my laptop.

The other app I use a lot is NewsRack, which tracks about 100 random rss feeds for me by keeping in sync with google reader. I get most of my news and keep up with friends blogs this way. Compared to the iPhone version it seems to update much faster and the extra screen space is well used. The BBC app provides a more visual way to track stories, and the Guardian Eyewitness photography app contains stunning images, it's one of the most popular free apps at the moment.

Typing on the iPad is better than expected with two caveats. I keep hitting the b or n keys when I mean to hitbspace (like that), and the auto correction system is a bit over-active so I have to proofread what I write more than usual.

Of course I have been watching movies with the Netflix app, and have used the eBay and Pandora iPad apps, which look good. I'm still using the iPhone version of echofon as my Twitter reader, but I will try out the official twitter app when it comes out.

Garage - Pouring and Finishing the Foundation

Last Friday we poured concrete, and then it poured rain. Here's a stop-motion video of the work. Six concrete trucks came, we are high on the mountain, so they may not have been carrying a full load each time. The foundation is very deep on the downhill side, and the slab is 6" deep rather than the 4" minimum used for the walkway down the side and the parking apron.

The first few seconds of each video look messy, this was introduced when youtube encoded it, my original video is fine.



On Saturday, the weather was better, and the team came back to finish the surface and tidy up.



Final bill was about $25K, and I would recommend Amaya Concrete of Morgan Hill, they did a good job.

This week, we let the concrete cure, wait for the ground to dry out and do some basic grading of the ground levels and slopes. Next week the building will be delivered and installed.

IPad makes it hard to embed YouTube in blog

I'm trying to compose a blog post that embeds a YouTube video, but YouTube doesn't want to give me the HTML fragment to embed the video. If you browse to YouTube on an iPad it doesn't show any options to pick up the embed because it renders in "touch" mode. At the bottom of the page I found an option to switch to desktop mode and get the full page. This renders an embed option but doesn't let me select it to copy here.

So I need to use my MacBook to compose posts.

As the iPad moves the boundaries between content consumption and content generation, people need to rethink the functionality of the mobile and iPhone optimized web sites.

Garage - Inspected and delayed

We passed inspection by the county this morning, but the weather has turned too bad to pour concrete tomorrow, so we are postponing until after Wednesday's storm has passed, and will resume work on Friday. Today the electrician finished his work on the pipe that carries the power, and the Tuffshed foreman Mike visited to inspect the site and check and adjust the position of the hold-down bolts that are set into the foundation.