Adrian Cockcroft's Blog

Thoughts on SimpleDB, DynamoDB and Cassandra

2012-01-19T14:25:00.000-08:00

I've been getting a lot of questions about DynamoDB, and these are my personal thoughts on the product, and how it fits into cloud architectures.

I'm excited to see the release of DynamoDB, it's a very impressive product with great performance characteristics. It should be the first option for startups and new cloud projects on AWS. I also think it marks the turning point on solid state disks, they will be the default for new database products and benchmarks going forward.

There are a few use cases where SimpleDB may still be useful, but DynamoDB replaces it in almost all cases. I've talked about the history of Netflix use of SimpleDB before, but it's relevant to the discussion on DynamoDB, so here goes.

When Netflix was looking at moving to cloud about three years ago we had an internal debate about how to handle storage on AWS. There were strong proponents for using MySQL, SimpleDB was fairly new, and other alternatives were nascent NoSQL projects or expensive enterprise software. We started some pathfinder projects to explore the two alternatives and decided to port an existing MySQL app to AWS, while building a replication pipeline that copied data out of our Oracle datacenter systems into SimpleDB to be consumed in the cloud. The MySQL experience showed that we would have trouble scaling, and SimpleDB seemed reliable, so we went ahead and kept building more data sources on SimpleDB, with large blobs of data on S3.

Along the way we put memcached in front of SimpleDB and S3 to improve read latency. The durability of SimpleDB is its strongest point, we have had Oracle and SAN data corruption bugs in the Datacenter over the last few years, but never lost or corrupted any SimpleDB data. The limitations of SimpleDB are its biggest problem. We worked around limits on table size, row and attribute size, and per-request overhead caused by http requests needing to be authenticated for every call.

So the lesson here is that for a first step into NoSQL, we went with a hosted solution so that we didn't have to build a team of experts to run it, and we didn't have to decide in advance how much scale we needed. Starting again from scratch today, I would probably go with DynamoDB. It's a low "friction" and developer friendly solution.

Late in 2010 we were planning the next step, turning off Oracle and making the cloud the master copy of the data. One big problem is that our backups and archives were based on Oracle, and there was no way to take a snapshot or incremental backup of SimpleDB. The only way to get data out in bulk is to run "SELECT * FROM table" over HTTP and page the requests. This adds load, takes too long, and costs a lot because SimpleDB charges for the time taken in SELECT calls.

We looked at about twenty NoSQL options during 2010, trying to understand what the differences were, and eventually settled on Cassandra as our candidate for prototyping. In a week or so, we had ported a major data source from SimpleDB to Cassandra and were getting used to the new architecture, running benchmarks etc. We evaluated some other options, but decided to take Cassandra to the next stage and develop a production data source using it.

The things we liked about Cassandra were that it is written in Java (we have a building full of Java engineers), it is packed full of state of the art distributed systems algorithms, we liked the code quality, we could get commercial support from Datastax, it is scalable and as an extra bonus it had multi-region support. What we didn't like so much was that we had to staff a team to own running Cassandra for us, but we retrained some DBAs and hired some good engineers including Vijay Parthasarathy, who had worked on the multi-region Cassandra development at Cisco Webex and who recently became a core committer on the Apache Cassandra project. We also struggled with the Hector client library, and have written our own (which we plan to release soon). The blessing and a curse of Cassandra is that it is an amazingly fast moving project. New versions come fast and furiously, which makes it hard to pick a version to stabilize on, however the changes we make turn up in the mainstream releases after a few weeks. Saying "Cassandra doesn't do X" is more of a challenge than a statement. If we need "X" we work with the rest of the Apache project to add it.

Throughout 2010 the product teams at Netflix gradually moved their backend data sources to Cassandra, we worked out the automation we needed to do easy self service deployments and ended up with a large number of clusters. In preparation for the UK launch we also made changes to Cassandra to better support multi-region deployment on EC2, and we are currently running several Cassandra clusters that span the US and European AWS regions.

Now that DynamoDB has been released, the obvious question is whether Netflix has any plans to use it. The short answer is no, because it's a subset of the Cassandra functionality that we depend on. However that doesn't detract from the major step forward from SimpleDB in performance, scalability and latency. For new customers, or people who have outgrown the scalability of MySQL or MongoDB, DynamoDB is an excellent starting point for data sources on AWS. The advantages of zero administration combined with the performance and scalability of a solid state disk backend are compelling.

Personally my main disappointment with DynamoDB is that it doesn't have any snapshot or incremental backup capability. The AWS answer is that you can extract data into EMR then store it in S3. This is basically the same answer as SimpleDB, it's a full table scan data extraction (which takes too long and costs too much and isn't incremental). The mechanism we built for Cassandra leverages the way that Cassandra writes immutable files to get a callback and compress/copy them to S3 as they are written, it's extremely low overhead. If we corrupt our data with a code bug and need to roll back, or take a backup in production and restore in test, we have all the files archived in S3.

One argument against DynamoDB is that DynamoDB is on AWS only, so customers could get locked in, however it's easy to upgrade applications from SimpleDB, to DynamoDB and to Cassandra. They have similar schema models, consistency options and availability models. It's harder to go backwards, because Cassandra has more features and fewer restrictions. Porting between NoSQL data stores is trivial compared to porting between relational databases, due to the complexity of the SQL language dialects and features and the simplicity of the NoSQL offerings. Starting out on DynamoDB then switching to Cassandra when you need more direct control over the installation or Cassandra specific features like multi-region support is a very viable strategy.

As early adopters we have had to do a lot more pioneering engineering work than more recent cloud converts. Along the way we have leveraged AWS heavily to accelerate our own development, and built a lot of automation around Cassandra. While SimpleDB has been a minor player in the NoSQL world DynamoDB is going to have a much bigger impact. Cassandra has matured and got easier to use and deploy over the last year but it doesn't scale down as far. By that I mean a single developer in a startup can start coding against DynamoDB without needing any support and with low and incremental costs. The smallest Cassandra cluster we run is six m1.xlarge instances spread over three zones with triple replication.

I've been saying for a while that 2012 is the year that NoSQL goes mainstream, DynamoDB is another major step in validating that move. The canonical CEO to CIO conversation is moving from 2010: "what's our cloud strategy?", 2011: "what's our big data strategy?" to 2012: "what's our NoSQL strategy?".

How Netflix gets out of the way of innovation

2011-12-30T18:25:00.000-08:00

#defrag 2011 presentation script.

I'm the cloud architect for Netflix, but rather than tell you about why we moved Netflix to a cloud architecture or how we built our cloud architecture, I'm going to tell you what we do differently at Netflix to create a culture that supports innovation.

What is it that lets us get things done very quickly. Sometimes a bit too qwikly…. but how did we keep making big strategic moves, from DVD to streaming, from Datacenter to Public Cloud, from USA only to International, all in very short timescales with a fairly small team of engineers.

My presentation slides are just box-shots of movies and TV shows that are available on Netflix streaming. This script is based on the notes I made to figure out what I was going to say for each box shot. If some of you see a show you didn't know we had and want to watch that would make me happy, you can click on the box shot to visit that movie at Netflix, they were all available for streaming in the USA at the time of writing.

I've attempted to match the box shots loosely as cues to what I'm saying, but I've also used a musical theme in places since this is for Defrag and Defrag rocks!

Netflix is now one of the largest sites that runs almost entirely on public cloud infrastructure. We have become a poster child for how to build an architecture that takes full advantage of the Amazon Web Services cloud. But when I talk to other large companies about what we have done, they seem to have a lot of reasons why they couldn't or didn't do what we did, even if they wanted to.

Why is that? Why are we heading in one direction while everyone else is going the other way? Are we crazy or are they zombies? Well, I've worked at other large companies so I have some perspective on the issues.

Before I joined Netflix I worked at eBay for a few years, and helped found eBay Research Labs. This was setup because eBay felt it wasn't innovating fast enough, and they were looking for the one missing ingredient that would drive more innovation into the company.

This is a fairly common approach. "You guys go and be innovative, then hopefully we will find ways to spread it around a bit." Unfortunately the end result of setting up a separate group to add innovation to a big company is more comical than useful.

The most interesting projects got tied in knots, they trod on too many toes or were scary. We visited Xerox Parc and IBM Santa Teresa Labs to discuss how they were setup, to try and learn what might work., and we went to an Innovation Forum in New York. That was weird, some of the primary examples they were talking about emulating were eBay and Paypal!

The projects that did get out were minor tweaks to existing ideas, they could be fun, but ultimately not very interesting.

So I had to break out of there and find something new to do, and in 2007 I joined Netflix just as they first launched streaming.

One of the key attractions for me was the Netflix culture I heard about in the interviews, I wanted to get inside their heads and figure out if what they were describing was real, and if so, was it sustainable as the company grew.

What I found out over the next few years is that the culture is what enables innovation, so that Netflix can get things done quickly that other companies are too scared or too slow to try. The rest of this talk is about the key things that we do differently at Netflix.

Before I get into them I want to warn you that even with a roadmap and a guide, you probably won't be able to follow this path if you are in a large established company. Your existing culture won't let you. However if you are creating a new company from scratch, I hope you can join me in what I hope is the future of cool places to work.

Here's the key insight. It's the things you don't do that make the difference. You don't add innovation to a company culture, you get out of its way.

I'm mostly going for SciFi at this point, because it's going to sound like I was beamed in from the future to some of you.

Let me repeat that. You have to setup a company that doesn't do many of the things you would consider business as usual. That's why it's so hard to retrofit.

How about some audience participation? Hands up everyone who hates answering questions by putting their hands up..

Who works at a company that has more than one product line? Do you get along? The problem is that the company loses focus and has trouble allocating resources where they are needed so there are big fights. Pick one big thing and do it well. For Netflix, our addressable market is everyone in the universe who likes watching movies and TV shows, that should keep us busy for a while.

Who has teams spread over multiple sites and countries? We don't. It adds communication and synchronization overhead that slows your organization down. For the geeks, think of Amdahl's law applied to people. We have as many people as possible in the same building on the same site. We are planning a new bigger building to make sure we can keep everyone close together. High bandwidth, low latency communication.

Who's worked for a place that bought another company, then run it into the ground, laid everyone off and wrote down the value. Over and over again. It's crazy. I don't think Netflix has ever bought another company. It's a huge disruption to the culture, if you see something you like just hire away their best people and out execute them in the market.

Who has junior engineers, graduate hires and interns writing code? We don't. We find that engineers who cost twice as much are far more than twice as productive, and need much less management overhead. Reducing management overhead is a key enabler for an innovative culture. Engineers who don't need to be managed are worth paying extra for. We are tiny compared to companies like Google, they take on raw talent and develop it, we sometimes take a chance on someone with only five years experience.

Who has an architecture review board and centralized coding standards? We don't have that either. What we do have is tooling that creates a path of least resistance, which combined with peer pressure keeps quality high. The engineers are free and responsible for figuring it out for themselves.

Who has an ITops team that owns keeping code running in production? We don't. The developers run what they wrote. Everyone's cell phone is in the pagerduty rota, the trick is making sure you don't need to get called. All the ops people here have horror stories of stupid developers, and vice versa, but it doesn't have to be that way. We have one dev organization that does everything and no IT ops org involvement in our AWS cloud deployment.

Who has to ask permission before deploying 100's or 1000's of servers? We don't. The developers use our portal directly, they have to file a Change Management ticket to record what they did if it's in production, that's all. We've trained our developers to operate their own code. We create and destroy up to 1000 servers a day, just pushing new code. AWS takes about 5 minutes to allocate 100 servers, it takes longer than that just to boot Linux on them.

Who has a centralized push cycle and has to wait for the next "train" before they can ship their code? We don't. Every team manages their own release schedule. New code updates frequently, and the pace slows for mature services. Teams are responsible for managing interface evolution and dependencies themselves. Freedom and responsibility again.

Who has project managers tracking deliverables? We don't. The line managers do it themselves. They own the resources and set the context for their teams. They have time to do this because we took the BS out of their role.
Managers have to be great at hiring, technical and hands on enough to architect what their team does, and project manage to deliver results. Don't split this into three people. Reduce management overhead, minimize BS and time wasted. Teams are typically 3-7 people. Have a weekly team meeting and 1on1 with each engineer to maintain context.

Who has a single standard for development tools? We don't. We assume developers already know how to make themselves productive. We provide some common patterns to get new hires started, like Eclipse, IntelliJ, on Mac, Windows. Some people use Emacs on Linux. Hire experienced engineers who care, and they will take care of code quality and standards without being told how to.

Who has to work with people they don't respect? It's much too disruptive. The only way to get high talent density is to get rid of the people who are out of their depth or coasting.

That also applies to what you might call brilliant jerks. Even if they do great work, the culture can't tolerate prima donna anti-social behavior, so people who don't trust others or share what they know don't fit in.

So does that mean we value conformity? No but it's really important to be comfortable as part of a high performance team, so we look for people who naturally over-communicate and have a secure confident personality type.

If you haven't experienced a high performance culture, think about what it's like to drive flat out at a race track. Some people will be too scared to deal with it and drive around white knuckled at 40 mph, some will be overconfident and crash on the first corner, but for people who fit into the high performance culture it's exhilarating to push yourself to go faster each lap, and learn from your peers without a speed limit. When you take out the BS and friction, everyone gets so much more done that productivity, innovation and rapid agile development just happen. This is the key message, removing obstacles to a high performance culture is how innovation happens throughout an organization. Doing less to get more.

We don't pay bonuses. We don't have grades other than senior engineer, manager, director, VP. We don't count the hours or the vacation days, we say "take some". Once a year we revise everyones salary to their peers and current market rate - based on what we are paying now to hire the best people we can find.

We also have what sounds like a crazy stock option plan that grants options every month, vests the same day, and they last 10 years even if you leave Netflix. The net of this is less work for managers, they can concentrate on hiring top people, and almost everyone that leaves takes a pay cut. The test we make is "would you fight to keep your engineers if they tried to leave". If not, let them go now and get someone better. We don't make it hard to let people go.

Some of you may be thinking this sounds expensive, but what is the value of being incredibly productive and able to move faster than your competition? You can get out ahead and establish a leading position before anyone else realizes you are even in the game. Remember how a few years ago the "Analysts" said that Netflix the DVD company was going to get killed by other companies streaming, then all of a sudden people realized that we were streaming more bandwidth than anyone else?

So what could possibly go wrong? We had a near miss recently, we went too fast, partly because we could, got unlucky and screwed up. The good thing is that Netflix could re-plan and execute on the fixes we need very quickly as well, with no internal angst and finger-pointing. Also there was an Asteroid nearby earlier this week. By the way, my stepdaughter @raedioactive was the art director for this movie.

So, you are at a crossroads, you could be on stage with Eric Clapton, or in the audience watching and wondering why you can't do what they are doing. It's a radically different way to construct a corporate culture, it doesn't work for everyone, and we can't all be up on stage with Eric, but the talent is out there if you start by building a culture focused on talent density to find it and keep it.

Is it going to be the goats or the glory? I just told you all to stop doing things, what could be easier than that? It takes less process, fewer rules and simpler principles. Give people freedom, hold them responsible, replace the ones that can't or won't perform in that environment. Focus on talent density and conserving management attention span by removing the BS from their jobs.

This is your challenge, can you get a band together and go on a mission to save your company? Stop doing all the things that are slowing you down, and get rid of the unproductive BS that clogs up your management and engineers.

I will take questions in the comments or on twitter to @adrianco. Thank you.

Each question got a new box shot, but all the answers were musical.

The economics of Nissan Leaf ownership

2011-09-22T00:33:00.000-07:00

We got our @NissanLeaf at the end of June, and are very happy with it. It's fun to drive and is our first choice for any journey within range. Since the car is nearly silent and has over 200 lb-ft of torque, I often use full power for acceleration away from the lights, which quietly leaves the other cars behind. The low noise level is also great for listening to speech or music, and the steering wheel controls and bluetooth integration with my iPhone works well for phone calls and iPod or Pandora.

Laurel usually takes it for her 66 mile round trip commute, and if not, I take it on my 20 miles round trip commute, and any shopping trips at the weekend. So far we have averaged about 1000 miles per month, mostly mountain roads and freeways, which is the worst case for electrical consumption. The Leaf collects its milage and power use, and we can go back to look at the activity.

The record shows 1003 miles at 3.8 miles/KWh and a total of 264.6 KWh. We pay 8.5c/KWh, so if we had charged at home that's $22.49. Since we charge at work for free it's more like $15. We have grid-tied solar power and time of use metering for a cheap overnight rate. The meter runs backwards during the day at a higher price, and the Leaf has a charging delay timer so that we can plug it in when we get home, then it starts charging when the cheapest rate starts at 9pm.

1003 miles in the cars we would normally drive, which get about 20mpg, uses 50 gallons of premium gas at about $4.10 and about $206. So we actually saved $191 in August.

If Laurel drives every day, 22 work days a month at 66 miles is 1452 miles, she tops up the charge at work each day and gets about 4.2 miles/KWh on that route so that's still under a dollar a day. The gas cost would be $298, so she is saving about $280/month. That takes a big chunk out of the cost of buying the car in the first place. On top of that, the servicing costs are minimal, no oil changes, and the brakes last longer because the regenerative braking system takes a lot of the load. We have had to buy a new tire after popping it on a pothole, that was about $150, installed, but could happen to any car.

The icing on the cake is our white (for pure electric) car pool lane stickers, so Laurel can take the freeway in rush hour and zip silently past all the Prius drivers whose yellow (for hybrid) stickers no longer get them in the car pool lane. It took a total of ten weeks to get the license plates, then apply and get the white stickers.

So the value proposition for the Leaf is that it is much more fun to drive than a high mpg economy car like a Prius, gets you in the car pool lane (if you live in California), and the purchase cost is offset by ultra-low running costs if you use it regularly.

We aren't alone in figuring out that this is a good deal. At last count there are six Nissan Leaf owners at Netflix, along with one Volt (which doesn't get the carpool lane sticker) and two Tesla Roadsters. At Informatica, Laurel is one of at least three Leafs and one Volt sharing the chargers.

We leased our car on a three year 36,000 mile plan. We included the home charger installation in the payment (about $2K), put down a $2K initial payment, got a $7500 federal rebate bundled into the deal. Actual payment including taxes is $576/month. We get a $2500 state rebate paid directly to us after signing up for it which covers the initial payment. We leased because we think that in three years time there may be big advances in electric car technology, we could decide to keep the Leaf, or give it back and get a 2014 Tesla Model S or a Fiat 500 Electric (or both!).

For much more discussion about the car, the My Nissan Leaf forum is quite active. One thing I found there is that upgraded springs and dampers are available, since we do a lot of mountain driving, I'm planning to upgrade the suspension to be lower and stiffer than stock.

The first question everyone asks is how far will the Leaf go, and the answer is between 80 and 120 miles per charge, but it depends on where you live and how hard you drive. The usable capacity of the battery pack is about 21KWh, the actual spec is 24KWh, so there is a little bit of extra capacity beyond it's "I'm empty" point. if you drive a lot of freeways at speed and climb mountains like we do, 4 miles/KWh gets you 84 miles. In a flat urban environment 6 miles/KWh is quite possible to get 120 miles.

Since we live at the top of a mountain (2400ft) and work near sea level, we usually charge the car to 80% full at home, and 100% full at work. This way there is regenerative braking for the initial downhill run, which is free power and also saves the brake pads.

That's enough of the brain dump, yet another 2am blog post...

Our "carwings" summary page for August is shown below.

Moving Planet - Day of Action Saturday 24th Sept

2011-09-21T01:06:00.000-07:00

There is a world-wide movement coordinated by the global warming action group 350.org which has set this coming Saturday as a day of action to raise awareness.

You can find an event near you at the Moving Planet Site. Many of the events are based around bike rides.

For people local to the bay area, there is also a large Electric Vehicle car show in Palo Alto organized by the Electric Auto Association of Silicon Valley, where people can see and get rides in electric cars like the Nissan Leaf. The electric world speed record holding 216mph Lightning Motorbike should also be there.

We have been having a lot of fun driving our Nissan Leaf, we use it almost every day, and going back to a conventional car seems so broken. What's this gearbox thing and why does it keep having to change gear? The quiet instant torque of direct drive electric power is addictive. At last count there are six Nissan Leaf's parked at Netflix every day. We have several chargers for people who need a top up, but most of us just charge at home.

For more mainstream activities, the San Jose Mercury is also organizing a big car show in central San Jose on the same day, hybrids2hotrods.com, they have a Fisker hybrid on their poster and there will be electric cars and lots of classic cars as well.

I'm aiming to write some more blog posts on our solar system and the Nissan Leaf over the next few days.

I come to use clouds, not to build them...

2011-08-30T21:56:00.000-07:00

[Update: Thanks for all the comments and Ryan Lawler's GigaOM summary - also I would like to credit James Urquhart's posting on Competing With Amazon Part 1. for giving me the impetus to write this.]

My question is what are the alternatives to AWS from a developer perspective, and when might they be useful? However I will digress into a little history to frame the discussion.

There are really two separate threads of development in cloud architectures, the one I care about is how to build applications on top of public cloud infrastructure, the other is about how to build cloud infrastructure itself.

In 1984 I didn't care about how the Zilog Z80 or the Motorola 6809 microprocessors were made, but I built my own home-brew 6809 machine and wrote a code generator for a C compiler because I thought it was the best architecture, and I needed something to distract me from a particularly mind-numbing project at work.

In 1988 I joined Sun Microsystems and was one of the people who could argue in detail how SPARC was better than MIPS or whatever as an instruction set, or how Solaris and OpenLook window system were better. However I never designed a RISC microprocessor, wrote kernel code or developed window systems libraries. I helped customers use them to solve their own problems.

In 1993 I moved to the USA and worked to help customers scale their applications on the new big multiprocessor systems that Sun had just launched. I didn't re-write the operating system myself, but I figured out how to measure it and explain how to get good performance in some books I wrote at the time.

In 1995 when Java was released and the WWW was taking off, I didn't work on the Java implementation or IETF standards body, I helped people to figure out how to use Java and to get the first web servers to scale, so they could build new kinds of applications.

In 2004 I made a career change to move from the Enterprise Computing market place with Sun to the Consumer Web Services space with eBay. At the time eBay was among the first sites to have a public web API. It seemed to me that the interesting innovation was now taking place in the creation and mash-up of web services and APIs, no-one cared about what operating system they ran, what hardware that ran on, or who sold those computers.

Over time, the interesting innovation that matters has moved up the food chain to higher and higher levels of abstraction, leveraging and taking for granted the layers underneath. A few years ago I had to explain to friends who still worked at Sun, how I was completely uninterested in whether my servers ran Linux or Solaris, but I did care what version of Java we were using.

Now I'm working on a cloud architecture for Netflix, we don't really care which Content Delivery Network is used to stream the TV shows over the Internet to the customers, we interchangeably use three vendors at present. I also don't care how the cloud works underneath, I hear that AWS uses Xen, but it's invisible to me. What I do care about is how the cloud behaves, i.e. does it scale and does it have the feature set that I need.

That brings me back to my original question, what are the alternatives to AWS and when might they be useful.

Last week I attended an OpenStack meetup, thinking that I might learn about its feature set, scale and roadmap as an alternative to AWS. However the main objective of the presenters seemed to be to recruit the equivalent of chip designers and kernel developers to help them build out the project itself, and to appeal to IT operations people who want to build and run their own cloud. There was no explanation or outreach to developers who might want to build applications that run on OpenStack.

I managed to get the panel to spend a little while explaining what OpenStack consists of, and figured out a few things. The initial release is only really usable via the AWS clone APIs and doesn't have an integrated authentication system across the features. The "Diablo" release this fall should be better integrated and will have native APIs, it is probably suitable for proof of concept implementations by people building private clouds. The "Essex" version targeted at March next year is supposed to be the first production oriented release.

There are several topics that I would like to have seen discussed, perhaps people could discuss them in the comments to this blog post? One is a feature set comparison with AWS, and a discussion of whether OpenStack plans to continue to support the AWS clone APIs for equivalent features as it adds them. So far I think OpenStack has a basic EC2 clone and S3 clone, plus some networking and identity management that doesn't map to equivalent AWS APIs.

The point of my history lesson in the introduction is that a few very specialized engineers are needed to build microprocessors, operating systems, servers, datacenters, CDNs and clouds. It's difficult and interesting work, but in the end if its done right it's a commodity that is invisible to developers and their customers. One of the slides proudly showed how many developers OpenStack had, a few hundred, mostly building it from scratch. There wasn't room on the slide to show how many developers AWS has on the same scale. Someone said recently that the far bigger AWS team has more open headcount than the total number of people working on OpenStack. When you consider the developer ecosystem around AWS, there must be hundreds of thousands of developers familiar with AWS concepts and APIs.

Some of the proponents of OpenStack argue that because it's an open source community project it will win in the end. I disagree, the most successful open source projects I can think of have a strong individual leader who spends a lot of time saying no to keep the project on track. Some of the least successful are large multi-vendor industry consortiums.

The analogy that seems to fit is Apple's iOS vs. Google's Android in the smartphone market. The parts of the analogy that resonate with me are that Apple came out first and dominates the market, taking most of the profit and forcing it's competitors to try and band together to compete, changing the rules of the game and creating new products like the iPad that leave their competition floundering. By adding together all the incompatible fragmented Android market together it's possible to claim that Android is selling in a similar volume to iPhone. However it's far harder for developers to build Android apps that work on all devices, and then they usually make much less money from them. Apple and it's ecosystem is dominant, growing fast, and extremely profitable.

In the cloud space, OpenStack appears to be the consortium of people who can't figure out how to compete with AWS on their own. AWS is dominant, growing its feature set and installed capacity very fast. Every month that passes, AWS is refining and extending it's products to meet real customer needs. Measured by the reserved IP address ranges used by its instances AWS has more than doubled in the last year and now has over 800,000 IP addresses assigned to its regions worldwide.

The problem with a consortium is that it is hard to get it to agree on anything, and Brooks law applies (The Mythical Man-Month - adding resources to a late software project makes it later). While it seems obvious that adding more members to OpenStack is a good thing, in practice, it will slow the project down. I was once told that the way to kill a standards body or consortium is to keep inviting new people to join and adding to its scope. With the huge diversity of datacenter hardware and many vendors with a lot to lose if they get sidelined I expect OpenStack to fracture into multiple vendor specific "stacks" with narrow test matrixes and extended features that lock customers in and don't interoperate well.

I come to use clouds, because I work for a developer oriented company that has decided that building and running infrastructure on a global scale is undifferentiated heavy lifting, and we can leverage outside investment from AWS and others to do a better job than we could do ourselves, while we focus on the real job of developing global streaming to TVs.

Operations oriented companies tend to focus on costs and ways to control their developers. They want to build clouds, and may use OpenStack, but their developers aren't going to wait, they may be allowed to use AWS "just for development and testing" but when the time comes to deploy on OpenStack, it's lack of features is going to add a significant burden of complexity to the development team. OpenStack's lack of scale and immaturity compared to AWS is also going to make it harder to deploy products. I predict that the best developers will get frustrated and leave to work at places like Netflix (hint, we're hiring).

I haven't yet seen a viable alternative to AWS, but that doesn't mean I don't want to see one. My guess is that in about two to three years from now there may be a credible alternative. Netflix has already spent a lot of time helping AWS scale as we figured out our architecture, we don't want to do that again, so I'm also waiting for someone else (another large end-user) to kick the tires and prove that an alternative works.

Here's my recipe for a credible alternative that we could use:

AWS has too many features to list, we use almost all of them, because they were all built to solve real customer problems and make life easier for developers. The last slide of my recent cloud presentations at http://slideshare.net/adrianco contains a partial list as a terminology glossary. AWS is adding entirely new capabilities and additional detailed features every month, so this is a moving target that is accelerating fast away from the competition...

From a scale point of view Netflix has several thousand instances organized into hundreds of different instance types (services), and routinely allocates and deallocates over a thousand new instances each day as we autoscale to the traffic load and push new code. Often a few hundred instances are created in a few minutes. Some other cloud vendors we have talked to consider a hundred instances a large customer, and their biggest instances are too small for us to use. We mostly use m2.4xl and we need the 68GB RAM for memcached, Cassandra or our very large Java applications, so a 15GB max doesn't work.

In summary, although the CDN space is already a commodity with multiple interchangeable vendors, we are several years from having multiple cloud vendors that have enough features and scale to be interchangeable. The developer ecosystem around AWS concepts and APIs is dominant, so I don't see any value in alternative concepts and APIs, please try to build AWS clones that scale. Good luck :-)

Understanding and using Amazon EBS - Elastic Block Store

2011-03-18T21:41:00.000-07:00

There has been a lot of discussion in the last few days about EBS since it was implicated in a long outage at reddit.com.

Rule of Thumb

The benchmarking Netflix did when we started on AWS highlighted some inconsistent behavior in EBS. The conclusion we reached is a rule of thumb for EBS - If you sustain less than 100 iops (input+output per second) long term average it works fine. Short term bursts can be 1000 iops. By short term I mean less than a minute, long term more than 10 minutes. YMMV.

If you are doing benchmarks like this, collect response time and throughput and plot your data over time. You need to run long enough that the performance shows steady state behavior. The problem with EBS is that it doesn't have a particularly steady state. To explain why we need to look at the underlying architecture. I don't know the details of how EBS is implemented, but there is enough information available to explain how it behaves.

EC2

The AWS EC2 architecture is built out of commodity low cost servers, they have a single 1 Gbit network, a few CPUs, a few disks and a few GBytes of RAM. Over time the models have changed, and EC2 does have a 10Gbit network option now, but for the purposes of this discussion, we will concentrate on the 1Gbit network models. Individual servers are virtualized into the familiar EC2 models by slicing up the RAM, CPUs and disk space, and sharing the network bandwidth and disk iops. When EC2 instances break or are de-configured any data on the internal disks is lost.

Elastic Block Store http://aws.amazon.com/ebs/

The AWS EBS service provides a reliable place to store data that doesn't go away when EC2 instances are dropped, but it provides the same mounted filesystem capability as the internal disks. If you need more disk space or iops you can mount more EBS volumes on a single EC2 instance and spread out the load. The EBS volume is connected to the EC2 instance over the same 1Gbit network as everything else. In a datacenter this would normally be built using commercially available high end storage from NetApp, EMC or whoever, it would be quite expensive (cost much more than the EC2 instance itself) and be fast and reliable up to the limits of the network. To build a low cost cloud, the alternative is to use RAIN (Redundant Array of Inexpensive Nodes) which could be based on standard EC2 instances, or variants that have more disks per CPU. Software is then used to coordinate the RAIN systems and provide an EBS service that will be slower than high end storage, but still be very reliable and be limited by the 1Gbit network.

S3 and Availability Zones

AWS also has an S3 storage service that behaves like a key/value store accessed via http requests and a REST API rather than a directly mounted filesystem. It is possible to rapidly snapshot an EBS volume to and from S3, including incremental backups and restores that fill as they go so you don't have to wait before using them. This implies to me that they share a common back-end infrastructure to some extent. The primary additional difference is that EBS volumes only exist in a single AWS Availability Zone, and S3 data is replicated across two or three Availability Zones. It takes longer to replicate the data for S3, so it is slower, but it is very robust and it is almost impossible to lose data. You can think of an Availability Zone as a complete datacenter. All the zones in a region are separate datacenters that are close enough together to support a high bandwidth and low latency network between them, but they have separate power sources and connections to the Internet.

Multi-Tenancy

The most efficient chunk of compute and storage resource to buy and deploy when building a cloud is either too big or too small for the actual use cases of real applications. Virtualization is used to sub-divide the chunks, but then each individual machine is supporting several independent tenants. For local disks, the space is divided between the tenants, and for network, everyone is sharing the same 1Gbit interface. This works well on average, because most use cases aren't network or disk bound, but you cannot control who you are sharing with and some of the time you will be impacted by the other tenants, increasing variance within each EC2 instance. You can minimize the variance by running on the biggest instance type, e.g. m1.xlarge, or m2.4xlarge. In this case there isn't room for another big tenant, so you get as much as possible of the disk space and network bandwidth to yourself. The virtualization layer reserves some of the capacity. It's possible to tell that another tenant is keeping the CPU busy by looking at the "stolen time", but there are no metrics for stolen iops or network bandwidth.

The EBS service is also multi-tenant. Many clients mount disk space from a common backend pool of EBS disks. You don't get to see how the disk space is allocated, or how data is replicated over more than one disk or instance for durability, but it is limited to that availability zone. A busy client can slow down other clients that share the same EBS service resources. EBS volumes are between 1GB and 1TB in size. If you allocate a 1TB volume, you reduce the amount of multi-tenant sharing that is going on for the resources you use, and you get more consistent performance. Netflix uses this technique, our high traffic EBS volumes are mostly 1TB, although we don't need that much space.

This is actually no different in principle to large shared storage area network (SAN) backends (from companies like EMC or NetApp) that are in common datacenter use. Those also have unpredictable performance when pushed hard, and they mask this issue with lots of battery backed memory. The difference is cost. EBS is 10c per Gbyte per month. If you build a competing public cloud service using high end storage, you could get better performance but your cost base would be far higher.

Visualizing Multi-Tenant Disk Access

I have come up with some diagrams to help show what happens. I'm basing them on a simplified view of AWS where the only instance type family is m1 and everything they have is made out of one underlying building block. This consists of a fairly old specification system, 8 cores, 16GB RAM, four 500GB disks and a single 1Gbit network. In reality, AWS is much more complex than this, but the principles are the same.

Starting with internal disks, this is what an m1.xlarge looks like, it takes up the whole system apart from a small amount of memory, disk space and network traffic for the VM and AWS configuration/management information. You can expect to have minimal multi-tenant contention for network or disk access.

The m1.large instance type halves the system, each instance has two disks rather than four, so it shares the network and some of the disk controller bandwidth, but it should have minimal iops contention with the other tenant.

The low cost m1.small instance type has 160GB of disk per instance, so we can fit three per disk for a total of 12 instances per machine. (Note that the memory for a real m1.small is 1.7GB, so only 9 would fit in 16GB RAM, however the c1.medium instance has 1.7GB, 350GB disk, and more CPU, so six m1.small and three c1.medium fits). You can see the multi-tenancy problem here, any of the instances could generate enough traffic to fill the network and make one of the disks busy, and that is going to affect other instances in an unpredictable and random manner.

Here's an analogy, you can rent a whole house, rent a room in a house, or rent a couch to sleep on, you get what you pay for.

If you ever see public benchmarks of AWS that only use m1.small, they are useless, it shows that the people running the benchmark either didn't know what they were doing or are deliberately trying to make some other system look better. You cannot expect to get consistent measurements of a system that has a very high probability of multi-tenant interference.

EBS Multi-Tenancy

The next few diagrams show the flow of traffic from an instance to the EBS service, which makes two copies of the data on disks connected to separate instances. I don't know if this is how EBS works, but if we wanted to build an EBS-like system using the same building block it could look like this. In practice it would make sense to have specialized back-end building blocks with much more disk space.

The first diagram shows how Netflix runs EBS, we start with an instance that has the maximum network bandwidth with no other tenants, we allocate maximum size 1TB volumes (we stripe many of them together) and the service has to use most of the disk space in the back-end to support us, so we have less chance of another tenant making the EBS disks busy. The performance of EBS in this simplified case would be higher latency than local disk, but otherwise similar. I suspect that in reality the EBS volume is spread over more disks in the backend which gives higher throughput but with higher variance.

If we drop down to a more typical m1.large configuration with 100GB of EBS each, two instances are sharing network bandwidth, the EBS service is servicing two sets of requests, and the EBS back end has many more tenants per disk, so we would expect better peak performance than the two internal disks in the m1.large but more variance.

For the case where we have many m1.small instances each accessing a 10GB EBS volume, it is clear that the peak performance is going to be far better than a share of a local disk, but the contention for network, EBS service and backend disks will be extremely variable, so performance will be very inconsistent.

How To Measure Disk and Network Performance

Someone should write a book on that (I already did, but for Solaris), however there is a useful AWS forum post that explains how to interpret Linux iostat. This blog post is too long already, so Linux iostat will have to wait for another time.

Best Practices for Cloud Storage with Cassandra

There are two basic patterns for Cassandra, one is a persistent memory cache, where we size the data to fit in memory so that all reads are fast, and writes go to disk. The m2.4xl instance type with 68GB RAM and two 850GB disks is best. The second pattern is where there is a much larger data set than memory, and m1.xlarge with 16GB RAM and four 420GB disks will have the best iops for reads, and a much lower overall cost per GB for storage. In both cases, we get all the network bandwidth for servicing clients and the inter-node replication traffic, and minimal multi-tenant variance.

How not to build a Private Cloud

2011-03-12T08:12:00.000-08:00

It's all $, FUD, and internal politics. An MBO Cloud is what you get when the CEO tells the CIO to "figure out that cloud thing" (Management By Objective - i.e. the CIO bonus depends on it).

There is no technical reason for private cloud to exist.

[update: to clarify, that doesn't mean that I'm against private clouds or don't think they exist, because $, FUD and internal politics are a fact of life that constrain what can be done. Change also takes time and you have to "go to war with the army you have". However, this post is about what happens if your organization reallocates the $, isn't afraid, and has effectively no internal politics getting in the way.

This post was written in the middle of a debate on twitter between @adrianco @reillyusa @beaker and others including key insights from @swardley.

You should also read Christian Reilly's follow-up post "The Hollywood Culture" http://bit.ly/ePsisJ and many thanks to @bernardgolden for pointing out the excellent Business Week cover story on Cloud Computing http://ow.ly/4dm07 - after reading it I was amazed how well it aligned with what I write here - then I saw that it was by Ashlee Vance, one of the most clueful journalists around.

Netflix ITops Security Architect Bill Burns also wrote a very interesting post on the security challenges of cloud, we've been working together and he's on the interview team for the "Global Cloud Security Architect" I mention below.]

Too big for public cloud? You should *be* a public cloud.

Organizations who run infrastructure at the scale of tens to hundreds of thousands of instances have created cloud based models and opened them up to other organizations as public clouds. Amazon, Google, Microsoft are the clearest examples, they have expertise in software architecture, which is why they dominate the API definition. Telcos and hosting companies are adopting this model to provide additional public cloud capacity, using clones and variants of the API. Other organizations at this scale are already figuring out how to expose their capacity to their customers, partners and supply chain. The task you take on is to simultaneously hire the best people to run your cloud (competing with Amazon, Google etc.), and run it at low cost, which is why you need to be at huge scale and you need to decide that running infrastructure is a core competency of your business. Netflix is too small, doesn't regard infrastructure as core, and doesn't want to hire a bunch of ITops people.

It costs too much to port our apps? Your $ are mis-allocated.

What does it cost to build a private cloud, and how long does it take, and how many consultants and top tier ITops staff do you have to hire? Sounds like a nice empire building opportunity for the CIO. The alternative is to allocate that money to the development organization, hire more developers and rewrite your legacy apps to run on the public cloud, and give the development VP the budget to run public cloud directly. The payback is more incremental and manageable, but this is effectively a re-org of your business to move a large chunk of budget and headcount around. This is what happened at Netflix. It probably takes an act-of-CEO at most companies, the barriers are mostly political. Yes it will take time, but so will bringing up a private cloud.

Replace your apps with Saas offerings.

Many internal apps can be replaced by cloud services, we just outsourced our internal help desk and incident management software. No-one I know does payroll in-house. This is uncontroversial and is happening.

We can't put confidential data in a public cloud? This is just FUD.

The enterprise vendors are desperate to sell private clouds, so they are sowing this fear, uncertainty and doubt in their customer base to slow down adoption of public clouds. The reality is that many companies are already putting confidential data in public clouds. I asked the question "when will someone using PCI level 1 be in production on AWS" at a Cloud Connect panel, and was told that it is already being done, and Terremark pointed out that they host H&R Block's tax business. There are several models of public cloud with different SLA, cost and operational models that can support confidential data securely. There is also an argument that datacenter security is not as strong as people would like to think, and that the large cloud vendors can do a better job than most enterprises at keeping the infrastructure secure. At Netflix, we are about to transition to a global cloud based business, we are currently hiring a "Cloud Security Architect" who understands compliance rules like PCI (the credit card standard) on a global basis (we didn't need global expertise before). Part of their job is going to be to implement this.

There is no way my execs will sign off on this! Do they care about being competitive?

The biggest champion at Netflix for doing public cloud and doing it "properly" with an optimized architecture was our CEO Reed Hastings. He personally argued that we should try to do NoSQL rather than MySQL to push the envelope. Why? Because the bigger risk for Netflix was that we wouldn't scale and have the agility to compete. He was right, we have grown faster than our ability to build datacenters, and we have the agility we need to outrun our competition. Netflix has never had a CIO in the first place, we do have an excellent VP of operations though, and there is plenty to do running the CDN's and Saas vendors that support Enterprise IT.

Will private clouds be successful? I think there will be a few train wrecks.

The train wrecks will come as ITops discover that it's much harder and more expensive than they thought, and takes a lot longer than expected to build a private cloud. Meanwhile their developer organization won't be waiting for them, and will increasingly turn to public clouds to get their jobs done. We could argue about definitions but there are private clouds that are effectively the back ends for specialized large scale ecosystems like engineering companies that have to interface to the things that build stuff, or operate in places where there is no effective connection into the public clouds. For example, on board a ship that has limited external bandwidth, or to support a third world construction project. My take is that these will be indistinguishable from specialized Saas offerings within a supply chain ecosystem.

How not to build a public cloud - The Netflix Way

Re-org your company to give budget and headcount to the developers, let them run the public cloud operations
Ignore the FUD, best practices and patterns for compliance and security already exist and are audit-able
Get the CEO to give the CIO a different MBO, to shrink their datacenter.

Good luck with that :-)

Maslow's Hierarchy of NoSQL Reads (and Writes)

2011-03-02T09:04:00.000-08:00

I tried out Prezi to create this presentation, it was more fun to create than powerpoint but has a few limitations. I would like better drawing tools. The talk itself puts some of the main NoSQL solutions into context.

NoSQL - Maslow's Hierarchy of Reads (and Writes) on Prezi

Solar Power - More panels on the garage roof

2011-02-16T22:25:00.000-08:00

The SolarCity installation team arrives.

Product specification on a panel, 210W peak output, Silicon by Kyocera made in Mexico.

Fronius inverter installed inside the garage, wired to the distribution panel shown next to it.

Mounting rails on the garage roof.

Closeup of how the rails mount on the shingles.

All the panels in place, four rows of nine panels, 36 x 210W = 7560W DC, after the inverter this works out to about 6.5KW delivered AC power.

View from the front of the garage early in the morning.

It's all wired in, we are waiting for inspection and approval before we can turn it on. The same monitoring system is used as was installed for my previous solar installation on the house roof. This time we leased the panels, there are several options, from no money down to buying the whole thing up front. I've opted for an initial payment of $10K, and $90/month fixed for the duration of the lease (20 years). I don't pay anything until its turned on.

We currently generate about two thirds of our usage of electricity. The extra panels should triple the generated capacity, so we will generate about twice what we use in the short term. That evens out when we start charging a Nissan Leaf (delivery due in April), and change out our propane furnace for a heat pump and air conditioner (hopefully in time to cool us this summer).

A useful side effect is that the garage itself will be a lot cooler inside during the summer, as the panels shade the roof.

Solar Power - More panels and Nissan Leaf

2010-12-27T15:29:00.000-08:00

Solar City will triple our solar power output in February, and we just ordered a Leaf, which they say is deliverable in the April-July timeframe.

We are adding 6.5KW of net delivered AC capacity (gross is about 7.5KW DC) to the garage roof. This time we are leasing it, and there are several options for how much downpayment vs. recurring payment to make.

There was a power cut last night for about two hours, and the automatic generator that runs off our propane tank kicked in after about 20 seconds. When the power came back, there wasn't a glitch, everything kept running. That was the first real test. We have a Generac 17KW unit that powers most of the house. With hindsight, it would have been better to get the 20KW generator that can power a whole 200Amp distribution panel rather than routing individual circuits. The extra cost would be offset by a simpler installation. We hope to replace the propane furnace with a heat pump at some point, then the generator will be the only thing running on propane, and since our solar system is grid-tied, it needs to work well to keep an all electric house going.

We were originally looking a geothermal heat pumps, but the extra cost of digging wells to get a more efficient heat pump is not cost effective versus adding extra solar panels to make up the difference. In climates where it is colder and there is less solar irradiance available, it makes more sense to use geothermal.

During 2010 we generated about 60% of the electricity we consumed, and also greatly reduced our propane consumption by switching appliances from propane to electric. In 2011 we should be almost 100% solar powered electric, and reduce our gasoline usage as well.

NoSQL Netflix Use Case Comparison for Translattice

2010-11-17T22:56:00.000-08:00

[There is some discussion of this posting with comments by Michael at Slashdot]

Michael Lyle @mplyle CTO of Translattice kindly provided a set of answers that I have interspersed with the questions below. Translattice isn't technically a NoSQL system, but it isn't a conventional database either. It's a distributed relational SQL database that supports eventual consistency, as Michael puts it:

These answers are for the Translattice Application Platform (TAP)'s database component. Unlike other stores that have answered this question set, TAP contains a relational database that scales out over identical nodes. TAP further allows applications written to the J2EE platform to scale out across the same collection of nodes.

The original set of questions are posted here. Each respondent will get their own blog post with answers, when there are enough to be interesting, I will write some summary comparisons.

If you have answers or would like to suggest additional questions, comment here, tweet me @adrianco or blog it yourself.

Use Case Scenario for Comparison Across NoSQL Contenders

While each NoSQL contender has different strengths and will be used for different things, we need a basis for comparison across them, so that we understand the differences in behavior. Here is a sample scenario that I am publishing to put to each vendor to get their answers and will post the results here. The example is non-trivial and is based on a simplified Netflix related scenario that is applicable to any web service that reliably collects data from users via an API. I assume that is running on AWS and use that terminology, but the concepts are generic.

Use Case
A TV based device calls the API to add a movie to its favorites list (like the Netflix instant queue, but I have simplified the concept here), then reads back the entire list to ensure it is showing the current state. The API does not use cookies, and the load balancer (Amazon Elastic Load Balancer) is round robin, so the second request goes to a different API server, that happens to be in a different Amazon Availability Zone, and needs to respond with the modified list.

Favorites Storage
Favorites store is implemented using a NoSQL mechanism that persistently stores a single key=user value=movielist record on writes, and returns the movielist on reads.

Question 1: Availability Zones
When an API reads and writes to a queue store using the NoSQL mechanism, is the traffic routing Availability Zone aware? Are reads satisfied locally, or spread over all zones, is the initial write local or spread over the zones, is the write replication zone aware so data is replicated to more than one zone?

In the Translattice Application Platform, data in relational tables is transparently sharded behind the scenes by the data store. These shards are stored redundantly across the nodes. Reads are satisfied with the most local copy of data available on the network, unless that resource is currently overloaded in which case the system may fall back to reads from more distant locations.

When it comes to writes, applications have the choice on the durability and isolation levels for changes. Each transaction may be made in a fully synchronous, serializable isolation level, or may be made in a locked eventually consistent mode that provides ACID serializable semantics except that durability may be sacrificed if an availability zone fails. A further asynchronous mode allows potentially conflicting changes to be made and allows a user-provided reconciliation function to decide which change "wins". A final commit requires a majority of nodes storing a shard to be available; in the case of the fully synchronous mode this would delay or prevent the return of success if a critical subset of the cluster fails.

Policy mechanisms in the system allow administrators to specify how physical and cloud database instances correspond to administratively-relevant zones. An administrator can choose to require, for instance, that each piece of information is replicated to at least three database nodes across a total of two availability zones. An administrator may also use these mechanisms to require that particular tables or portions of tables must or must not be stored in a given zone (for instance, to meet compliance or security requirements). Within the constraints set by policy, the system tracks usage patterns and places information in the most efficient locations.

Question 2: Partitioned Behavior with Two Zones
If the connection between two zones fails, and a partition occurs so that external traffic coming into and staying within a zone continues to work, but traffic between zones is lost, what happens? In particular, which of these outcomes does the NoSQL service support?

one zone decides that it is still working for reads and writes but half the size, and the other zone decide it is offline
both zones continue to satisfy reads, but refuse writes until repaired
data that has a master copy in the good zone supports read and write, slave copies stop for both read and write
both zones continue to accept writes, and attempt to reconcile any inconsistency on repair

Assuming that the SQL transaction in question is running in the fully synchronous or eventually consistent locked mode, writes will only be allowed in one of the two zones. Reads will continue in both zones, but will only be able to satisfy requests for which at least one replica of the requested data exists in the local zone (policy can be specified to ensure that this is always the case). In the eventually consistent mode, multiple partitioned portions of the system can accept writes and reconcile later. Essentially, any of the above desired modes can be used on a transaction-by-transaction basis depending on application and performance requirements.

Question 3: Appending a movie to the favorites list

If an update is performed by read-modify-write of the entire list, what mechanisms can be used to avoid race conditions? If multiple attribute/values are supported for a key, can an additional value be written directly without reading first? What limits exist on the size of the value or number of attribute/values, and are queries by attribute/value supported?

Because fully relational primitives are provided, there can easily be one row in the database per favorite. Read-modify-write of the whole list is not required, and the only practical limits are application-defined.

Any SQL queries are supported against the store, and are transformed by the query planner into an efficient plan to execute the query across the distributed system. Of course, how efficient a query is to execute will depend on the structure of the data and the indexes that an administrator has created. We think this allows for considerable flexibility and business agility as the exact access methods that will be used on the data do not need to be fully determined in advance.

Question 4: Handling Silent Data Corruption

When the storage or network subsystem corrupts data without raising an error, does the NoSQL service detect and correct this? When is it detected and corrected, on write, on read or asynchronously?

Network activity is protected by cryptographic hash authentication, which provides integrity verification as a side benefit. Distributed transactions also take place through a global consensus protocol that uses hashes to ensure that checkpoints are in a consistent state (this is also how the system maintains transactional integrity and consistency when changes cross many shards). Significant portions of the on-disk data are also presently protected by checksums and allow the database to "fail" a disk if corrupt data is read.

Question 5: Backup and Restore

Without stopping incoming requests, how can a point in time backup of the entire dataset be performed? What is the performance and availability impact during the backup? For cases such as roll-back after a buggy application code push, how is a known good version of the dataset restored, how is it made consistent, and what is the performance and availability impact during the restore? Are there any scalability limits on the backed up dataset size, what's the biggest you have seen?

A good portion of this relational database's consistency model is implemented through a distributed multi-version concurrency control (MVCC) system. Tuples that are in-use are preserved as the database autovacuum process will not remove tuples until it is guaranteed that no one could be looking at them anymore. This allows a consistent version of the tables as of a point in time to be viewed from within a transaction; so BEGIN TRANSACTION; SELECT ... [or COPY FROM, to backup] ; COMMIT; works. We provide mechanisms to allow database dumps to occur via this type of mechanism.
In the future we are likely to use this mechanism to allow quick snapshots of the entire database and rollbacks to previous snapshot versions (as well as to allow the use of snapshots to stage development versions of application code without affecting production state).

NoSQL Netflix Use Case Comparison for Riak

2010-11-09T09:36:00.000-08:00

Justin Sheehy @justinsheehy of Basho kindly provided a set of answers that I have interspersed with the questions below.

The original set of questions are posted here. Each NoSQL contender will get their own blog post with answers, when there are enough to be interesting, I will write some summary comparisons.

If you have answers or would like to suggest additional questions, comment here, tweet me @adrianco or blog it yourself.

Use Case Scenario for Comparison Across NoSQL Contenders

Favorites Storage
Favorites store is implemented using a NoSQL mechanism that persistently stores a single key=user value=movielist record on writes, and returns the movielist on reads.

There are two possibilities with Riak. The first would be to spread a single Riak cluster across all three zones, for example one node in each of three zones. In this case, a single replica of each item would exist in each zone. Whether or not a response needed to wait on cross-zone traffic to complete would depend on the consistency level in the individual request. The second option would require Riak EnterpriseDS, and involves placing a complete cluster in each zone and configuring them to perform inter-cluster replication. This has multiple advantages. Every request would be satisfied entirely locally, and would be independent of latency or availability characteristics across zone boundaries. Another benefit is that (unlike either the first scenario or some other solutions that spread clusters and quorums over a long haul) read requests would not generate any cross-zone traffic at all. For an application with a high percentage of reads, this can make a large difference.

one zone decides that it is still working for reads and writes but half the size, and the other zone decide it is offline
both zones continue to satisfy reads, but refuse writes until repaired
data that has a master copy in the good zone supports read and write, slave copies stop for both read and write
both zones continue to accept writes, and attempt to reconcile any inconsistency on repair

As write-availability is a central goal achieved in Riak, the fourth option will be the observed behavior. This is the case regardless of the strategy chosen for Question 1. In the first strategy, local nodes other than the canonical homes for given data will accept the writes instead, using the hinted-handoff technique. In the second strategy, the local cluster will accept the write, those changes will be replayed across the replication link when the zones are reconnected. In all cases, vector clocks provide a clean way of resolving most inconsistency, and various reconciliation models are available to the user for those cases which cannot be syntactically resolved.

For more information on vector clocks in Riak, see:

http://blog.basho.com/2010/01/29/why-vector-clocks-are-easy/

and

http://blog.basho.com/2010/04/05/why-vector-clocks-are-hard/

Question 3: Appending a movie to the favorites list

Riak will use vector clocks to recognize causality in race conditions. In the case of two overlapping writes to the same value, Riak will retain both unless explicitly requested to simply overwrite with the last value received. If one client changes A to B and another changes A to C, then (unless told to overwrite) Riak will return both B and C to the client. When that client then modifies the object again, the single descendant "D" that they created will be the new value. For applications such as sets which are mostly added to and rarely deleted from, the application code to perform this reconciliation is trivial and in some cases is simply a set union operation. This would look a bit like this in terms of vector clock ancestry:

http://dl.dropbox.com/u/751099/ndiag1.png

Riak allows values to be of any arbitrary content type, but if the content is in JSON then a JavaScript map/reduce request can be used to query by attribute/value.

Question 4: Handling Silent Data Corruption

When the storage or network subsystem corrupts data without raising an error, does the NoSQL service detect and correct this? When is it detected and corrected, on write, on read or asynchronously?

Many layers of Riak perform consistency checking, including CRC checking in the persistence engine and object equality in the distributed state machines handling requests. In most cases where corruption can be detected in a given replica of some item, that replica will immediately but asynchronously be fixed via read-repair.

Question 5: Backup and Restore

There are two approaches to back up Riak systems: per-node or whole-cluster. Backing up per-node is the easiest option for many people, and is quite simple. Due to bitcask (the default storage engine) performing writes in an append-only fashion and never re-opening any file for writing once closed, Riak nodes can easily be backed up via the filesystem backup method of your choice. Simply replacing the content of the data directory will reset a node's stored content to what it held at the time. Alternately, a command line backup command is available which will write out a backup of all data on the cluster. This is fairly network and disk intensive and requires somewhere to put a whole-cluster backup, but is very useful for prototyping situations which are not holding enormous amounts of data.

Are we ready for spotcloud yet?

2010-11-01T15:21:00.000-07:00

Launched today by Enomaly (@ruv) Spotcloud is a "Cloud Capacity Clearinghouse and Marketplace". There was a lot of discussion on twitter about whether this is really new, and previous attempts to do something similar.

My background in this is that I was working at Sun in 2003/2004 when we were thinking about a marketplace for public grid computing capacity, I was chief architect for Shahin Khan's High Performance Technical Computing group at the time, and we "owned" Grid for Sun. We were both RIFd in the summer of 2004, but some of our projects stayed alive, and @ruv mentioned some of these ideas from Sun surfacing in 2005.

I moved to eBay, and one idea that I tried to get eBay interested in at the time was building a marketplace for compute capacity. The problem was that eBay is a retail product focused company, and had no product managers looking at digitally delivered products. I couldn't find a marketplace manager who understood what I was proposing and thought it might be worth working on. In practice, it was too early, but Amazon had the vision to build a cloud at this time, and eBay could have done the same if it wanted to create a market, rather than make existing markets more efficient.

In 2006 (while I was working at eBay Research Labs) I wrote a blog post about a maturity model for innovation. The key point is:

"the evolution of a marketplace goes from competing on the basis of technology, to competing on service, to competing as a utility, to competing for free. In each step of the evolution, competitors shake out over time and a dominant brand emerges.

To use this as a maturity model, take a market and figure out whether the primary competition is on the basis of technology, service, utility or search"

Today the cloud marketplace is somewhere between the service and utility phases. Each individual cloud has their own specific services and service interfaces, and they have not turned into a standard commodity yet, so we do not have the basis for competition purely on the basis of a Utility (i.e. on service quality - uptime, not on service features).

From this point of view, it is still too early for Spotcloud to take off. Cloud's problem is not "finding generic capacity at low cost" (the cloud utility search problem), the cloud marketplace is still evolving it's differentiated service interfaces towards a common set of functionality and standards. Spotcloud is starting out based on Enomaly's interfaces, and say they will add others, while the market leader is Amazon, who have already implemented their own spot pricing model.

One thing I did learn at eBay, is how hard it is to manage marketplaces. One unfortunate measure of success is that it attracts people whose aim is to make money by manipulating the market rather than contributing to it. There are a lot of non-intuitive details that you have to get right for a marketplace to scale and be robust enough to build and maintain trust, while also having very low "friction" so that it attracts and retains buyers and sellers.

So one way to tell that the marketplace for cloud capacity is viable is when you see eBay entering that marketplace :-)

NoSQL Netflix Use Case Comparison for MongoDB

2010-10-31T11:09:00.000-07:00

Roger Bodamer @rogerb from 10gen.com kindly provided a set of answers for MongoDB that I have interspersed with the questions below. The original set of questions are posted here. Each NoSQL contender will get their own blog post with answers, when there are enough to be interesting, I will write some summary comparisons. If you have answers or would like to suggest additional questions, comment here, tweet me @adrianco or blog it yourself.

Use Case Scenario for Comparison Across NoSQL Contenders

Let's assume for discussion purposes that we are using MongoDB across three availability zones in a region. We would have a replica set member in each of the three zones. One member will be elected primary at a given point in time.

All writes will be sent to the primary, and then propagate to secondaries from there. Thus, writes are often inter-zone. However availability zones are fairly low latency (I assume the context here is EC2).

Reads can be either to the primary, if immediate/strong consistency semantics are desired, or to the local zone member, if eventually consistent read semantics are acceptable.

one zone decides that it is still working for reads and writes but half the size, and the other zone decide it is offline
both zones continue to satisfy reads, but refuse writes until repaired
data that has a master copy in the good zone supports read and write, slave copies stop for both read and write
both zones continue to accept writes, and attempt to reconcile any inconsistency on repair

Let's assume again we are using three zones - we could use two but three is more interesting. To be primary in a replica set, the primary must be visible to a majority of the members of the set: in this case, two thirds of the members, or two thirds of the zones. If one zone is partitioned from the other two, what will happen is: a member in the 2 zone side of the partition will become primary, if not already. It will be available for reads and writes.

The minority partition will not service writes. Eventually consistent reads are still possible in the minority partition.

Once the partition heals, the servers automatically reconcile.

http://www.mongodb.org/display/DOCS/Replica+Set+Design+Concepts
Question 3: Appending a movie to the favorites list
If an update is performed by read-modify-write of the entire list, what mechanisms can be used to avoid race conditions?

MongoDB supports atomic operations on single documents via both its $ operators ($set, $inc) and also by compare-and-swap operations. In MongoDB one could model the list as a document per favorite, or, put all the favorites in a single BSON object. In both cases atomic operations free of race conditions are possible. http://www.mongodb.org/display/DOCS/Atomic+Operations

This is why mongodb elects a node primary: to facilitate these atomic operations for use cases where these semantics are required.

If multiple attribute/values are supported for a key, can an additional value be written directly without reading first?

Yes.

What limits exist on the size of the value or number of attribute/values?

A single BSON document must be under the limit -- currently that limit is 8MB. If larger than this, one should consider modeling as multiple documents during schema design.

and are queries by attribute/value supported?

Yes. For performance, MongoDB supports secondary (composite) indices.

Question 4: Handling Silent Data Corruption

When the storage or network subsystem corrupts data without raising an error, does the NoSQL service detect and correct this? When is it detected and corrected, on write, on read or asynchronously?

The general assumption is that the storage system is reliable. Thus, one would normally use a RAID with mirroring, or a service like EBS which has intrinsic mirroring.

However, the BSON format has a reasonable amount of structure to it. It is highly probable, although not certain, that a corrupt object would be detected and an error reported. This could then be correct with a database repair operation.

Note: the above assumes an actual storage system fault. Another case of interest is simply a hard crash of the server. MongoDB 1.6 requires a --repair after this. MongoDB v1.8 (pending) is crash-safe in its storage engine via journaling.

Question 5: Backup and Restore

Without stopping incoming requests, how can a point in time backup of the entire dataset be performed? What is the performance and availability impact during the backup?

The most used method is to have a replica which is used for backups only; perhaps an inexpensive server or VM. This node can be taken offline at any time and any backup strategy used. Once re-enabled, it will catch back up. http://www.mongodb.org/display/DOCS/Backups

With something like EBS, quick snapshotting is possible using the fsync-and-lock command.

For cases such as roll-back after a buggy application code push, how is a known good version of the dataset restored, how is it made consistent, and what is the performance and availability impact during the restore? Are there any scalability limits on the backed up dataset size, what's the biggest you have seen?

One can stop the server(s), restore the old data file images, and restart.

MongoDB supports a slaveDelay option which allows one to force a replica to stay a certain number of hours behind realtime. This is a good way to maintain a rolling backup in case of someone "fat-fingering" a database operation.

NoSQL Netflix Use Case Comparison for Cassandra

2010-10-29T09:38:00.000-07:00

Jonathan Ellis @spyced of Riptano kindly provided a set of answers that I have interspersed with the questions below.

The original set of questions are posted here. Each NoSQL contender will get their own blog post with answers, when there are enough to be interesting, I will write some summary comparisons.

If you have answers or would like to suggest additional questions, comment here, tweet me @adrianco or blog it yourself.

Use Case Scenario for Comparison Across NoSQL Contenders

Favorites Storage
Favorites store is implemented using a NoSQL mechanism that persistently stores a single key=user value=movielist record on writes, and returns the movielist on reads.

The most natural way to model per-user favorites in Cassandra is to have one row per user, keyed by the userid, whose column names are movie IDs. The combination of allowing dynamic column creation within a row and allowing very large rows (up to 2 billion columns in 0.7) means that you can treat a row as a list or map, which is a natural fit here. Performance will be excellent since columns can be added or modified without needing to read the row first. (This is one reason why thinking of Cassandra as a key/value store, even before we added secondary indexes, was not really correct.)

The best introduction to Cassandra data modeling is Max Grinev's series on basics, translating SQL concepts, and idempotence.

Briefly, both reads and writes have a ConsistencyLevel parameter controlling how many replicas across how many zones must reply for the request to succeed. Routing is aware of current response times as well as network topology, so given an appropriate ConsistencyLevel, reads can be routed around temporarily slow nodes.

On writes, the coordinator node (the one the client sent the request to) will send the write to all replicas; as soon as enough success messages come back to satisfy the desired consistency level, the coordinator will report success to the client.

For more on consistency levels, see Ben Black's excellent presentation.

one zone decides that it is still working for reads and writes but half the size, and the other zone decide it is offline
both zones continue to satisfy reads, but refuse writes until repaired
data that has a master copy in the good zone supports read and write, slave copies stop for both read and write
both zones continue to accept writes, and attempt to reconcile any inconsistency on repair

Cassandra has no 'master copy' for any piece of data; all copies are equal. The other behaviors are supported by different ConsistencyLevel values for reads (R) and writes (W):

R=QUORUM, W=QUORUM: One zone decides that it is still working for reads and writes, and the other zone decides it is offline
R=ONE, W=ALL: Both zones continue to satisfy reads, but refuse writes
R=ONE, W=ONE: Both zones continue to accept writes, and reconcile any inconsistencies when the partition heals

I would also note that reconciliation is timestamp-based at the column level, meaning that updates to different columns within a row will never conflict, but when writes have been allowed in two partitions to the same column, the highest timestamp will win. (This is another way Cassandra differs from key/value stores, which need more complex logic called vector clocks to be able to merge updates to different logical components of a value.)

Question 3: Appending a movie to the favorites list

Cassandra's ColumnFamily model generally obviates the need for a read before a write, e.g., as above using movie IDs as column names. (If you wanted to allow duplicates in the list for some reason, you would generally use a UUID as the column name on insert instead of the movie ID.)

The maximum value size is 2GB although in practice we recommend using 8MB as a more practical maximum. Splitting a larger blob up across multiple columns is straightforward given the dynamic ColumnFamily design. The maximum row size is 2 billion columns. Queries by attribute value are supported with secondary indexes in 0.7.

Question 4: Handling Silent Data Corruption

When the storage or network subsystem corrupts data without raising an error, does the NoSQL service detect and correct this? When is it detected and corrected, on write, on read or asynchronously?

Cassandra handles repairing corruption the same way it does other data inconsistencies, with read repair and anti-entropy repair.

Question 5: Backup and Restore

Because Cassandra's data files are immutable once written, creating a point-in-time snapshot is as simple as hard-linking the current set of sstables on the filesystem. Performance impact is negligible since hard links are so lightweight. Rolling back simply consists of moving a set of snapshotted files into the live data directory. The snapshot is as consistent as your ConsistencyLevel makes it: any write visible to readers at a given ConsistencyLevel before the snapshot will be readable from the snapshot after restore. The only scalability problem with snapshot management is that past a few TB, it becomes impractical to try to manage snapshots centrally; most companies leave them distributed across the nodes that created them.

Comparing NoSQL Availability Models

2010-10-27T14:14:00.000-07:00

let's risk feeding the CAP trolls, and try to get some insight into the differences between the many NoSQL contenders. I have circulated an earlier version of this to a few people and got at least one good response. If you have answers, or would like to suggest additional questions, comment here, tweet me @adrianco or blog it yourself.

Use Case Scenario for Comparison Across NoSQL Contenders

Favorites Storage
Favorites store is implemented using a NoSQL mechanism that persistently stores a single key=user value=movielist record on writes, and returns the movielist on reads.

Question 1: Availability Zones
When an API reads and writes to a queue store using the NoSQL mechanism, is the traffic routing Availability Zone aware? Are reads satisfied locally, or spread over all zones, is the initial write local or spread over the zones, is the write replication zone aware so data is replicated to more than one zone?

Question 2: Partitioned Behavior with Two Zones
If the connection between two zones fails, and a partition occurs so that external traffic coming into and staying within a zone continues to work, but traffic between zones is lost, what happens? In particular, which of these outcomes does the NoSQL service support?

one zone decides that it is still working for reads and writes but half the size, and the other zone decide it is offline
both zones continue to satisfy reads, but refuse writes until repaired
data that has a master copy in the good zone supports read and write, slave copies stop for both read and write
both zones continue to accept writes, and attempt to reconcile any inconsistency on repair

Question 3: Appending a movie to the favorites list
If an update is performed by read-modify-write of the entire list, what mechanisms can be used to avoid race conditions? If multiple attribute/values are supported for a key, can an additional value be written directly without reading first? What limits exist on the size of the value or number of attribute/values, and are queries by attribute/value supported?

Question 4: Handling Silent Data Corruption

When the storage or network subsystem corrupts data without raising an error, does the NoSQL service detect and correct this? When is it detected and corrected, on write, on read or asynchronously?

Question 5: Backup and Restore

Netflix in the Cloud

2010-10-10T01:06:00.000-07:00

I'm presenting this talk on Thursday at the Cloud Computing Meetup, and again on Nov 3rd at QConSF. So far I have posted a "teaser" summary on slideshare. After QCon I will post the full slide deck [update: combined deck from both talks posted here].

The meetup is the "beta test" of the presentation. It's in Mountain View at "Hacker Dojo", and at the time of writing 437 people have signed up to attend. If everyone turns up it's going to be crazy and over-flowing trying to park and get in, so get there early.... I will focus the meetup talk more on the operational aspects of the cloud architecture, and migration techniques.

At QConSF, the presentation is in the "Architectures you've always wondered about" track, and I will spend more time talking about the software architecture.

Why give these talks? We aren't trying to sell cloud to CIOs, it's all about hiring, we are talking at engineer focused events in the Bay Area. Netflix is growing fast, pathfinding new technologies in the public cloud to support a very agile business model, and is trying to attract the best and brightest people. We use LinkedIn a lot when we search for positions, so feel free to connect to me if you think this could be interesting, or follow @adrianco on Twitter.

Time for some climate action on 10/10/10

2010-10-06T00:53:00.000-07:00

organized by 350.org there are events all over the world. The significance of 350 is that it is the safe level of CO2 in the atmosphere. We are already at 390ppm, which is why the climate is changing and extreme weather events are becoming common. The level is going up by 2ppm a year at the moment and the changes needed to reverse this haven't started, so its going to go much higher and have increasingly worse effects.

Even if all you do is buy and start reading a copy of James Hansen's book Storms of my Grandchildren (you can get it instantly from the Kindle store) it will help.

Here is a useful discussion of "The Value of Coherence in Science" which is how you can tell what makes sense from what must be wrong. I'm a Physics graduate, and I was taught that science works by making predictions and testing them, and by eliminating incoherent propositions. i.e. if an argument contradicts itself it must be false. The denialist counter-arguments fail this test. To see why the denialist arguments are getting any air time at all, read Merchants of Doubt and Climate Cover Up.

For an example of incoherent and shoddy denialist work John Mashey has systematically deconstructed the Wegman Report which was presented to Congress as an impartial study when it was nothing of the sort.

More solar power

2010-10-03T00:50:00.000-07:00

We just signed up for almost 8KW on the garage roof. It's justified based on replacing our propane furnace with a heat pump that also gives us aIr conditioning, and also running an electric car (e.g. Nissan Leaf). We can use our existing monitoring system, but unlike last time, when we bought the system, we plan to lease it from Solar City this time. There Is a four month lead time, so we can pick the exact terms of the lease next February before it is installed. There are several options, zero down with a monthly charge that increases slightly each year, or options to pay various amounts up front with a fixed monthly payment. The monthly payments are all less than a typical electricity bill.

The goal for next year is to only use propane to run the backup generator (usually for a few days a year), and to make a dent in our petrol usage for the daily commute. We are on the Nissan Leaf list, but not in the first wave of owners. We should be able to get a test drive soon, I will report on any progress as it happens.

If you are interested in electric cars, look for Robert Llewellyn's Fully Charged video show on YouTube. He has been test driving everything and he's good fun to watch. You may recognize him as the actor who played Kryton in Red Dwarf.

Solar Power - The Year in Review

2010-09-26T16:55:00.001-07:00

For our full story on solar click here.

We installed Solar panels in August 2009, and they were turned on during September. We have now had a whole year of power which is shown below. The time-of-use metering means that we get paid much more for the power we generate in the afternoon than the power we use overnight, so the "zero point" for billing is different than that for power consumption. PG&E recently sent us our annual bill, which was about $500. Added to the monthly bills for basic service this comes to about $700 for the whole year. Our previous electricity bill was about $2000 a year. Since we changed our hot water, clothes dryer and range from propane to electric, we saved over $1000 in propane cost as well. That puts payback at around ten years at todays prices, given the likelihood of increased propane and electricity prices over time, actual payback would be earlier than that.

The panels have got a lot of dust on them (the roof is too high to get at easily to clean them), so aren't running at peak efficiency, the best day we saw was around 28KWh, with a lot of days of 26-27 KWh through the summer. The totals for each month are shown in the screenshots below.

Since we have built a new garage, we now have a lot more roof area. So we are now looking to add another 4KW of panels, and to swap out our propane furnace for a heat pump that will give us heating and cooling (yay - its hot today...) sometime next spring. Then the only use for propane will be the emergency generator. It should also make the garage a bit cooler in the summer by shading half of the roof with panels.

Netflix for iPhone in the cloud and HTML5

2010-08-26T14:55:00.001-07:00

I posted on this subject a while ago, and generated a lot of confused comments, who read things into my post that weren't there. Today I hope it's a bit clearer what I was talking about, because we released the Netflix iPhone app, which is based on HTML5 for its user interface elements, back ends into the Amazon cloud, and uses conventional, non-HTML5 video playback and DRM. The playback mechanism is the same as the iPad, but to support the scale of expected usage we needed more capacity (it's currently the top free app in the app store as I write this) and we got that by rebuilding our API tier and personalized movie choosing backend to run on the Amazon cloud. The user interface runs in a webkit based browser window in the app, just like on iPad, but the entire UI is built using Javascript with advanced CSS and HTML5 animations to get it to feel like a native iPhone app.

Returning to the theme of my last post, Netflix is hiring engineers to work on cloud tools, platforms and performance, and advanced user interfaces. I think we are breaking new ground in both areas and its an exciting place to be. We have very high standards and are looking for the best people in the industry to come and help us...

Eventual Consistency of Cloud?

2010-08-18T22:24:00.000-07:00

Lori MacVittie wrote that eventually cloud standards will converge around the private cloud standards (RT @swardley). I disagree, and there are several examples I can think of that point to the opposite conclusion, that public cloud will be the standard, and AWS will continue to dominate. The original article is here.

The first point of disagreement is the claim that public clouds aren't getting the input they need from customers to mature their services, because real enterprise customers aren't running in the public cloud. Well, here at Netflix, we are giving Amazon exactly that kind of input. We use every feature they have, we are driving them hard and Amazon is taking the input and improving their product rapidly in ways that benefit all the users of AWS.

My main disagreement is the claim that lots of individual IT departments will converge on a single standard that will win out over public cloud standards. I find this highly implausible, there are a host of vendors feeding technology to enterprise clouds, and they will all do the usual vendor thing of looking for ways to lock in the customer, even if they base on the same standards their implementations will be different. Here's an example, the "private" Enterprise Unix variants Solaris, AIX, HP-UX, IRIX, OSF/1 etc. are all based on the same Unix standards, however the "public" alternatives are Linux and BSD, and Linux has won the mind share in this space. I see Linux as an analogy for the public cloud in the sense that there is a very low barrier to adoption. For Linux, you can download it to run on any old computer for nothing, learn to use it, then build very low cost solutions out of it. For AWS, for a few dollars on the existing Amazon account you use to buy books etc, you can explore all the features and learn to build very powerful systems in a few hours. This has produced a large population of very productive engineers, who know how to use AWS, Linux and other open source tools to solve problems rapidly at low cost using the same tools. In contrast, if every enterprise cloud moves ahead by solving their problems independently they will produce a variety of architectures, each optimized to their own problem, and with their own tooling, and a very small number of people who know how to run each variant. You will also find that every company also uses the public cloud to get stuff done more quickly and cheaply than the IT department, so that will become the common standard.

Part of the thinking behind Netflix' move to the cloud is that large public cloud providers like Amazon will have far more engineers working on making their infrastructure robust, scalable, well automated and secure than any individual enterprise could afford. By using AWS we are leveraging a huge investment made by Amazon, and paying a small amount for it on a month by month basis. We also get to efficiently allocate resources, for example how much does it cost to provision a large cage of equipment in a new datacenter and how long does it take from deciding to do it, to having it running reliably in production? Let's say $10M and many months. Instead we could spend $10M on licensing more movies and TV shows to stream, and grow incrementally in the cloud. In a few months time we have more customers than if we spent the money up front on buying compute capacity and we just keep re-provisioning new instances in the cloud, so we never end up with a datacenter full of inappropriately sized or obsolete equipment. At present, Netflix' growth is accelerating, so it is difficult to guess in advance how much capacity to invest in, but we have already flat-lined our datacenter capacity, and have all incremental capacity happening on AWS. For example, we can just fire up a few thousand instances for a week to encode all the new movies we just bought the rights to, then stop paying for them until another big deal closes. Likewise on The Oscars Awards night, there is a big spike in web traffic, and we can grow on the day and shrink afterwards as needed without planning it and buying hardware a long time in advance.

While the other public and private cloud vendors are competing to come up with standards, we are finding that resumes from the kind of engineers we want to hire already reference their experience with AWS as a de-facto cloud standard. It's also easier to attract the best people if they will learn transferable skills and work on the very latest technologies.

That might sound like a lock-in, but a well designed architecture is layered, and the actual AWS dependencies are very localized in our code base. The bet on the end game is that in coming years, other cloud vendors produce large scale AWS compatible offerings (full featured, not just EC2 and S3), and a very large scale multi-vendor low cost public cloud market is created. Then even a large and fast growing enterprise like Netflix will be an insignificant and ever smaller proportion of the cloud. By definition, you can't be an insignificant proportion of your own private cloud....

Reducing TCP retransmit timeout?

2010-08-16T10:38:00.000-07:00

Cloud networks are lossy and low latency, reducing TCP_RTO_MIN and TCP_DELACK_MIN looks like a good idea, but it looks as if this needs a linux kernel recompile. Anyone else looked at this?
Here is a relevant paper “Safe and Effective Fine-grained TCP Retransmissions for Datacenter Communication“
http://www.cs.cmu.edu/~vrv/papers/sigcomm147-vasudevan.pdf

Open letter to my Sun friends at Oracle

2010-08-06T23:52:00.000-07:00

I recently heard about Illumos via a tweet from Alec Muffett, and responded with my own tweet "I predict that #illumos will be just as irrelevant as Solaris has been for the last few years. Legacy." - personally I haven't logged into a Solaris or SPARC machine for about four years now. There are none at Netflix.

I have also been talking to a few friends who stayed at Sun and are now at Oracle, and there is a common thread that I decided to put out there in this blog post.

This week I presented at a local Computer Measurement Group meeting, talking about how easy it is to use the Amazon cloud to run Hadoop jobs to process terabytes of data for a few bucks [slideshare]. I followed a talk on optimizing your Mainframe software licensing costs by tweaking workload manager limits. There are still a lot of people working away on IBM Mainframes, but it's not where interesting new business models go to take over the world.

The way I see the Oracle/Sun merger is that Oracle wanted to compete more directly with IBM, and they will invest in the bits of Sun that help them do that. Oracle has a very strong focus on high margin sales, so they will most likely succeed in making good money with help from Solaris and SPARC to compete with AIX, z/OS and P-series, selling to late-adopter industries like Banking, Insurance etc. Just look where the Mainframes are still being used. Sun could never focus on just the profitable business on its own, because it had a long history of leading edge innovation that is disruptive and low margin. However, what was innovative once is now a legacy technology base of Solaris and SPARC, and it's not even a topic of discussion in the leading edge of disruptive innovators, who are running on x64 in the cloud on Linux and a free open source stack. There is no prospect of revenue for Oracle in this space, so they are right to ignore it.

That is what I meant when I tweeted that Illumos is as irrelevant as Solaris, and it is legacy computing. I don't mean Solaris will go away, I'm sure it will be the basis of a profitable business for a long time, but the interesting things are happening elsewhere, specifically in public cloud and "infrastructure as code".

You might point to Joyent, who use Solaris, and now have Bryan Cantrill on board, but they are a tiny bit-player in cloud computing and Amazon are running away with the cloud market, and creating a set of de-facto standard APIs that make it hard to differentiate and compete. You might point to enterprise or private clouds, but as @scottsanchez tweeted: "Define: Private Cloud ... 1/2 the features of a public cloud, for 4x the cost", that's not where the interesting things are happening.

So to my Sun friends at Oracle, if you want to work for a profitable company and build up your retirement fund Oracle is an excellent place to be. However, there are a lot of people who joined Sun when it was re-defining the computer industry, changing the rules, disrupting the competition. If you want some of that you need to re-tool your skill set a bit and look for stepping stones that can take you there.

When Sun shut down our HPC team in 2004 I deliberately left the Enterprise Computing market, I didn't want to work for a company that sold technology to other companies, I wanted to sell web services to end consumers, and I had contacts at eBay who took me on. In 2007 I joined Netflix, and it's the best place I've ever worked, but I needed that time at eBay to orient myself to a consumer driven business model and re-tool my skill set, I couldn't have joined Netflix directly.

There are two slideshare presentations on the Netflix web site, one is on the company culture, the other on the business model. It is expected that anyone who is looking for a job has read and inwardly digested them both (its basically an interview fail if you haven't). These aren't aspirational puff pieces written by HR, along with everyone else in Netflix management (literally, at a series of large offsites), I was part of the discussion that helped our CEO Reed Hastings write and edit them both.

What can you do to "escape"? The tools are right there, you don't need to invest significant money, you just need to carve out some spare time to use them. Everything is either free open source, or available for a few cents or dollars on the Amazon cloud. The best two things you can have on your resume are hands on experience with the Amazon Web Services tool set, and links to open source projects that you have contributed to. There isn't much demand for C or C++ programmers, but ObjectiveC is an obvious next step, it's quite fun to code in and you can develop user interfaces for iPhone/iPad in a few lines of code, that back-end into cloud services. Java code (for app servers like Tomcat) on Android phones, Ruby-on-Rails, and Python are the core languages that are being used to build innovative new businesses nowadays. If you are into data or algorithms, then you need to figure out how to use Hadoop, which as I describe in one of my slideshare decks is trivially available from Amazon. You can even get an HPC cluster on a 10Gbit ethernet interconnect from Amazon now. There is hadoop based open source algorithm project called Mahout that is always looking for contributors.

To find the jobs themselves, spend time on LinkedIn. I use it to link to anyone I think might be interesting to hire or work with. Your connections have value since it is always good to hire people that know other good people. Keep your own listing current and join groups that you find interesting, like Java Architecture or Cloud Computing, and Sun Alumni. At this point LinkedIn is the main tool used by recruiters and managers to find people.

Good luck, and keep in touch (you can find me on LinkedIn or twitter @adrianco :-)

Running Netflix in the (public #aws) cloud

2010-08-05T00:17:00.000-07:00

my talk at #qconsf in Nov http://qconsf.com/sf2010/presentation/Running+Netflix+in+the+Cloud

Can OpenStack catch up with AWS? Looks unlikely to me.

2010-07-20T13:18:00.000-07:00

There has been a lot of chatter about the new OpenStack standard in the last few days, and how it could become a third option in the cloud infrastructure market alongside Amazon AWS and VMWare vCloud. For example Randy Bias/Cloudscaling has a good overview of the players.

The real question is whether OpenStack can catch up and become viable. Amazon is always described as being years ahead of the competition and having the lions share of the market. Estimates of the size of their lead depends on who you talk to, but I don't see anyone disputing their lead. On top of that, Amazon is investing heavily, has big customers like Zynga and Netflix stretching and hardening their systems (along with a huge number of small customers), and has already added many features beyond the basic compute (EC2) and storage (S3) that have been copied by others including OpenStack.

So for OpenStack to catch up, means that they have to move faster than Amazon, and leverage the slipstream effect where Amazon has had to educate the market and figure out what works, so competitors can copy their successes. However when I look at the details of the OpenStack specification they appear to be copying some of Amazon's problems as well. In particular the account and authentication model, which does not scale for enterprise use.

At the most basic level, it is infeasible to change the security model of a platform architecture, it's one of the most fundamental starting points that conditions the layers above. Changes to account and authentication management break all the layers of applications and tools that are built on the platform. One of the first problems that Netflix had with Amazon, was the lack of sub-accounts and role based access control (RBAC), and while beating up Amazon on this point for the last two years, we have built our own platform layers and tools to compensate. As a result, we find it impossible to use any of the web consoles or tools produced by Amazon or the many cloud vendors, which assume that there is a single account owner who can do anything. We hope that Amazon will eventually support sub-accounts, and when they do, I expect it will break everyone's tools.

At this point OpenStack is just the base level of the platform, it doesn't really have layers of tools on top yet, but its account and authentication model appears to be exactly the same as Amazon, so they will end up with layers of tooling that don't meet the needs of enterprise customers.

What's the difference between a startup and an enterprise? In a startup, everyone in IT knows the root password to every machine in their infrastructure. In an enterprise root passwords are carefully controlled, they change when someone in-the-know leaves, and specialist groups manage different parts of the system (Network ops can only mess with the switches, DBA's can only mess with the database etc.). The problem with a single account for the cloud is that everyone who needs to do anything to that account can do everything to it, and the common tools are oriented to a single user, managing several accounts, rather than a hierarchy of users managing parts of one account. All the systems in the account need an authentication key to access cloud services, and changing the password and key means you have to re-key every system. Get this wrong and your cloud will evaporate in an instant.

So in my opinion, a necessary but not sufficient condition for OpenStack to eventually catch up with AWS is that they need to build sub-accounts and RBAC into their spec from the start. However it seems much more likely that Amazon will just disappear into the distance from what I've seen so far.

iPhone Tethering Review

2010-07-19T15:22:00.000-07:00

Along with iOS4 and the new ATT data plans they now also support tethering. I setup my wife’s personal iPhone 3GS for tethering, and it worked fine on her work laptop with Windows XP over the USB cable (she could work from the passenger seat while we were driving freeways). You just need to have iTunes installed on the XP machine, even if that isn’t the machine that sync’s the iPhone.

I also got Netflix to switch on tethering on my company iPhone 3GS, and I’m using it via bluetooth to my MacBook Air, easy setup and it also worked well for me.

The downside is that you lose the unlimited data plan, but for a few dollars a month, it can be very useful.

Netflix in the cloud and HTML5

2010-05-19T09:54:00.000-07:00

One of the fun things about working at Netflix is that we are always "leaning forward" and looking for new technologies to leverage. As we roll out our website to run on the Amazon cloud, we are also re-architecting the code base, adding internationalization support and bringing in the latest technologies. One of these is HTML5, which is raising the bar for cross browser support for advanced user interface features, and is now supported by a large and rapidly growing percentage of the visitors to netflix.com. In addition many TV based devices now embed webkit, which is the HTML5 compatible technology that underpins the Safari and Chrome browsers. The user interface engineering team is looking to hire the best and brightest to work on these cutting edge technologies. Here's a write-up for one of the open positions:

Are you passionate about building great website experiences used by millions of visitors each day? Come to Netflix where we are using HTML5 based web technologies to move ecommerce directly onto televisions in our customers’ living rooms. As part of our Customer Acquisition team, you will lead the way to our internationalized television user interface designed to help new customers find Netflix and start streaming movies in seconds. This new experience will be deployed to HTML5 capable embedded browsers and served from our cutting edge cloud based backend service.

---------
Update - Hacking Netflix and TechCrunch picked up on this posting. MG Siegler at TechCrunch decided that I was talking about streaming video and Silverlight, which I wasn't. I was thinking of HTML5 features that let us build very cool user interfaces with drag-and-drop, canvas transforms etc. for the web site, and for embedded TV devices specifically. The Silverlight player is used for PC/Mac playback only, and the basic HTML5 Video doesn't have a viable DRM solution at this point. I'm the Cloud Architect for Netflix, so my involvement is to architect robust and scalable support in the cloud for these new user interfaces.

Garage Structure Completed

2010-05-12T08:43:00.000-07:00

After lots of stops and starts, all that is left to do is some exterior paint and to fit the main doors. Work has already started on the electrical installation and water for fire sprinklers. Since the walls got so wet, they warped and parts of the siding had to be replaced. For some reason Tuffshed did other jobs on days where the weather was nice, and turned up to work on this one whenever it rained. We had what was probably the last rain of the winter on Sunday and Monday, but the roof was already tiled by then. The last video was the trusses being installed, here is the rest of the sequence.

Moving the viewpoint to catch trench digging around the back of the garage.

Cloud Architect

2010-05-06T07:27:00.000-07:00

my role at Netflix just changed, from managing development of personalization in the cloud, to over-all cloud architect for Netflix. New challenges and a much broader scope. One challenge we have as we migrate to the cloud is that we need to build a new way to log and analyze what the systems in the cloud are doing, and we are basing this on Chukwa, Hadoop and Hive, all running in AWS. So we are looking to hire a Hadoop developer, and are looking at the research going on at Berkeley's RAD lab with interest.

AT&T 3G Microcell doesn't support corporate account phones

2010-04-28T22:21:00.000-07:00

I just got an ATT 3G Microcell for home, since our service is intermittent, sometimes so bad I can't send an SMS message successfully (although at times it works fine). We live in the mountains, and have several different poor signals to bond to, so that's probably why it is variable.

In the store, they checked my phone number and said I couldn't activate a Microcell on my account, since it's a Netflix corporate account iPhone. However, my wife has an iPhone on a personal account, so I used that as the activation account.

However, when I tried to add my own phone number to the Microcell's whitelist of up to 10 numbers, I got this error message

"FTC309: invalid subscriber information found" - which got no hits on Google, however after this blog post it should get one..

So the website told me to call the generic ATT support number. After a few short waits, a very polite first level support guy (who had probably never heard of the Microcell before I called) found a well informed business support specialist who was able to confirm that the error code was due to the Microcell only supporting consumer accounts. She agreed that it didn't make a lot of sense, found that they had been running a trial of corporate accounts a few months ago, but didn't know when/if it might change. She forwarded me an internal support document that discussed the limitations of the trial, including the lack of corporate support. Although it does say that it supports all ATT 3G devices.

I installed it anyway, it takes a few hours to set itself up the first time. We will see how well it works for other iPhones, meanwhile I will just have to use Skype to make calls and send SMS... If it doesn't work out I will take it back.

------- Update -------
I contacted the corporate account support team via work, and they over-rode the system to add me to the Microcell. So my corporate account iPhone is now working fine with the Microcell. They are investigating further to see if they can make this a more general fix for other people, but if you are in the same situation as I was, you should be able to escalate to support to get it fixed.

Hiring a cloud platform engineer at Netflix

2010-04-27T09:12:00.000-07:00

If you are interested in the CAP theorem NoSQL systems like Voldemort, Cassandra and SimpleDB, scalability and expect to be at the Velocity Conference in June, then you are the kind of engineer I'm interested in hiring. Netflix is aggressively moving to new technology, hiring only the very best and most experienced engineers and our growth is accelerating.

Garage Trusses Video

2010-04-22T00:21:00.000-07:00

On Tuesday a big truck had some difficulty delivering a 40ft trailer containing the roof framing trusses, but it was raining so that is all that happened.

On Wednesday 21st April it started out dry, rained for a while then cleared up, but they did manage to put up all the trusses, and I made another stop-motion video of the work.

Tuffshed aren't great at letting us know what is going to happen next, we have to call them or ask the workers. I think the next step is to put plywood sheeting on the roof, then get an inspector up to check that the nailing pattern is correct, then they can put the doors and windows in, tile the roof add the external trim and paint it. It should be much warmer and sunny for the next few days.

garage - roof and rain on Tuesday

2010-04-19T02:53:00.000-07:00

the roof trusses will be delivered tomorrow and rain is forecast for Tuesday and Wednesday - so we will see how they deal with that... it sucks that we don't have a roof before the rain.

Garage Framing Walkaround

2010-04-16T09:24:00.000-07:00

The walls are up, but the trusses for the roof aren't ready, so it will be finished off next week. Luckily the weather looks good for the next week or so.

Garage Framing - first day of construction

2010-04-14T18:33:00.000-07:00

Tuffshed turned up with their pre-built sections, assembled them and added some siding and the main door frame. Another stop-motion video on YouTube. It was wet over the weekend, and the ground is very soft, but we have a few dry and warm days in a row forecast, so I'm hoping it will dry out now.

Experiences using an iPad as my primary work laptop

2010-04-10T23:09:00.000-07:00

I spent the last week carrying my iPad with me to meetings and using it as my main system at home an in my cube. My laptop was used for a few things, like the performance monitoring tool AppDynamics - which needs a bigger screen and is flash based, but just about everything else was usable on the iPad.

I have the apple case, which I think is essential, as it holds the iPad up at the right angle to use on a desk or lap, as well as protecting it and making it less slippery to carry around. I got the Video out cable but haven't had a chance to use it yet. After intensive use, the battery is still at 60% or more each evening.

I like the speed at which apps like mail and calendar come up, and they are both very nice to use, I prefer them to entourage on my Mac laptop. I spend a lot of my day reading writing and deleting email. Safari is also fast and worked well for gmail. I imported a PowerPoint presentation to keynote on the iPad and it needed adjustments to font sizes in diagrams, and lost some color coding, but seems like a usable tool. I will switch to keynote on my Mac as well and hopefully it will be compatible.

I have a VNC app, but moving the mouse cursor around is a a pain so it's only useful for emergency access to my laptop.

The other app I use a lot is NewsRack, which tracks about 100 random rss feeds for me by keeping in sync with google reader. I get most of my news and keep up with friends blogs this way. Compared to the iPhone version it seems to update much faster and the extra screen space is well used. The BBC app provides a more visual way to track stories, and the Guardian Eyewitness photography app contains stunning images, it's one of the most popular free apps at the moment.

Typing on the iPad is better than expected with two caveats. I keep hitting the b or n keys when I mean to hitbspace (like that), and the auto correction system is a bit over-active so I have to proofread what I write more than usual.

Of course I have been watching movies with the Netflix app, and have used the eBay and Pandora iPad apps, which look good. I'm still using the iPhone version of echofon as my Twitter reader, but I will try out the official twitter app when it comes out.

Garage - Pouring and Finishing the Foundation

2010-04-08T08:49:00.000-07:00

Last Friday we poured concrete, and then it poured rain. Here's a stop-motion video of the work. Six concrete trucks came, we are high on the mountain, so they may not have been carrying a full load each time. The foundation is very deep on the downhill side, and the slab is 6" deep rather than the 4" minimum used for the walkway down the side and the parking apron.

The first few seconds of each video look messy, this was introduced when youtube encoded it, my original video is fine.

On Saturday, the weather was better, and the team came back to finish the surface and tidy up.

Final bill was about $25K, and I would recommend Amaya Concrete of Morgan Hill, they did a good job.

This week, we let the concrete cure, wait for the ground to dry out and do some basic grading of the ground levels and slopes. Next week the building will be delivered and installed.

IPad makes it hard to embed YouTube in blog

2010-04-07T07:29:00.000-07:00

I'm trying to compose a blog post that embeds a YouTube video, but YouTube doesn't want to give me the HTML fragment to embed the video. If you browse to YouTube on an iPad it doesn't show any options to pick up the embed because it renders in "touch" mode. At the bottom of the page I found an option to switch to desktop mode and get the full page. This renders an embed option but doesn't let me select it to copy here.

So I need to use my MacBook to compose posts.

As the iPad moves the boundaries between content consumption and content generation, people need to rethink the functionality of the mobile and iPhone optimized web sites.

Garage - Inspected and delayed

2010-03-29T21:06:00.000-07:00

We passed inspection by the county this morning, but the weather has turned too bad to pour concrete tomorrow, so we are postponing until after Wednesday's storm has passed, and will resume work on Friday. Today the electrician finished his work on the pipe that carries the power, and the Tuffshed foreman Mike visited to inspect the site and check and adjust the position of the hold-down bolts that are set into the foundation.

My CMG paper on Crunching Data In the Cloud is published in MeasureIT

2010-03-29T21:01:00.000-07:00

The slides are also available at http://www.slideshare.net/adrianco/crunch-your-data-in-the-cloud-with-elastic-map-reduce-amazon-emr-hadoop - and MeasureIT has a PDF of the whole paper at http://www.cmg.org/measureit/

Garage - Working on Saturday

2010-03-27T21:35:00.000-07:00

The concrete team were back today, putting in more rebar and tidying everything up. Chris Ramoutar the Electrician was also here putting in a 2-inch pipe to carry power to the garage. I'm having a high spec 220V and 200Amps dedicated circuit configured so that it's ready to recharge electric cars in the future. I got a work-out clearing up the felled fir tree and chipping the small branches, and collecting large rocks that had been dug up, and arranging them around the pond.

Here's a view from the deck, you can see the apron on the right, and the garage foundation on the left, and Chris working in the trench that takes the power line.

Here's a view from where the driveway will be as it joins the apron, you can see the walkway to the left of the garage.

Later that evening some very confused deer wandered over the foundations, since it has been built over their usual path from the forest out to the front yard.

Garage Site - rebar ready

2010-03-26T21:15:00.000-07:00

Another stop motion video at 30s intervals shows most of the day's work. The tree that was falling over was pushed back the other way and cut up, and they added the parking "apron" in front of the garage.

There is a 4ft wide walkway along the side of the garage by the house, so the main pad is 36 by 36ft. It slopes to the right by three inches over that width so it will drain. The walkway slopes away from the garage by one inch. The apron is 26ft deep, 36ft wide and slopes by 12 inches (4%). The driveway will connect to the side of the apron that is nearest the house.

Garage Site - digging the foundations

2010-03-25T23:00:00.000-07:00

Today the new leach-field pipe was inspected, and the foundations were dug, I did a stop-motion video using iStopMotion on an old MacOS laptop looking down from the house, and most of the day's work is compressed into one minute.

Tomorrow they will do the detailed preparations for the pour on Monday of the garage slab itself (6" thick), perimeter foundation (at least 18" deep, more on the downhill side where it is on in-fill), a walkway down the side of the garage, an "apron" in front of the garage to park on, and pads for the propane tank and generator.

Garage Site - full speed ahead

2010-03-24T22:37:00.000-07:00

Pictures taken this morning, before digging started for the new leach field pipe that's getting inspected and filled back in on Thursday. The foundation layout and trenching will be done on Thursday and Friday, hopefully pouring concrete Monday and inspections on Tuesday. Everything just went from glacially slow to full speed....

In the first picture, taken from a window in the house, the white blob is the propane tank, and you can see the leaning tree.

The second picture is looking back from by the tree. The two low black things by the house are the caps for the septic tank.

I'm setting up a stop motion camera, will try to capture the building process. The MacOS iStopMotion app seems to work fine and has a five day free trial, so we will see how that goes.

Garage Permit - now the fun starts

2010-03-24T12:31:00.000-07:00

I went into the county office and picked up the permit, paid them some more money to cover their costs of inspecting the work as we do it. They gave me back one of the marked up copies of the final site and building plans, and the permit sheet, and I stopped off at a Fedex office to get several copies made. Next stop is Tuffshed, to talk to the foreman who gets it built, and the concrete contractor (Jose Amaya Concrete of Morgan Hill) who normally works with Tuffshed. Jose is available immediately so we may actually get going fairly quickly. First step is to mark out the foundation and pour it.

The new septic tank connection pipe is going in today and being inspected tomorrow.

The Periodic Table of Periodic Tables

2010-03-23T22:18:00.001-07:00

The Periodic Table of Periodic Tables
Originally uploaded by ★keaggy.com

Excellent science geekery

Garage plans approved

2010-03-19T22:49:00.000-07:00

The county turned around the final update immediately and I got a letter in the mail today! Now I just need to visit them to get the actual permit document.

I'm getting started with the first step next week...

Solar Power - Spring Equinox Update

2010-03-19T12:46:00.000-07:00

Spring is here, and we just had a few clear days, shown below. The sunrise and sunset are about 12 hours apart. We installed the system just before the fall equinox, so we now have a good calibration of the output levels through the least sunny part of the year, from now on it will be interesting to track the output as we go to the longest day.

Actual average output over the last 200 or so days is about 14.3 KWh/day. Yesterday was 22.5 KWh, and at the winter solstice low point it was 16 KWh on a clear day as previously blogged.

Garage Plan Resubmitted

2010-03-17T12:04:00.000-07:00

This morning I took the requested clarifications on the building design into the county offices. We should hear back in a week or two, it's minor stuff so I hope this is the last step. We already have our permit for the leach field repair, so I'm starting on that work. We got the septic tank pumped and inspected (it was in good condition) this week, and filed the inspection results with the county, so the next step is to put in a new re-routed pipe and diversion valve between the tank and the leach field.

Garage Plan Detail

2010-03-13T18:52:00.000-08:00

To get everything approved we had to survey the land and get a detailed plan drawn up by a Civil Engineer - we used Terry Szewczyk and he was helpful, competent and kept the costs to a minimum. The picture below is a detail from one of the drawings, which were produced at 1inch to 20ft scale, and shows the important part where the garage goes. For some reason the contour lines altitude is shown based around 2000ft, whereas the real altitude is about 2400ft, but no-one seems to care if that aspect of it is accurate.

Our first step is to replace the pipe that goes from the septic tank to the diversion valve and the remaining leach field, so we start by getting the tank emptied (which needs to be done every few years anyway). The pipe has to dive down a bit then go across under the concrete "apron" that will lead up to the garage, and it won't be buried very deep, so we will be putting down a 4" pipe that is placed inside a 6" pipe for the part that goes under the driveway, to give it extra protection from being crushed.

The final detailed changes we need on the building design plan are now in hand (the layout plan shown above is already approved), so I'm back to the county office again next week to resubmit them and hopefully get the permit.

The other details you can see on the plan are the electrical connection, which will be a 220V -200Amp dedicated circuit capable of charging electric cars one day. The propane tank will be relocated, and hooked up to a generator and the house. There is also a water connection to a fire sprinkler system.

WiFi upgrade

2010-03-10T19:56:00.000-08:00

My five year old wifi hardware got flaky and slow, so I looked around and decided to go with the D-Link DIR-825 as my central node and a DAP-1522 to bridge to my TV etc.

They are both N spec dual frequency systems. Arrived today from Amazon for $225 for the pair.

[Update: they are working fine, I have a 130Mbit/s link speed reported on the 5GHz link between the DIR-825 and the DAP-1522, and I've shut down all the old WiFi networks].

sketching the new garage

2010-02-26T23:49:00.000-08:00

While figuring out what we wanted to build, I used Google Sketchup (the free version) to create a 3D model of the house and garage. Working with 3D takes a bit of getting used to, but it's very similar to the solid modeling I was doing to make custom 3D printed parts for my homebrew phone, and Sketchup is very good at getting these kind of artists-impression images created.

The house really is a box, and the full width deck with steps going back down the side doesn't exist yet, but I was able to get the relative position and size of the garage about right so we could see if it fitted in. We then went to TuffShed and got them to draw up plans for our oversized garage. If you get a garage custom "stick built" at the site, the cost is quite high, mostly labor. Tuffshed pre-fabricate the parts, ship them in, and put up the entire building in about three days. The cost (not including foundations) is about $30K for an 1150 square foot triple wide garage (32ft wide, 36ft deep), with a 16ft wide double door, 8ft wide single door, and 10ft high ceilings. The basic cost is more like $20K, but it cost extra to go from 8ft to 10ft ceilings, and we added a bunch of other extra options.

So far, we have had a few minor issues getting the Tuffshed plans approved by the county planning department. Really just clarifications of materials and codes on the drawings, but the first two tries have failed. Everything is now approved apart from the final tweaks to the drawings of the garage, so I'm hoping to get it taken care of next week.

the garage saga, it gets complicated

2010-02-23T22:56:00.000-08:00

We live out of town, on top of the Santa Cruz mountains. This makes a few things easier, we only have Santa Clara County to deal with, no city rules. We also have more space. The downside is that it can be hard to find people to work on the house, and it can cost more to deliver heavy stuff like gravel than it costs to buy it.

The complications start with a scenic highway. We can't build anything within 100ft of the road, Then there is a well, we can't do things within a 100ft radius of that. Then there is a septic tank and leach field. The leach field is two banks of 250ft long buried perforated pipes which occupies a lot of space, including the space that we want to put a garage on. Then a driveway has to find it's way through and have a turn around area big enough to take a fire engine. All of a sudden, a few acres of land is full.

A few years ago we got a plan drawn up for a modified leach field layout. This removed the upper 250ft field so that we could put the garage there, and added a new 250ft field below. To comply with planning rules we had to show that we could on theory add another 500ft of leach field if the first one failed. So that took up almost all the space.

After all that the biggest possible garage we could fit is a free standing building 32ft wide and 36ft deep. It will have one single and one double door and sit alongside the house about 20ft away and slightly downhill.

To prove that the space existed I got my local ground moving specialist Bob Whalen to use an excavator to dig holes and locate the leach field pipes, and made a big flat area in the right place. Along the way we destroyed the upper leach field before I got permission, we got told off but since the rest of the plan made sense we got away with it and were able to move forward.

At some point the county realized that the plan they had for my driveway bore no relation to what had been put in by the people who built the house, this raised the spectre of a grading violation and possible grading permit, which would be needed if we had to move more than 150 cubic yards to get a driveway that conformed to code.

With a lot of help from Terry and crew at TS Civil Engineering we ended up with a plan for the driveway and leach field that met code, didnt raise a violation and didn't need over 100 cubic yards of grading. With a few tweaks the county finally approved the plan.

cars and a place to keep them...

2010-02-23T00:36:00.000-08:00

I'm going to blog about cars over the coming months, partly because I'm about to start building a big garage to keep them in and I want to document the process.

We bought a house about 5 years ago that had a wooden shed as it's single garage. We had five cars at the time and there seemed to be enough space to build a big garage. However other priorities intervened and it wasn't until the middle of 2009 we finally got serious.

The first action was to make a large flat space, to see how big it could be, then we got bogged down in the planning process and this week we finally have the key approvals in place!

More details to come....

Netflix Free Trial link

2010-02-18T18:23:00.000-08:00

If you were thinking of joining Netflix, click this link to get a two week free trial - it's a new affiliate program that Netflix is starting via the Google Advertiser Network. I will update the free trial link in my iPhone app in the next version.

Bil2010 Millicomputing - The Future In Your Pocket

2010-02-12T07:55:00.001-08:00

Here are my slides for BIL tomorrow, predictions for 2010 and 2011:

Bil2010 Millicomputing - The Future In Your Pocket

View more presentations from Adrian Cockcroft.

Book: Release It! Design and Deploy Production Ready Software by Michael Nygard

2010-01-24T13:55:00.000-08:00

Essential reading for anyone architecting web sites. It contains clear explanations and real-life anecdotes of many of the patterns and anti-patterns that I've run into over the years, and I'm learning some new ones. Now I just have to get all the engineers on my team to read it... Here it is on Amazon.

My iPhone apps 2009

2010-01-02T16:50:00.000-08:00

the apps that I use most frequently: Echofon, Skype, Newsstand, Instant Queue Add for Netflix, Amazon, Kindle, Pandora, WootWatch, CoPilot.

In particular I get most of my news via rss feeds using Newsstand, which synchronizes very well with Google Reader on my laptop. I don't own a kindle, but whenever possible I buy books to read on my iPhone using the kindle app.

My Music 2009

2010-01-02T16:04:00.000-08:00

an eclectic list of the CDs that I played most: by Amanda Palmer, The Duckworth Lewis Method, Fractal, Japonize Elephants, Zoe Keating, Tony Levin, Mirthkon, Muse, Queens of the Stone Age, Neil Young.

Some of it was released in 2009, some of it was new to me in 2009, here are links to the albums.

Amanda Palmer - Who Killed Amanda Palmer
Neil Hannon - The Duckworth Lewis Method
Fractal - Sequitur
Japonize Elephants - 40 Years of our Family
Zoe Keating - One Cello x 16: Natoma
Tony Levin - Stick Man
Mirthkon - Vehicle
Muse - The Resistance
Queens of the Stone Age - Era Vulgaris
Neil Young - Fork in the Road

Solar Power - Winter Solstice and New Ducting Update

2009-12-24T11:49:00.000-08:00

I've previously shared the first few months of output from our solar system, we turned it on at the start of September, and on a clear day we were getting around 24KWh. As the days got shorter and the Sun got lower in the sky, the clear day output dropped off, and earlier this week we hit the winter solstice, the shortest day. Wednesday was a clear day, and we generated about 16KWh, as shown in the plot below.

We have also seen the benefit of our E6-time of use rate plan. We generate electricity during the week while we are mostly out of the house at the highest rate, and use it at the cheapest rate. So we continue to be net consumers of electricity, generating between 50% and 70% of what we use, but our electricity bill is basically zero at this point. In addition, PG&E just gave us a rebate of $67 due to lower energy costs, so the standing charge of $11.50/month is zeroed out for almost six months.

We converted our gas appliances to electric, for heating water, drying clothes and cooking, and currently only use gas for the furnace that heats the house. We looked into replacing the furnace with a ground source heat pump (GSHP) but got wildly varying designs and costs from everyone we talked to so have postponed that idea. The one thing everyone agreed on was that our ducting was sub-standard, with insufficient flow for A/C and barely enough for the furnace, and it was poorly constructed using the lowest quality materials.

So we have just replaced the ducts throughout the house, changing the configuration to include separate zones for upstairs and downstairs, adding a new return path from upstairs and one extra outlet. The ducts are insulated to R8, rather than R4.2, so they don't lose heat, and they have fewer leaks and a properly designed balance of air flow into each room. This is done using something called a Manual-J calculation, and there is also an independent analysis service that verifies that everything is working correctly. The main effect of this is that we will use less propane, and have much better control over temperature upstairs. We often want to heat the bedrooms without wanting to heat downstairs to the same level, and we now have a thermostat in the main bedroom for upstairs, as well as one downstairs.

I'm planning to blog some more about GSHP options, its quite complicated....

Global Warming

2009-12-16T01:48:00.000-08:00

Two essential books to read: Climate Cover-Up: The Crusade to Deny Global Warming by James Hoggan - how the oil and coal industries are behind the denialist spin doctors - will make you angry. Storms of My Grandchildren: The Truth About the Coming Climate Catastrophe and Our Last Chance to Save Humanity by James Hansen - will make you scared.

James Hansen is the NASA scientist who is probably the worlds top climate expert. Industry and politicians don't want to hear the message he has been trying to get through for years, that the situation is much worse than the IPCC projections already. He provides the research background for the safe limit of 350ppm of CO2 that is being promoted by some at Copenhagen at the moment. We are already at 387ppm, and increasing rapidly, and we are seeing irreversible changes in climate as the ice melts. James is calling for a worldwide moratorium on building coal fired power plants as a first step. The oil and coal has to be left underground.

Human civilization has occurred during a time of very stable sea levels, over the last 7000 years. Before that sea levels have changed more rapidly and over a range of hundreds of feet. Human induced global warming is forcing change an order of magnitude stronger than the natural changes in climate, and is still accelerating. With very unstable sea levels, ports and coastal areas are going to flood, and it will not be possible to establish new ports on a permanent basis, as they will be flooded in turn within the lifetimes of our children and grandchildren. The huge numbers of refugees is an even bigger problem.

What can we do?
Read those two books.
Sell coastal property while there are still people who will buy it.
Don't invest in oil, coal and insurance companies.
Put up solar panels and switch from gas to electric appliances.
Sign up at the 350.org web site, join the 350org facebook group.

I get books via the Kindle service and read them on my iPhone, instant delivery, and always with me to read.
Climate Cover-Up: The Crusade to Deny Global Warming

Storms of My Grandchildren: The Truth About the Coming Climate Catastrophe and Our Last Chance to Save Humanity

A good summary of the arguments is at Skeptical Science

I have read everything I can find on this subject over the last year or two, I have an Applied Physics degree and enough science and statistics that I can tell who and what makes sense and who doesn't. Every single denier and contrarian that I have heard of is backed by the coal and oil industries, often through opaque organizations and astroturf groups with innocent sounding names. They have been exposed as liars and cherry pickers, but really they are spin doctors sowing confusion. If you listen to them you are deluding yourself, and you should be investing in real estate in Florida...

CMG09 - Last Time Attendee

2009-12-12T12:15:00.000-08:00

I presented a half day training class and a paper at CMG09 in Dallas last week. I won't be at CMG10. The conference has shrunk to a few hundred attendees, mostly mainframe oriented, a few vendors, and a bunch of independent consultants. Its main value for me was to maintain and extend my social network of capacity planning and performance people. There were some useful and interesting papers, but not enough to justify a full week long conference in a very expensive hotel. CMG10 is in Orlando FL, and I can't justify traveling to the east coast for more of the same.

A few years ago I discussed with the CMG board members what was needed to keep CMG from shrinking into irrelevancy, at the time I was at eBay, and early December is the peak business level for retail industry, so very few capacity/performance experts from retail can get away at that time of year, and I suggested they move the date. The dominant industries attending CMG have been banking, insurance and finance, which have been hit hard in the recession. The other change I advocated was that CMG should be held in the Bay Area, so that it could attract a lot more people from the major Web companies and computer hardware and software companies that are based here. Unfortunately CMG is locked into a long series of Hotel commitments for several years, and can't change its plans.

So my position now, is that I will attend CMG again when it comes to me. In the meantime, I will encourage the people I met every year at CMG to attend the Velocity Conference in San Jose next June. http://en.oreilly.com/velocity2010 - the call for papers closes in January, so we have a few weeks to come up with abstracts.

Heading to CMG09 in Dallas today

2009-12-08T15:41:00.000-08:00

various talks on capacity planning in the cloud on Wed, then I'm presenting on Thurs. My slides are at http://www.slideshare.net/adrianco - this is probably my last visit to CMG, its been shrinking for a while, we will see how it looks on the ground.

HPTS at Asilomar

2009-10-24T17:38:00.000-07:00

I'm attending the High Performance Transaction Systems workshop http;//hpts,ws over the next few days. I'm chairing a session on moving to the cloud. Quality crowd - Bechtolsheim, Stonebraker, Patterson, and so on, see the Agenda... Second time for me, it's an honor to be invited back.

FREE this weekend, iPhone Instant Queue Add for Netflix

2009-10-23T09:39:00.000-07:00

The latest version 5 of IQ+ was released this week, and it is currently Free until Monday morning.

You get your queue, recently watched, and up to 100 movies per list for new releases, new TV, the 12 top level genres, movies you'll love, and seven highly personalized sub-genre and similars lists. All the lists are personally optimized for you by Netflix and will be the same as the lists currently shown on Xbox 360 or Media Center if you have one.

It's a free update for existing users (the app has been out for over a year now) and normally 99c for new users. The app also provides searching the instant catalog and sorting the lists by title, rating, maturity, ending soon etc. Sorting the queue doesn't (yet) change it at Netflix, but helps you find titles that you can then pop to the front or back of the queue with one click.

http://itunes.apple.com/

Please review/rate the latest release. So far thousands of existing users have upgraded, iPhone OS 3.0 or later is required.

iPhone Copilot GPS navigator

2009-10-07T08:43:00.000-07:00

this excellent app is a turn by turn talking navigator that works like a dedicated Garmin or Tomtom unit but it cost $35, with no additional subscription charges, and is amazing value. It's fast, has a very clear display and easy user interface. To mount and power it in the car I got a Belkin TuneBase FM ($89 at Frys) that plugs very firmly into the lighter socket and has a gooseneck that lets you adjust the position. The iPhone clamp works with a case fitted, and I was able to tune the car FM to hear the turn by turn directions along with the music playing. There is an output jack if you have an aux input socket on your car stereo, and a USB to charge a second iPhone on long trips or whatever else you have.

Solar Power - High and low usage and costs for the first month

2009-10-05T23:49:00.000-07:00

The solar array continues to generate over 20KWh on a clear day, but our daily consumption varies a lot. On Sunday we were running the washing machine, dryer, pumping water from the well for a few hours, watching TV and baking/cooking/dish-washing for seven people, and we hit an all time high using almost 50KWh.

The next day, running pretty close to the minimum possible, we used about 13 KWh.

Our total generation for the month of September was about 700 KWh. The output is dropping as the day gets shorter and the sun gets lower, so our 4.2KW system peaked at about 3.4KW at the start of the month, after a rain shower cleaned the panels and the air we saw a peak of 3.7KW for a short period of one day, then it dropped back to 3.3KW and it ended the month at 3.2KW. The daily output on a clear day was 26KWh at the start of September and 24KWh at the end.

Our approximate total consumption for september was 1100 KWh, and since we are on the time-of-use E6 rate the net metering subtracted the 700 we generated and broke down the rest as -100 KWh at peak time (1-7pm), and 500 KWh mid and off-peak. We were billed $11.50 which is the base level for PG&E, and a total of $17 for 400 KWh was added to the annual accumulation.

In a year's time the net for the year is due to get settled. The very complex details are available from PG&E, but basically instead of costing us on a sliding scale of 11c-38c per KWh, the solar array keeps us at the bottom end of the scale and at the E6 rate it costs 8c off peak, 14c mid peak, and while we are running the meter backwards at peak PG&E pay us 30c per KWh. So our typical electric bill of $250 was reduced to $28.

I have been asked about the payback time for the installation. This is part of the calculation that the vendor sales-rep makes for your particular situation. In my case we made it much more complex by replacing the roof shingles, shutting down our use of propane, buying new appliances and increasing our use of electricity. However If we ignore all that and average $200/month savings, then after all the rebates the solar system installation of about $20K has a payback time of less than 8 years on a naive calculation. If we take into account a 5% interest rate on the installation cost it's more like 12 years, and if we assume that electricity costs would rise over that time it's somewhere in-between. I'm also assuming that the value of the house has gone up.

Power Usage Logging with the WELserver

2009-09-27T08:16:00.000-07:00

Reading some forums about ground source heat pump installations, we noticed references to a logging system called the Web Energy Logger that collects data in your house and provides you with a detailed and customized view of how the systems in your house are operating. I bought a WELserver kit and have started installing some of the data collectors.

Here's the summary of what it does:

Simply put, the Web Energy Logger (or WEL) is the most economical way to monitor lots of temperature sensors, and a host of other energy related devices in your home or office. It's ideal to verify the performance or a solar thermal system, or to calculate the efficiency of a geothermal installation, or even to prove that your home really is Net Zero. Not only does the WEL give you a window into your energy environment, but it also provides a live snapshot and a range of trend charts that you can share on the WEB in real-time. Finally, the WEL records all your sensor data in monthly log files that are easy to download and import into programs like Excel. It's compact, versatile, and extremely configurable.

The minimal WEL in a Box is ideal for a DIY'er, and costs $375. In contrast, the more complete WEL Starter Kit is ready to run out-of-the-box with 10 temperature sensors and costs $525. All WEL's come with full monitoring software on the board, and a dedicated WEL logging site included in the price.

The monitors are all strung together in a daisy-chain configuration on a single twisted pair wire using the Dallas 1-Wire signaling technology. I got one of the optional current sensors and installed it in the electric hot water tank. Here is the sensor, one of the power wires needs to be threaded through the hole so it can detect current flow.

Here is the water tank. The orange wire had been installed without its protective shielding, so when that was fitted, we took the opportunity to add the sensor.

The recess in the top of the water tank has enough space to hold the sensor, with the black wire threaded through the hole.

The cover is in two parts, which gives easy access to the wiring again if needed.

At this point, I don't have the WEL configured. I need to add temperature sensors to the water pipes, and work out how to monitor the heat pump once we get it installed in place of the propane furnace. The WEL control box has an ethernet connection, and I'm not sure whether to run ethernet into the basement, or run the 1-wire cable around the house to monitor all the rooms. Spare wires in the existing telephone cabling can be used for this purpose.

GP Musumeci joins Netflix....

2009-09-25T00:52:00.000-07:00

Adrian Cockcroft's Blog: Are any (ex-)PAE Snorclers looking for a new career?

My call was answered.... GP Musumeci started work last week at Netflix. He's the author of O'Reilly System Performance Tuning, and we worked together at Sun PAE in 2000-2002. Since then he worked on the operations side, building Time Warner AoL's video streaming service, and Salesforce.com's infrastructure, then moved to become a developer of recommendation systems and distributed databases. The perfect mix...

Solar Power - Monitoring Consumption

2009-09-25T00:23:00.000-07:00

We finally got Solarcity to install the consumption monitor, and we are now figuring out what the various bumps in the graph correlate to in terms of appliances. The first graph shows consumption on a hot day, with a portable room A/C unit left on all day. There is a characteristic bump in the morning that seems to correlate to re-heating hot water after showers, and random bumps during the day that are probably the hot tub cycling on to filter and maintain water temperature.

Another day where we didn't leave the A/C unit on shows much lower consumption. One of the common psychological effects of measuring something is a compulsion to optimize it, so we are now challenging ourselves to try and reduce consumption...

Since we are now on a time-of-use meter, the cost of electricity at nights and weekends is very low, and the cost in the afternoons during the week is high, but that is when we are generating more than we are using, so in $ terms, the meter is running backwards at high speed, and forwards slowly. This also makes it very hard to calculate the cost savings.

Total generation for the month is looking good. We turned on the system right at the start of the month, so the lifetime total includes a week or so before the output monitoring started to log data, and you can see the last few days of consumption data. We have had typical (for us) weather, sunny from the start of the day most days. We drive down into the clouds each morning. The drop off a few weeks ago was a cloudy spell that included about 0.25in of rain, enough to make sure the solar panels are nice and clean. There is a slow downward trend in the daily output, since the days are getting shorter and the Sun is lower in the sky every day. It's around the equinox at the moment. Over-all, I'm happy with how much energy we are getting, I expected to get longer effective days than most people, and it looks as if we will be generating about 600 KWh this month, our typical consumption has been around 1000 KWh.

Solar Power - Monitoring Output

2009-09-06T11:59:00.000-07:00

The SolarCity web based monitoring system didn't work at first, there was a problem with the data-logging board in the inverter but they were helpful and sent someone round to fix it. So now we can track how much power we are generating. The graph updates every 15 minutes, and it reports the amount of power every half hour, so when it shows 1.7 KWh that is equivalent to 3.4KW averaged over 30 minutes. So far the highest peak instantaneous output reported on the Inverter is about 4.2KW.

We are still waiting for the energy consumption metering to be added, which puts a second line on the same graph.

The day shown was a typical day up here on top of the mountain, it was cloudy elsewhere, but we had basically clear sky all day. and generated over 25KWh.

Are any (ex-)PAE Snorclers looking for a new career?

2009-08-31T23:52:00.000-07:00

I'm looking for someone who has been benchmarking and testing web services/cloud performance to work in Los Gatos CA, no remote work, sorry.

For the uninitiated, PAE = Performance Applications Engineering, one of the groups I was in years ago at Sun that built benchmarks and did performance analysis for a living. Snorcle = Sun + Oracle sinking beneath the waves...

This is a great time to join Netflix, we are growing fast, there are a lot of open positions and we pay very well for star players.

Solar Power - "City" Inspection

2009-08-30T18:03:00.000-07:00

We passed last week (we live outside the city, so it was actually Santa Clara County), and how have another 12 days for PG&E to put in a new time-of-use power meter and bless the installation so we can turn it on.

It took a bit of prodding for SolarCity to get the inspections setup, at least that's the way it seemed. After the install, they sent me an email saying please pay the invoice, but they hadn't sent me an invoice. I waited a few days then asked them if they were going to send me one, then they sent it. After they get paid they are supposed to schedule the inspection. They say it can take a month or so to complete the inspections. After a week or two I emailed and asked when the inspection might happen, and a few days later it was scheduled for the following week. They sent someone to wait at my house for the inspector, so we didn't need to be there, other than making sure they had access to the basement where the inverter is installed.

The ground source heat pump is taking a bit longer than hoped. The supplies have been hard to source on the West Coast, it seems that there are relatively few GSHP installations in California so far. I'm hoping to get the detailed planning approval and install done in the coming week.

Solar Power - Installation Day 4

2009-07-29T22:42:00.000-07:00

All the panels are on the roof, the trench has been re-filled, the wiring is done, it was tested and span the meter backwards. Next step is inspections that can take a few weeks before we can turn it on for real....

I installed the Solarcity Internet monitoring system base station by plugging it into a spare Ethernet port on my DSL modem. It flashed some lights that told me that it had connected to Solarcity successfully, then started scanning to find the monitored devices. It talks to the inverter to track how much power is being generated, and I will have an optional consumption monitor that tells me how much power the house is using. I have to wait for the install to be complete before I can see any output on the solarcity.com web site.
Also, they were out of stock of the consumption monitor, so it will need to be added on later, but it doesn't affect the operation of the system.

I have also figured out the Ground Source Heat Pump, I'm going for a vertical well based installation, its a 3-ton A/C unit and heater made by Carrier that hooks up to my existing propane furnace ducting, and includes a water heater that replaces my propane hot water tank. It exchanges heat with the ground using three 200ft deep wells, and for every KW of electricity it uses, it pumps around 4KW of heat in or out of the house. This is over twice as efficient as a conventional air based A/C unit. It costs more, but there is a 30% federal tax credit on the full price, just like the solar power installation, thanks to Obama's stimulus package (this credit started in February 2009). I'm hoping to have enough solar power to zero out the running costs, we will see how that goes over time.

Classic car events in August

2009-07-29T07:29:00.000-07:00

I go to the Monterey Historic Races every year, and this year I'm also taking my Citroen SM to a new event La Dolce Vita Automobili - it's set up to be a smaller and more fun alternative to the Concorso Italiano, which is on the same day Friday August 14th.

Here is the SM at Crater Lake a few weeks ago.

Solar Power - Installation Day 3

2009-07-28T22:27:00.000-07:00

Today about half the panels were put in place on the roof, and a lot of wiring was put in place.

Here are the installers starting to put panels in place in the morning.

In the evening the lowest row of 20 are in place, and about half of the next row. There will be a total of 60 panels when it's done.

Here is the underside of a panel, in a shipping rack on the truck.

A closeup of the panel's specification label, as I mentioned before, its a Cadmium Telluride CdTe thin film technology panel from First Solar. Each panel is rated at 75W, and the total installation is rated at 4.5KW.

Solar Power - Installation Day 2

2009-07-28T00:31:00.000-07:00

More mounting hardware is now on the roof, the brackets that hold the panels to the rails are now in place. Since the PG&E meter isn't mounted on the house itself, they also had to dig a trench. Tomorrow a county planning dept. inspector will come by to make sure everything is to code.

I also got an email from PG&E today, informing me that they know I have solar being installed and will schedule an appointment to do their own verification and turn the system on, once the hardware is in place.

One subtlety, since I also want to connect a generator for when we get power cuts (I'm on top of the Santa Cruz mountains) we had to decide what order to wire everything together. The sequence is

PG&E service meter - Solar Power system - Generator - House breaker panel

The generator is only wired into a subset of the house circuits, so it is looped into the house breaker panel. When installed, it will have a sensor that can tell whether there is any incoming power from PG&E, and if not, it fires up after 20 seconds or so.

The solar power system detects the PG&E power, synchronizes to the 60Hz phase and adds its own power by providing a few volts more than PG&E, to get the power to flow into the system. If PG&E goes away, the solar system also shuts down, then the generator detects no power, isolates the downstream components and takes over powering the house.

At one point I looked at having batteries rather than a generator, but it is a lot less efficient, they don't last very long and they are far more expensive. A 6KW battery system with inverter was around $10K, while I can get a 8KW propane generator for $2K, and I'm thinking of putting in a 14KW Generac 5503 that is currently available from Amazon for $3.3K delivered, since that can supply a 40A 220V circuit for the heat pump. I'm likely to need it for up to a week each year, and that will be the only thing that uses propane at that point.

Solar Power - Installation Day 1

2009-07-25T15:10:00.001-07:00

Here's the house, showing the mounting points and rails on the roof.

Top end of conduit at the roof line.

Bottom end of conduit at the basement level.

Bottom end and A/C shutoff switch on the front of the house at basement/ground level.

A/C shutoff switch. Safety rules mandate that this is easily accessible on the outside of the house.

Inside the basement, showing the connection to the back of the A/C cutoff switch on the right. In the middle is the D/C cutoff switch that has the conduit from the roof routed to it. There will be another D/C shutoff switch on the roof itself, again for safety.

This is the inverter on the left, and the D/C shutoff switch on the right. The pipe coming up carries the phone line into the house.

Solar Power - Installation Scheduled

2009-07-23T12:33:00.000-07:00

The plans were approved earlier this week, and Solarcity called me on the phone and emailed to setup an installation time, which I was able to arrange very conveniently via email. The assigned installation team are finishing up another job, then I'm next, starting tomorrow (Friday 24th July) for a few days.

There has been nothing to show so far, but I will be taking pictures and posting them.

Looking at the options for electric water heating led us to consider a ground source heat pump (GSHP), plans are still in flux, but I'm looking at adding a heat pump that will heat hot water as well as heat and cool the house. The ground source part is that it doesn't have a conventional air fan based heat exchanger, it runs water through underground pipes. This is relatively expensive, but there are rebates, and it is very efficient. More later...

Velocity conference summary

2009-07-02T07:37:00.000-07:00

Steve Souders writes a good summary of the Velocity Conference. I'd like to present something next year, I have a few ideas already, and have been inspired to try some latency testing projects at work.

Solar Power - plan submitted

2009-07-01T17:53:00.001-07:00

solarcity.com emailed me a very detailed pdf of the plan for 4.5KW on my roof, and submitted it to the County for planning approval today. Meanwhile the house is being re-wired this week for 220V and a 200Amp feed to replace the 110V/100A setup. Also putting in 220V outlets for the kitchen range, hot tub, dryer and hot water heater. Propane tank is now disconnected and I have to go shopping for new appliances...

Playing with Hadoop in the Cloud

2009-06-14T22:28:00.000-07:00

I just finished writing a paper for CMG that is a tutorial on Amazon Elastic Map Reduce. I signed up, it was easy, and after I ran two of their demo jobs I owed them 26 cents. Very cool...

Solar Power - first step

2009-06-03T21:39:00.000-07:00

There is a rebate for solar power systems, and the first step is that the power company (PG&E) reserves a rebate and notifies me and SolarCity that if we install a system by a deadline we will get a certain amount of rebate. I have also got an appointment on Monday for the SolarCity engineer to come and spend a whole day taking measurements and designing the system installation.

Last week we got the shingles replaced, since the old ones were getting near the end of their life, and its a good idea to put the solar panels over a roof that will last for a long time, so they won't need to be removed to fix anything.

I will be at Usenix 09 - San Diego - June 15-17

2009-06-03T21:35:00.000-07:00

I'm attending a tutorial on care and feeding of Hadoop on Monday, teaching my own tutorial on tuning and tools tuesday, and staying for the first day of papers on wednesday. See you there...

New music: MirthKon's Vehicle

2009-06-02T23:31:00.000-07:00

Some of the most inventive new music is coming from "unknown" bands in the greater San Francisco region. Through my connections with Fractal I have been a fan of Headshear for many years and now MirthKon have released their first full CD, "Vehicle". Fractal and Headshear have played many gigs together, and MirthKon founder Wally Scharold played in Headshear for a while a few years ago.

I have seen MirthKon play a few times, and they are currently on a west coast tour, having recently played SF and Berkeley, they are in Hollywood on June 5th and San Diego on June 6th, then Chico on June 18th, and Eugene and Portland on June 19th and 20th. They have a conceptual video extravaganza to enhance the musical experience at live shows.

So, what do they sound like? Well, lets start by saying that if you like Frank Zappa, and wished he was still putting out new music, then you will be very happy to take this Vehicle for a ride to Joe's Garage. Its impossible to really say what Zappa sounds like, since the mix of styles and influences is so diverse, but MirthKon have some of the same elements, along with their own unique character. There is a strong dose of humor, very complex rhythms and time signatures, some jazzy vocals on a few tracks, but a largely instrumental mix that combines clarinet and saxophone with guitar, bass and drums

They dub it "The first full-length CD from the Bay Area's most hyphenated thrash-jazz-prog-chamber-core ensemble", which probably doesn't help much. I have been happily playing it over and over again in my car for the last week, expand your musical horizons and support local music by getting your own copy....

miRthkon is:
Wally Scharold - guitars, vocals, composition, sound design, conceptual design, text, video, producer
Rob Pumpelly - guitars, composition
Nat Hawkes - bass guitar, vocals
Carolyn Walter - piccolo, flute, clarinet, bass clarinet, alto saxophone, tenor saxophone
Jamison Smeltz - alto saxophone, baritone saxophone, vocals
Matt Guggemos - drums and percussion
Jarred McAdams - video, text, conceptual collaborator

Solar Power order placed

2009-05-26T23:01:00.000-07:00

I signed up with http://www.solarcity.com for 4.5Kw of thin film panels (60 of them). Their web site is slick, walks you through the install process, they have online monitoring of the output of the solar array and I also added the option of monitoring the house's consumption of electricity, which is integrated into the same web based monitoring and graphing package. They visit later this week to do the engineering assessment, then I get plans (provided online) and planning approval etc. follows. Should be all done in 2-3 months.

Thin film is better in partial light, and high temperatures, and it can be mounted closer to the roof tiles since it doesn't need an air gap underneath to cool it. We have a hot microclimate with no shade, at 2400ft altitude, above the morning mist most of the time, so the longer power delivery period per day should compensate for the slight reduction in peak efficiency compared to silicon. Solarcity offered both at the same price. There is a combination of a rebate and a tax credit at the moment. The tax credit is about 30% of the cost, and was added recently by Obama's stimulus package.

As it progresses I will post pictures and updates.

FLOW - a homebrew Android gumstix phone (finally)

2009-05-25T08:53:00.000-07:00

See more here.

A few years ago the Silicon Valley Homebrew Mobile Phone Club (SVHMPC) formed out of frustration with the closed market for phones, and set out to build something we could program in our pocket. We made some progress but the prototype Gumstix Goliath board never made it into production, and along came the iPhone, which gave some of us the application platform we were looking for to customize. The OpenMoko project is also limping along, and Google Android is finally out there in volume. The FLOW project is open hardware running Android with a Gumstix CPU module. Nice combination for making custom devices.

Memcached 500K ops/s on Niagara T2

2009-05-20T18:45:00.000-07:00

This is some nice work, scalability analysis to fix lock contention in memcached, then running on an out of the box Solaris T2 gets 500K ops/s or 9.6Gbits/s depending on payload size. The Niagara 2 based T2 has built-in 10Gbit network, and 8 cores with 8 threads in one chip. Sub-millisecond response.

http://blogs.sun.com/zoran/entry/scaling_memcached_500_000_ops

Netflix App Gallery

2009-05-15T18:15:00.000-07:00

The Netflix API now has an App Gallery http://www.netflix.com/AppGallery which features my Instant Queue Add for Netflix iPhone app.

Usenix 09 - San Diego in June

2009-05-02T22:52:00.000-07:00

I'm presenting again next month, at Usenix 09 in San Diego. I'm giving an all day workshop/tutorial on Tuesday June 16th. I have just updated my slides, combining the talks I have been developing over the years into Solaris/Linux Performance Tools and Tuning, which includes the Capacity Planning with Free Tools material that Mario Jauvin and I developed for CMG.

I have also posted the slides to http://www.slideshare.net/adrianco
.

Please come along, I look forward to seeing you in San Diego.

Cheers Adrian

What I did for earth day...

2009-04-21T22:26:00.000-07:00

Update: the electric mower works well, cuts better than my old gas mower, and the drive option works very well. Since the batteries make it heavier than a gas mower, the power drive makes a big difference. Hacking through very long grass, the batteries last about half an hour, keeping previously cut grass trimmed short I think it would last a lot longer. Its enough for me]

Actually, what really happened was that I tried to mow the grass last weekend and decided that my old gas lawnmower wasn't up to the job, so I did some research and bought an electric one. Its quiet, doesn't need gas, the throttle and air intake don't need cleaning out and just starts when I want it to.

I ordered it from Amazon.com on Sunday evening, as usual, with the free 5-7 day shipping option, and it arrived on Tuesday. I guess Amazon had nothing better to do than ship stuff... but it was nice to save $67 for shipping a 100 lb package and still get it very fast.

I got the latest biggest electric I could find, the Epic EP21H 24Volt cordless self propelled 21 inch. The rear wheels are belt driven from the main motor via a gearbox. It has a high efficiency brushless motor that drives the blade directly and two 12V lead acid batteries and should run for 40-50 minutes per charge. This is plenty of time for me to get fed up with mowing and charge it overnight. Its a bit bigger and more rugged than my old 4HP 20inch gas mower, and has a side discharge option so won't get clogged trying to mulch tall grass. I'll try it out after work.

I've also been looking forward to getting solar power installed on my roof, and I'm finalizing plans to make a commitment to get it done this year. I will blog my way through the process as it happens...

Fractal iPhone app source code

2009-04-13T07:15:00.000-07:00

I have posted the source to Fractal's iPhone app to google code. I'm hoping that other bands can replace the artwork and URLs and quickly get their own promotional apps out there. If we end up with bug-fixes and improvements, then everyone gains....

iPhone apps that don't work on 3.0beta2

2009-04-09T10:29:00.000-07:00

Both dictionary.com and PocketFlix fail immediately after they launch with iPhone os 3.0beta2

Most other apps seem to be working, and beta2 is faster and more stable.

Fractal - free iPhone app

2009-04-05T09:02:00.000-07:00

App store link

It's finally ready for download.... Don't forget, Hotel Utah SF Sat April 18th is the next gig.

Using iPhone 3.0 beta

2009-03-18T09:41:00.000-07:00

So far its working well, all the previously installed apps that I have tried work. I did a cut and paste from notes to Facebook, listened to some Pandora, downloaded a new app from the AppStore. I was a bit concerned about jumping on the release so early, but haven't found any downsides yet.

I have an older Mac that my first gen iPhone treats as its home machine, I upgraded that one to SDK3.0beta and also used it to do the full backup and restore and re-sync needed to get my iPhone running 3.0. I have a newer MacBook Air that is my work machine, and I have left that with SDK2.2.1, so I can work on current apps and make sure they run on 3.0.

What's on my iPhone: Basic Instructions

2009-02-26T01:00:00.000-08:00

One of my favorite cartoons, a collection of them is bunded as an app. Its not a live feed, but its good to be able to show people "how to wash a cat"... There is also a book out.

What's on my iPhone: GoSkyWatch

2009-02-24T07:56:00.000-08:00

Years ago when I had a Treo, a my friend Gred Oakes introduced me a to planetarium app, and I really liked being able to look up at night, and be able to identify the stars and planets. I live on top of a mountain, and have a great view of the night sky, and one of the first apps I paid for on my iPhone was GoSkyWatch. It has a cool feature that the accelerometer can tell you are holding it upside down snd it attempts to align its view with what you should be seeing. On a 2G iPhone with no GPS it doesn't know which way north is, so that doesn't work so well.

What's on my iPhone: Fring (Skype and IM's)

2009-02-17T07:20:00.000-08:00

When I want to make an international call from my iPhone (or from my home phone) I use Skype. There is no native Skype application for the iPhone, so I use Fring,
although today's announcement of a deal with Nokia includes hints that they are working on more mobile platforms. They support Android already, so their main missing link at this point is the iPhone.

Fring is a decent substitute for now. I mostly use it to keep up with Skype and YIM chat sessions when I'm away from my laptop.

Bill Scott's Designing Web Interfaces book - and he's hiring...

2009-02-07T00:31:00.000-08:00

Here is a (long) slideshare presentation of Bill's talk, there is a webcast of it as well. Bill is also hiring for his group of UI engineers at Netflix, so if you are the best of the best of the best and want to work with the master himself, go for it!

Designing Web Interfaces Book - O'Reilly Webcast

View more presentations from Bill Scott. (tags: rich ajax)