Wednesday, July 29, 2009

Classic car events in August

I go to the Monterey Historic Races every year, and this year I'm also taking my Citroen SM to a new event La Dolce Vita Automobili - it's set up to be a smaller and more fun alternative to the Concorso Italiano, which is on the same day Friday August 14th.

Here is the SM at Crater Lake a few weeks ago.

Tuesday, July 28, 2009

Solar Power - Installation Day 3

Today about half the panels were put in place on the roof, and a lot of wiring was put in place.

Here are the installers starting to put panels in place in the morning.


In the evening the lowest row of 20 are in place, and about half of the next row. There will be a total of 60 panels when it's done.



Here is the underside of a panel, in a shipping rack on the truck.


A closeup of the panel's specification label, as I mentioned before, its a Cadmium Telluride CdTe thin film technology panel from First Solar. Each panel is rated at 75W, and the total installation is rated at 4.5KW.

Solar Power - Installation Day 2

More mounting hardware is now on the roof, the brackets that hold the panels to the rails are now in place. Since the PG&E meter isn't mounted on the house itself, they also had to dig a trench. Tomorrow a county planning dept. inspector will come by to make sure everything is to code.

I also got an email from PG&E today, informing me that they know I have solar being installed and will schedule an appointment to do their own verification and turn the system on, once the hardware is in place.

One subtlety, since I also want to connect a generator for when we get power cuts (I'm on top of the Santa Cruz mountains) we had to decide what order to wire everything together. The sequence is

PG&E service meter - Solar Power system - Generator - House breaker panel

The generator is only wired into a subset of the house circuits, so it is looped into the house breaker panel. When installed, it will have a sensor that can tell whether there is any incoming power from PG&E, and if not, it fires up after 20 seconds or so.

The solar power system detects the PG&E power, synchronizes to the 60Hz phase and adds its own power by providing a few volts more than PG&E, to get the power to flow into the system. If PG&E goes away, the solar system also shuts down, then the generator detects no power, isolates the downstream components and takes over powering the house.

At one point I looked at having batteries rather than a generator, but it is a lot less efficient, they don't last very long and they are far more expensive. A 6KW battery system with inverter was around $10K, while I can get a 8KW propane generator for $2K, and I'm thinking of putting in a 14KW Generac 5503 that is currently available from Amazon for $3.3K delivered, since that can supply a 40A 220V circuit for the heat pump. I'm likely to need it for up to a week each year, and that will be the only thing that uses propane at that point.

Saturday, July 25, 2009

Solar Power - Installation Day 1

Here's the house, showing the mounting points and rails on the roof.



Top end of conduit at the roof line.



Bottom end of conduit at the basement level.



Bottom end and A/C shutoff switch on the front of the house at basement/ground level.



A/C shutoff switch. Safety rules mandate that this is easily accessible on the outside of the house.



Inside the basement, showing the connection to the back of the A/C cutoff switch on the right. In the middle is the D/C cutoff switch that has the conduit from the roof routed to it. There will be another D/C shutoff switch on the roof itself, again for safety.



This is the inverter on the left, and the D/C shutoff switch on the right. The pipe coming up carries the phone line into the house.

Thursday, July 23, 2009

Solar Power - Installation Scheduled

The plans were approved earlier this week, and Solarcity called me on the phone and emailed to setup an installation time, which I was able to arrange very conveniently via email. The assigned installation team are finishing up another job, then I'm next, starting tomorrow (Friday 24th July) for a few days.

There has been nothing to show so far, but I will be taking pictures and posting them.

Looking at the options for electric water heating led us to consider a ground source heat pump (GSHP), plans are still in flux, but I'm looking at adding a heat pump that will heat hot water as well as heat and cool the house. The ground source part is that it doesn't have a conventional air fan based heat exchanger, it runs water through underground pipes. This is relatively expensive, but there are rebates, and it is very efficient. More later...

Thursday, July 02, 2009

Velocity conference summary

Steve Souders writes a good summary of the Velocity Conference. I'd like to present something next year, I have a few ideas already, and have been inspired to try some latency testing projects at work.

Wednesday, July 01, 2009

Solar Power - plan submitted

solarcity.com emailed me a very detailed pdf of the plan for 4.5KW on my roof, and submitted it to the County for planning approval today. Meanwhile the house is being re-wired this week for 220V and a 200Amp feed to replace the 110V/100A setup. Also putting in 220V outlets for the kitchen range, hot tub, dryer and hot water heater. Propane tank is now disconnected and I have to go shopping for new appliances...

Sunday, June 14, 2009

Playing with Hadoop in the Cloud

I just finished writing a paper for CMG that is a tutorial on Amazon Elastic Map Reduce. I signed up, it was easy, and after I ran two of their demo jobs I owed them 26 cents. Very cool...

Wednesday, June 03, 2009

Solar Power - first step

There is a rebate for solar power systems, and the first step is that the power company (PG&E) reserves a rebate and notifies me and SolarCity that if we install a system by a deadline we will get a certain amount of rebate. I have also got an appointment on Monday for the SolarCity engineer to come and spend a whole day taking measurements and designing the system installation.

Last week we got the shingles replaced, since the old ones were getting near the end of their life, and its a good idea to put the solar panels over a roof that will last for a long time, so they won't need to be removed to fix anything.

I will be at Usenix 09 - San Diego - June 15-17

I'm attending a tutorial on care and feeding of Hadoop on Monday, teaching my own tutorial on tuning and tools tuesday, and staying for the first day of papers on wednesday. See you there...

USENIX '09

Tuesday, June 02, 2009

New music: MirthKon's Vehicle

Some of the most inventive new music is coming from "unknown" bands in the greater San Francisco region. Through my connections with Fractal I have been a fan of Headshear for many years and now MirthKon have released their first full CD, "Vehicle". Fractal and Headshear have played many gigs together, and MirthKon founder Wally Scharold played in Headshear for a while a few years ago.

I have seen MirthKon play a few times, and they are currently on a west coast tour, having recently played SF and Berkeley, they are in Hollywood on June 5th and San Diego on June 6th, then Chico on June 18th, and Eugene and Portland on June 19th and 20th. They have a conceptual video extravaganza to enhance the musical experience at live shows.

So, what do they sound like? Well, lets start by saying that if you like Frank Zappa, and wished he was still putting out new music, then you will be very happy to take this Vehicle for a ride to Joe's Garage. Its impossible to really say what Zappa sounds like, since the mix of styles and influences is so diverse, but MirthKon have some of the same elements, along with their own unique character. There is a strong dose of humor, very complex rhythms and time signatures, some jazzy vocals on a few tracks, but a largely instrumental mix that combines clarinet and saxophone with guitar, bass and drums

They dub it "The first full-length CD from the Bay Area's most hyphenated thrash-jazz-prog-chamber-core ensemble", which probably doesn't help much. I have been happily playing it over and over again in my car for the last week, expand your musical horizons and support local music by getting your own copy....

miRthkon is:
Wally Scharold - guitars, vocals, composition, sound design, conceptual design, text, video, producer
Rob Pumpelly - guitars, composition
Nat Hawkes - bass guitar, vocals
Carolyn Walter - piccolo, flute, clarinet, bass clarinet, alto saxophone, tenor saxophone
Jamison Smeltz - alto saxophone, baritone saxophone, vocals
Matt Guggemos - drums and percussion
Jarred McAdams - video, text, conceptual collaborator

Tuesday, May 26, 2009

Solar Power order placed

I signed up with http://www.solarcity.com for 4.5Kw of thin film panels (60 of them). Their web site is slick, walks you through the install process, they have online monitoring of the output of the solar array and I also added the option of monitoring the house's consumption of electricity, which is integrated into the same web based monitoring and graphing package. They visit later this week to do the engineering assessment, then I get plans (provided online) and planning approval etc. follows. Should be all done in 2-3 months.

Thin film is better in partial light, and high temperatures, and it can be mounted closer to the roof tiles since it doesn't need an air gap underneath to cool it. We have a hot microclimate with no shade, at 2400ft altitude, above the morning mist most of the time, so the longer power delivery period per day should compensate for the slight reduction in peak efficiency compared to silicon. Solarcity offered both at the same price. There is a combination of a rebate and a tax credit at the moment. The tax credit is about 30% of the cost, and was added recently by Obama's stimulus package.

As it progresses I will post pictures and updates.

Monday, May 25, 2009

FLOW - a homebrew Android gumstix phone (finally)

See more here.

A few years ago the Silicon Valley Homebrew Mobile Phone Club (SVHMPC) formed out of frustration with the closed market for phones, and set out to build something we could program in our pocket. We made some progress but the prototype Gumstix Goliath board never made it into production, and along came the iPhone, which gave some of us the application platform we were looking for to customize. The OpenMoko project is also limping along, and Google Android is finally out there in volume. The FLOW project is open hardware running Android with a Gumstix CPU module. Nice combination for making custom devices.

Wednesday, May 20, 2009

Memcached 500K ops/s on Niagara T2

This is some nice work, scalability analysis to fix lock contention in memcached, then running on an out of the box Solaris T2 gets 500K ops/s or 9.6Gbits/s depending on payload size. The Niagara 2 based T2 has built-in 10Gbit network, and 8 cores with 8 threads in one chip. Sub-millisecond response.

http://blogs.sun.com/zoran/entry/scaling_memcached_500_000_ops

Friday, May 15, 2009

Saturday, May 02, 2009

Usenix 09 - San Diego in June

I'm presenting again next month, at Usenix 09 in San Diego. I'm giving an all day workshop/tutorial on Tuesday June 16th. I have just updated my slides, combining the talks I have been developing over the years into Solaris/Linux Performance Tools and Tuning, which includes the Capacity Planning with Free Tools material that Mario Jauvin and I developed for CMG.

USENIX '09

I have also posted the slides to http://www.slideshare.net/adrianco
.

Please come along, I look forward to seeing you in San Diego.

Cheers Adrian

Tuesday, April 21, 2009

What I did for earth day...

Update: the electric mower works well, cuts better than my old gas mower, and the drive option works very well. Since the batteries make it heavier than a gas mower, the power drive makes a big difference. Hacking through very long grass, the batteries last about half an hour, keeping previously cut grass trimmed short I think it would last a lot longer. Its enough for me]

Actually, what really happened was that I tried to mow the grass last weekend and decided that my old gas lawnmower wasn't up to the job, so I did some research and bought an electric one. Its quiet, doesn't need gas, the throttle and air intake don't need cleaning out and just starts when I want it to.

I ordered it from Amazon.com on Sunday evening, as usual, with the free 5-7 day shipping option, and it arrived on Tuesday. I guess Amazon had nothing better to do than ship stuff... but it was nice to save $67 for shipping a 100 lb package and still get it very fast.

I got the latest biggest electric I could find, the Epic EP21H 24Volt cordless self propelled 21 inch. The rear wheels are belt driven from the main motor via a gearbox. It has a high efficiency brushless motor that drives the blade directly and two 12V lead acid batteries and should run for 40-50 minutes per charge. This is plenty of time for me to get fed up with mowing and charge it overnight. Its a bit bigger and more rugged than my old 4HP 20inch gas mower, and has a side discharge option so won't get clogged trying to mulch tall grass. I'll try it out after work.

I've also been looking forward to getting solar power installed on my roof, and I'm finalizing plans to make a commitment to get it done this year. I will blog my way through the process as it happens...

Monday, April 13, 2009

Fractal iPhone app source code

I have posted the source to Fractal's iPhone app to google code. I'm hoping that other bands can replace the artwork and URLs and quickly get their own promotional apps out there. If we end up with bug-fixes and improvements, then everyone gains....

Thursday, April 09, 2009

iPhone apps that don't work on 3.0beta2

Both dictionary.com and PocketFlix fail immediately after they launch with iPhone os 3.0beta2

Most other apps seem to be working, and beta2 is faster and more stable.

Sunday, April 05, 2009

Fractal - free iPhone app

App store link

It's finally ready for download.... Don't forget, Hotel Utah SF Sat April 18th is the next gig.

Wednesday, March 18, 2009

Using iPhone 3.0 beta

So far its working well, all the previously installed apps that I have tried work. I did a cut and paste from notes to Facebook, listened to some Pandora, downloaded a new app from the AppStore. I was a bit concerned about jumping on the release so early, but haven't found any downsides yet.

I have an older Mac that my first gen iPhone treats as its home machine, I upgraded that one to SDK3.0beta and also used it to do the full backup and restore and re-sync needed to get my iPhone running 3.0. I have a newer MacBook Air that is my work machine, and I have left that with SDK2.2.1, so I can work on current apps and make sure they run on 3.0.

Thursday, February 26, 2009

What's on my iPhone: Basic Instructions

One of my favorite cartoons, a collection of them is bunded as an app. Its not a live feed, but its good to be able to show people "how to wash a cat"... There is also a book out.

Tuesday, February 24, 2009

What's on my iPhone: GoSkyWatch

Years ago when I had a Treo, a my friend Gred Oakes introduced me a to planetarium app, and I really liked being able to look up at night, and be able to identify the stars and planets. I live on top of a mountain, and have a great view of the night sky, and one of the first apps I paid for on my iPhone was GoSkyWatch. It has a cool feature that the accelerometer can tell you are holding it upside down snd it attempts to align its view with what you should be seeing. On a 2G iPhone with no GPS it doesn't know which way north is, so that doesn't work so well.

Tuesday, February 17, 2009

What's on my iPhone: Fring (Skype and IM's)

When I want to make an international call from my iPhone (or from my home phone) I use Skype. There is no native Skype application for the iPhone, so I use Fring,
although today's announcement of a deal with Nokia includes hints that they are working on more mobile platforms. They support Android already, so their main missing link at this point is the iPhone.

Fring is a decent substitute for now. I mostly use it to keep up with Skype and YIM chat sessions when I'm away from my laptop.

Saturday, February 07, 2009

Bill Scott's Designing Web Interfaces book - and he's hiring...

Here is a (long) slideshare presentation of Bill's talk, there is a webcast of it as well. Bill is also hiring for his group of UI engineers at Netflix, so if you are the best of the best of the best and want to work with the master himself, go for it!


Wednesday, February 04, 2009

Fractal CD Sequitur now available from cdbaby.com

Fractal's new CD "Sequitur" is now available to review and buy on CD Baby at:

http://cdbaby.com/cd/fractal3

'Music to tap your brain to'. Progressive rockers Fractal return with an eclectic musical trip featuring complex rhythms, style-bending and -blending, and existential musical and lyrical outcries.

Friday, January 30, 2009

Fractal Zoom video with Ellipsis

Here is a collaboration between Eric Bigas, who makes Fractal Zoom videos and the band Fractal who have provided Ellipsis as the soundtrack to this video.

Thursday, January 29, 2009

Fractal Performing Monkey's Paw at Hotel Utah

All the songs in Fractal's set in SF are now up on YouTube, I also recorded the Seattle and Portland gigs but had problems getting a good quality soundtrack to the video. The short and intense finale to their set is Monkey's Paw. Fractal and Headshear will be back at the Hotel Utah in SF on Saturday April 18th, put it in your calendar now!

Thursday, January 22, 2009

What's on my iPhone: i.Wund.com

My favorite weather app is actually a lauchpad bookmarked web site rather than an appstore download. http://i.wund.com/ is an iPhone optimized version of the Weather Underground site www.wunderground.com that I upload to from the personal weather station at my house. The best features are a zoomable local weather radar view, and the option to show the detailed technical weather forecast in the form that meterologists use to talk to each other, rather than the dumbed down version everyone else sees.

Tuesday, January 20, 2009

What's on my iPhone: apps I don't use

Last.fm, VNC, Remote, Weatherbug. I installed these but never use them. I tend to listen to Pandora, don't need a remote console, find the Apple remote inconvenient to start up and prefer other weather apps.

Saturday, January 17, 2009

Netflix: I'm hiring a very senior engineer / application architect

Here's the details and links to apply via http://jobs.netflix.com.

Netflix really is one of the best places to work and we are hiring strongly, there are several other openings for engineers and managers in Web Engineering. We are very picky, only hire the best people and the best fit for the job...

Good luck!

The Position:
As a leading member of the Merchandising Systems development team, you will re-define the website architecture into components and services, enhancing the agility and pace of development at Netflix as we scale up and leverage cloud computing technologies to support our personalized movie selection algorithms. As an exceptionally talented and experienced architect you will hold your own among a very demanding peer group, you will learn a lot and so will we.

Netflix is scaling the company by increasing talent density in high performance, experienced and motivated teams. There is a flat organization, minimal process, and an emphasis on inter-personal over-communication to stay coordinated while keeping everything extremely agile.

Responsibilities will include:
• Architectural leadership, broad knowledge of cloud/web services and object oriented technologies and designs
• You will need to clearly articulate your design choices, mentoring and educating our engineering community
• Hands on coding of the key objects and interfaces that define the architectural boundaries of components and services
• Untangling existing subsystems using tools such as Structure 101, re-organizing build systems to create re-usable component jars
• Migration of existing SQL based persistence models to distributed forms that efficiently scale to hold billions of data items for tens of millions of customers

Qualifications:
• 10+ years of application development experience, agile methods and architectural responsibility
• Mastery of object oriented architecture and design in Java
• Experience with high-traffic, highly scaling, multi-threaded distributed software design patterns
• Deep conceptual understanding of scalability, performance, queuing, availability, coherency, caching, synchronization
• Fluency in SQL and an expert in the evolution of persistent storage: Sharding, Hadoop, Eventual Consistency, SimpleDB, Dynamo
• Demonstrated ability to publish and present your ideas via conference presentations, technical blog postings, or published books or papers
• A Bachelors or Masters in Computer Science or equivalent engineering discipline

Characteristics of Success at Netflix:
• Freedom with responsibility. A focus on effective execution
• Strong customer orientation, both internally and externally
• Strategic thinking both technically and business-wise
• Self motivated / self starter
• Personal drive to achieve world class software development
• Desire to work in a fast-paced, evolving, growing, dynamic environment
• Love of technical challenges and a sense of pride in solving them
• Ability to work well in a cross functional environment

Friday, January 16, 2009

What's on my iPhone: Currency and Light

Currency makes it easy to keep track of how far the pound has fallen so far (which makes me happy because I get more for my dollars) and Light is trivial but very useful. It just turns on a full brightness white screen that makes a good flashlight.

Wednesday, January 14, 2009

Fractal Gig - Saturday Jan 17th - Hotel Utah San Francisco

I'm inflicting some new music by Fractal on people who visit my blog... They have just released their new CD Sequitur, which contains a varied mix of quite unique and hard to categorize music. I hope to see people at the Hotel Utah in San Francisco on Saturday, and for those of you in Seattle, they are playing Thursday 22nd, and Portland OR on Friday 23rd. Please connect to Fractal via Myspace or Facebook or Bebo.

Capacity Planning for Cloud Computing - circa 2003

Here is a link to a slideshare upload, of a talk I gave publicly in 2003, on Capacity Planning for Virtualized Datacenters (known as N1 at the time, which can be seen as a nascent approach to cloud computing).

I think it contains some interesting ideas, and dates reasonably well. I'm going to revisit some of these concepts as I think more about how to do capacity planning in the cloud.

Tuesday, January 13, 2009

what's on my iPhone: Bloomberg

The built in stock market app just gives prices but I use the Bloomberg app to get news and more detailed information on stocks.

Monday, January 12, 2009

Solid State Disks - 1 TB in a 2.5 inch announced

Like I said... the next thing for SSD's is to have higher storage capacity in each of the form factors than spinning rust. Here's another step in that direction. http://www.engadget.com/2009/01/10/puresilicon-introduces-worlds-first-1tb-2-5-inch-ssd/. SSD's already have disks beaten on all the performance metrics. This one maxes out the SATA interface and does 300MBytes/s transfer rates.

I've also been seeing prices dropping as volumes increase for laptops with SSDs. I think the end of the line for rust will be 2010, as SSD's will no longer have a worse cost/GB and they already have higher capacity and performance.

Sunday, January 11, 2009

What's on my iPhone: Easy Wi-Fi

You can get free AT&T Wifi at Starbucks, and this app streamlines the process of getting online.

Friday, January 09, 2009

What's on my iPhone: Facebook

Works better than the web site... its also my main mechanism for photo-blogging, sharing something I've seen.

Sunday, January 04, 2009

What's on my iPhone: my Pandora station

Pandora is one of my favorite apps. You can listen to Adrian's Prog Radio station too.

What's on my iPhone: Night Camera

Its very simple but it works amazingly well. When its dark, most non-flash digital pictures suffer from a lot of motion blur due to camera shake. The night camera app uses the accelerometer to wait until the camera has been still for long enough to get a clear picture.

Twitter fodder

I just used twitterfeed.com to tweet when I blog (infrequently).

Sunday, December 21, 2008

Capacity Planning for Cloud Computing



At the recent CMG conference I volunteered to lead a panel session on Capacity Planning for Cloud Computing, and these slides were the result.

Now, this was a conference full of people who already know how to do capacity planning, so they were interested in what was new about Cloud Computing (or even what it was all about from first principles), so the slides do not explain how to do capacity planning in the cloud, they talk about what changes for the capacity planner in this new world.

I'm going to be developing more material in this area over the coming year. I have a head start, in 2002 and 2003 I led projects at Sun where I researched how to do Capacity Planning in Virtualized Datacenters, built some tools, filed a couple of patents, presented papers at conferences and failed to get Sun's N1 project to implement some of it. Well, it seems I was a few years ahead of my time, so I'm going to start by digging out the papers I published at the time.

Solid State Disks - time to give up that iron oxide habit

An interesting note at http://www.theregister.co.uk/2008/12/19/sun_micron_extended_flash_life/ talks about a 2009 technology that gives flash a million write cycles at the NAND cell level. This adds two orders of magnitude to the "typical" cycle lifetime that is often used as an argument that flash isn't ready for prime time.

In any case this applies to each block on the SSD, and blocks are remapped using write leveling, and writes are cancelled using RAM based SSD controllers, and when the limit is reached, only that block becomes unusable... Some people seem to think that as soon as you do 10,000 writes to an SSD it fails totally, like a head crash on a disk.

I also see that at least one vendor has announced a 512GB SSD in a 2.5" laptop disk format package.

So to reiterate something I've been saying for a long time, spinning rust is dead, and a large number of basic assumptions about how computers behave and the best way to architect them are now wrong. In 2009 SSD's will be faster for read, faster for write, faster for sequential and much much faster for random access, more reliable, more durable, lower power, higher capacity, than discs. Give it another year or so and they will be cheaper as well.

SAN's are now a complete waste of time. There is so much reliable I/O performance available in a single drive, that it makes much more sense to put SSD's in the systems and access them directly. Accessing an SSD over a SAN adds a huge latency and cost overhead. It makes much more sense to use node-to-node replication for critical data (log scraping databases or cluster filesystems).

Monday, November 17, 2008

iPhone Instant Queue Add for Netflix - get it now...

I've released a second version of my very own iPhone app. I also reduced the price from 99c to FREE for the next few days. Please check it out and write a review for me on the app store. More info at http://www.millicomputer.com.

Wednesday, October 15, 2008

Music - Amanda Palmer and Estradasphere


I listen to a lot of music, and the really good stuff gets played over and over again while I'm driving. I also think while I'm driving that I should write a blog post about my favorites but then I forget... until now!

My favorite new music for 2007 was "Palace of Mirrors" by Estradasphere. I found that I never tired of it, and listened to it for months on end. It helps that their music is extremely varied in style and impossible to categorize. You can find out more and listen to music at their web site. I suggest you start with "Smuggled Mutation", there is no such thing as a typical Estradasphere song as they combine just about every musical genre, usually in the same song. However this track shows off their talents very well. There is a live DVD of Palace of Mirrors which is also extremely well done. I have seen them play live several times and they are an excellent live band.

During 2008 Estradasphere collaborated with Amanda Palmer of the Dresden Dolls, and her new album "Who killed Amanda Palmer" has become my favorite pick for 2008. The web site is very interesting, giving a lot of background story details about the making of the album and videos of most of the tracks. The music is distinctively Amanda, but expanded well beyond the Dresden Dolls format, with a much fuller sound on some tracks. The key for me is that the music is very moving and emotional, and can sustain interest for a lot of repeated listening.

I'm also still waiting for Fractal to finish their second album "Sequitur", I've been listening to pre-release versions as they work towards the final cut. I'm not holding my breath, as its taken years to get this far, but its getting very close to being completed.

Monday, September 29, 2008

Gene Saunders RIP

A good friend and well respected colleague from Sun is no more...
More info on Geoff Arnold's blog

For the last few years a group of us have been keeping in touch via email, sharing thoughts and amusing web links. Gene was a key part of that conversation, and I miss him already.

Tuesday, September 23, 2008

Tutorial at LISA08 San Diego Nov 9th

I'm going to LISA '08

I'm giving my usual performance tools and tuning for Solaris/Linux class on Sunday 9th Nov in San Diego.

Monday, August 11, 2008

Looking for a star/ninja/guru developer....

I'm hiring, looking for someone who regularly gets a "wow" response from co-workers:

As a key engineer/ researcher for our web application development team, you will be working on data mining and algorithm development to build the world’s best movie finding software with some of the brightest people in the E-commerce arena. Our goal is to know our customers’ movie tastes well enough to create the perfect store, always presenting amazing titles just for them.

We are looking for a very experienced engineer who can take the lead, researching, modeling, presenting, designing and building algorithms that run in a service oriented architecture to support millions of customers. The successful candidate will be a self-motivated, intellectually curious individual. A Doctorate or experience working in a research environment, work with personalization algorithms, machine learning, adaptive systems, and statistical analysis would be a bonus. Experience with large scale multithreaded Java applications is essential.

This is a hands-on individual contributor role, for example you may need to prototype or model a new concept, communicate it, architect a Hadoop cluster, write the code to load and analyze interesting data and build high performance scalable Java and SQL code to incorporate the results into the web site.


Netflix is a performance culture. Everyone is located in Los Gatos CA, no teleworking, contractors or offshore development. Pay is at the top end of the scale for the SF Bay Area. Netflix is scaling the company by hiring the very best people in the industry. There are no interns or junior staff, minimal need for processes, we hire fully formed adults and give them freedom with responsibility. Netflix is very focused, small enough to be fun and to know everyone and everything that is going on, yet big enough to be interesting, with a big impact in a very entertaining industry.

Tuesday, August 05, 2008

CMG08 Draft Agenda - Las Vegas in December

CMG just published the draft agenda for their annual conference on their website. This year I will be on a panel with other Michelson Award winners discussing Visualization, and probably on a second panel talking about Unix performance. I'm also planning to repeat the half day training seminars on Unix Performance, and Capacity Planning with Free Tools.

See you there!

Monday, July 28, 2008

Skype Downloads Approach One Billion

About eighteen months ago I wrote a blog post about Skype as it approached 500 million downloads. They have continued at around a million a day, and now, nearly 500 days later we are coming up on the one billion mark. I predict at the current rate that the last 42 million downloads will happen in 37 days, so it will pass a billion around September 3rd.

The usual caveats apply, most of the downloads are version upgrades, but there are over 300 million different registered skype names, and concurrent user counts peak at 12 million most days.

At the time I got a nice mention from Skype Journal, and you can find good analysis of the Skype usage levels at Jean Mercier's Skype Numerology blog. As ever, you can track the actual numbers via an RSS feed that Skype provides, or just view them at this graph.

Why does this matter? It matters because this is the kind of critical mass that the rest of the Telco industry and VOIP startups are only dreaming of. I'm about to turn off my long distance and international phone plans and just have local calls on my home phone. The home phone I just ordered will redirect long distance and international calls over Skype, and make local calls on the land line. This will save me hundreds of dollars a year in telco service costs, so it pays for itself in less than six months. There is even a special offer on the hardware right now. I fully expect that in a few years time I will also be making the bulk of my mobile calls over Skype as well.

All I want from the Telco's is reliable unlimited raw bandwidth at a fixed low price, which is a terrifying prospect for the dinosaurs of this industry...

Monday, July 14, 2008

Enhanced headroom plot in R

For some reason I seem to find time to write code in R when I'm on an airplane. The last two trips I made resulted in significant enhancements and debugging of the code for my headroom plot. It started off simple but it now has a lot of bells and whistles, including color coding.

Main changes: the quantile used to remove outliers now only removes outliers that exceed the 95th percentile response time by default. It keeps all the throughput values unless you use qx=True.

In each of the throughput bins used to draw the histogram, the maximum response time for that bin is now calculated and displayed as a staircase line unless you set max=False.

The set of data is now split into ranges and color coded. The times series plot is coded so you can see the split, and the scatterplot shows how those points fall. I have been plotting weekly data at one minute intervals with split=7, which looks pretty good.

I read in some data that has been extracted from vxstat into a csv format at a known URL and plotted it three ways.
I plot the first 2880 data points, picking the read data rather than write, two days at one minute intervals.


> stime <- read.csv(url("http://somewhere/vxstat_response"))
> sops <- read.csv(url("http://somewhere/vxstat_throughput"))

> names(sops)
[1] "DateTime" "vxstat_dg_operationsRead"
[3] "vxstat_dg_operationsWrite"

> chp(sops[1:2880,2],stime[1:2880,2])
> chp(sops[1:2880,2],stime[1:2880,2],q=1.0)
> chp(sops[1:2880,2],stime[1:2880,2],q=1.0,splits=8)










Here is the code that generates the plot.


> chp <-function(throughput,response, q=0.95, qx=F, xl="Throughput",yl="Response",tl="Throughput Over Time",
ml="Headroom Plot", fit=T, max=T, splits=0) {
# remove zero throughput and response values
nonzer <- (throughput != 0) & (response != 0) # array of true/false
y <- response[nonzer]
x <- throughput[nonzer]
# remove outliers, keep response time points inside 95% by default
if (q != 1.0) {
quant <- (y < quantile(y,q))
# optionally trim throughput outliers as well
if (qx) quant <- quant & (x < quantile(x, q))
x <- x[quant]
y <- y[quant]
}
# make histograms and record end points for scaling
xhist <- hist(x,plot=FALSE)
yhist <- hist(y,plot=FALSE)
xbf <- xhist$breaks[1] # first
ybf <- yhist$breaks[1] # first
xbl <- xhist$breaks[length(xhist$breaks)] # last
ybl <- yhist$breaks[length(yhist$breaks)] # last
xcl <- length(xhist$counts) # count length
ycl <- length(yhist$counts) # count length
xrange <- c(0.0,xbl)
yrange <- c(0.0,ybl)
xlen <- length(x)
# make a multi-region layout
nf <- layout(matrix(c(1,3,4,2),2,2,byrow=TRUE), c(3,1), c(1,3), TRUE)
layout.show(nf)
# set plot margins for throughput histogram and plot it
par(mar=c(0,4,3,0))
barplot(xhist$counts, axes=FALSE,
xlim=c(xcl*0.00-xbf/((xbl-xbf)/(xcl-0.5)),xcl*1.00),
ylim=c(0, max(xhist$counts)), space=0, main=ml)
# set plot margins for response histogram and plot it sideways
par(mar=c(5,0,0,1))
barplot(yhist$counts, axes=FALSE, xlim=c(0,max(yhist$counts)),
ylim=c(ycl*0.00-ybf/((ybl-ybf)/(ycl-0.5)),ycl*1.00),
space=0, horiz=TRUE)
# set plot margins for time series plot
par(mar=c(2.5,1.7,3,1))
plot(x, main=tl, cex.axis=0.8, cex.main=0.8, type="S")
if (splits > 0) {
step <- xlen/splits
for(n in 0:(splits-1)) {
lines((1+n*step):min((n+1)*step,xlen), x[(1+n*step):min((n+1)*step,xlen)], col=4+n)
}
}
# set plot margins for main plot area
par(mar=c(5,4,0,0))
plot(x, y, xlim=xrange, ylim=yrange, xlab=xl, ylab=yl, pch=20)
if (max) {
# max curve
b <- xhist$breaks
i <- b[2] - b[1] # interval
maxl <- list(y[b[1] < x & x <= (b[1]+i)])
for(n in b[c(-1,-length(b))]) maxl <- c(maxl,list(y[n < x & x <= (n+i)]))
#print(maxl)
maxv <- unlist(lapply(maxl,max)) # apply max function to elements of list
#print(maxv)
#lines(xhist$mids,maxv,col=2) # join the dots
#staircase plot showing the range for each max response
lines(rep(b,1,each=2)[2:(2*length(maxv)+1)],rep(maxv,1,each=2),col=3)

}
if (fit) {
# fit curve, weighted to predict high throughput
# create persistent chpfit object using <<-
chpfit <- glm(y ~ x, inverse.gaussian, weights=as.numeric(x))
# add fitted values to plot, sorted by throughput
lines(x[order(x)],chpfit$fitted.values[order(x)],col=2)
}
if (splits > 0) {
step <- xlen/splits
for(n in 0:(splits-1)) {
Sys.sleep(1)
points(x[(1+n*step):min((n+1)*step,xlen)],y[(1+n*step):min((n+1)*step,xlen)], xlim=xrange, ylim=yrange, col=4+n)
}
}
}

iPhone 2.0 upgrade and applications - Pandora

I waited until Saturday evening to attempt an upgrade on my iPhone, and it completed with no problems, so I guess Apple fixed their site and the rush had died down a bit.

I looked through some of the applications and loaded Pandora, eBay, Paypal (for old times sake), Weatherbug, Remote (I also upgraded my AppleTV so this will come in useful) and Facebook. Enough to get me started. I also saw a nice star mapping application, I used to have one of these on my Palm/Treo but it wasn't free, so I decided to leave it for later.

The Pandora application is my favorite. I am an intermittent user of Pandora, as I didn't like being tethered to my laptop to listen. However the iPhone version works fine even over a fairly poor EDGE signal. There are some pauses between songs and the occasional pause during a song, but overall its a nice way to listen to the channels I had already setup, and discover new music that is similar to my favorites. I suspect that this is going to get a lot of usage since I can listen to it on the go.

There are over 500 applications, so plenty of stuff to play with if I get bored.

I applied for iPhone developer program membership when it was announced, and got an email yesterday saying that I have been accepted, so went through the purchase process, and downloaded the final version of the iPhone SDK....

Tuesday, June 10, 2008

Variable Power and Performance in Enterprise Servers

Nice Infoworld article looking at the power savings from variable speed CPUs in Enterprise servers running Windows and Linux.

This feature messes with capacity planning tools and the calculation of headroom, as I described in past CMG papers (Utilization is Useless as a Metric).

Friday, May 16, 2008

Mobile disruptions and going mainstream

VC thoughts via Peter Cranstone

From Vinod Knosla: The mobile phone will be a mainstream personal computer.

From Roger McNamee: Betting on smart phones: The mobile device migration to smart phones from features phones will produce even greater disruption than PC industry moving from character mode to graphical interface.

Pretty much what I've been saying for the last year or two...

Friday, May 09, 2008

Speaking at UK CMG TEC Conference - May 19-21st 2008

UKCMG TEC 2008 is near Northampton in the UK. I'm giving two of the same presentations as I gave at the US CMG, the enterprise version of my Millicomputing talk, and a half day workshop on Unix/Linux Performance.

My old friend Phil Harman is also giving a talk on DTrace, and I'm staying in the UK for a few days after the conference to catch up with friends and family.

Saturday, May 03, 2008

Usenix 08 - Tutorials and Invited Paper - Boston June 23-27

I'm going to USENIX '08

I'm giving the two tutorials that I have been presenting at CMG the last few years, but at Usenix in Boston. The tutorials are the latest update of my Solaris/Linux performance metrics and tuning class, and Capacity Planning with Free Tools.

I'm also giving an invited speaker talk on Millicomputing, and have a nice big time slot, so will cover both the mobile and enterprise aspects in some depth, with time for discussion.

I hope to see you there...

Monday, April 28, 2008

XE Toolkit 1.2 - new portable performance tools from Rich Pettit

Rich just updated the portable Java based XE tookit, it even runs on mainframes now :-)

Rich (with some contributions from me) wrote many versions of the SE toolkit for Solaris. This is his latest project, please try it out, give him feedback and buy a support contract to encourage him...

Cheers Adrian

Subject: [Xetoolkit-interest] XE Version 1.2 Available
To: xetoolkit-interest@xetoolkit.com


XE Version 1.2 is available in pre-packaged form through the Captive
Metrics online store, http://store.captivemetrics.com.

New platforms for 1.2:
AIX (5.3 base)
Linux on Power (IBM pSeries & iSeries)
Linux on s390 (IBM zSeries)

Please report bugs to support@captivemetrics.com.

Changes since Version 1.1

Bug fixes
Solaris locking problems
Performance improvements in mdr
Locale problems in non-US-English Windows installs
Support for AIX (5.3 base)
Support for Linux on Power (IBM pSeries & iSeries)
Support for Linux on s390 (IBM zSeries)
Code restructuring and use of new "snap" utility
and command-line build for Windows
VirtualGuru rotates log files when the day changes
bbClient groovy script for monitoring BigBrother clients

The source code is available at
http://sourceforge.net/projects/xe-toolkit as an SVN tree.
There will be a web site with "tarballs" eventually. It's not
high on the priority list.

The build process uses "snap", a /bin/sh utility written by me that
generates Makefiles on the fly.

Please support future development by purchasing the package.

Thank you for your continued interest.

Rich
--
Richard Pettit
Captive Metrics Software Corporation
Em: richp@captivemetrics.com
Ph: 8-MONITOR-01

Thursday, March 27, 2008

Patent on Performance and Capacity Planning

Here is another patent that I filed while working at Sun.

System and method for generating a probability distribution of computer performance ratios

A system and method for generating a probability distribution indicating the probability of a range of performance ratios between computer systems. Benchmark and/or other testing results are obtained for both computer systems. For each test, a ratio of one system's performance to the other is calculated. The ratios and/or testing results may be weighted. From the performance ratios, a histogram is produced to indicate the probabilities of achieving the various ratios. Based on the distribution of probabilities, a particular performance ratio can be predicted with a corresponding level of confidence, and used for capacity planning or configuring a computing environment.

Patents on Peer to Peer Trading Platform

While working at eBay Research Labs I did some work using Skype that resulted in four patents in the area of P2P trading and reputation systems. I just noticed that they are now public, as of late 2007, and here they are:

Peer-to-peer trading platform

Peer-to-peer trading platform with relative reputation-based item search and buddy rating

Peer-to-peer trading platform with search caching

Peer-to-peer trading platform with roles-based transactions

The seed of the idea came from my good friend Josep Ferrandiz, who still works at PayPal. The ideas were developed with help from Zahid Ahmed who is now working for EMC, and with a lot of help from Neel Sundaresan who manages eBay Research labs. The patents are in all of our names.

I think we came up with some cool ideas, we also stress tested the Skype API's in some novel ways and got some things fixed. The lasting legacy of this work is more robust application to application messaging API's and improvements in Skype4Java.

Sunday, March 16, 2008

Monday, March 03, 2008

BIL - recording an excellent adventure

The closing slide of BIL by Garrett Lisi



Two days of intense expansion of my horizons, a self-organizing conference, and I am left holding a jewel. My contribution to BIL was that I basically did tech-support and audio for all the speakers in the main hall for two days. I brought my audio recorder a good mic and stand, a guitar amp (that was used as a PA speaker) and bags of cables and adaptors. Other BILsters provided a video projector and a karaoke machine for PA. I hooked up the mic and a mini-jack cable for computer audio out, and kept it all running as best I could. I recorded almost 800 minutes of content on my Korg D16 hard disk recorder, which is almost all of every talk. There were other people recording on Saturday, including video, but they had technical problems on Sunday, so we have little or no video of the Sunday talks.

Next I'm going to make a backup copy, then start working on extracting the raw data for other people to process. I'll make mp3's of my favorite talks as I get time. Several TED speakers talked at BIL, including Aubrey De Grey on Saturday and Garrett Lisi, who closed out the conference with a fantastic talk. I chatted to Garrett a bit, very cool...

For the stream of consciousness with pictures that I uploaded during the talk, look at my twitxr page. I will update my twitter account with progress on the audio files. I don't expect to use twitter and twitxr on a daily basis, but I think they are useful for conferences. I'll track progress at Ecomm08 on March 12-14th the same way.

Friday, February 29, 2008

Millicomputing at BIL this weekend

I'm presenting (and generally helping out) at the BIL un-conference this weekend in Monterey. I have posted some slides on my Millicomputing blog, on the subject of "The Future in Your Pocket", that speculates on what could happen over the next few years in the mobile space, and starts a discussion of what we might be able to use it for.

Thursday, February 28, 2008

Hiring a researcher at Netflix

Here is the job specification for a research oriented position working for me at Netflix.

We are looking for a very experienced engineer who can take the lead, researching, modeling, presenting, designing and building algorithms that run in a service oriented architecture to support millions of customers. The successful candidate will be a self-motivated, intellectually curious individual. A Doctorate or experience working in a research environment, work with personalization algorithms, machine learning, adaptive systems, and statistical analysis would be a bonus.

I manage a sister group to the one that does the Cinematch star ratings predictions and hosts the Netflix Prize, we use that data as an input along with everything else we know about movies and customers to come up with the actual lists of movies that are shown all over the site. The research position in my group works on algorithms and mining for new data sources.

Also, Netflix is a great place to work, big enough to be interesting and small enough to be fun.

Tuesday, February 12, 2008

Emerging Communications Conference - EComm March 12-14

I'm presenting at the EComm conference on the subject of Millicomputing on the morning of March 14th. There is a Homebrew Mobile Club meeting at the conference (at the Computer History Museum in Mountainview) the evening of March 12th.

Tuesday, January 15, 2008

MacWorld report: iZap, Apple MacBook Air, iPhone and AppleTV

The picture shows an electric car called the Zap, which someone appears to have decided that it needed a few upgrades to become the iZap. Jet motor, rotor blades and iPod interfaces....

I tracked the MacWorld keynote via the Macrumors site on my iPhone this morning. It worked very well, with updates being pushed every few minutes. Over-all there was some good stuff released, but I think expectations were higher.

Later in the afternoon I dropped in on MacWorld for an hour or so, which was plenty of time to check out the interesting stuff.

The MacBook Air is another hit product, destined to become a geek status symbol. It looks good, feels good to use, and has the option of a solid state disk. I had some hands-on time, and I want one.... They are expensive, but the format is very well designed, and I expect it will come down in price and get new options over the coming years. To make the case strong enough despite being thin, there are more curves in the case (curved metal is much stronger than flat sheet) and it has no large holes in it. So there is no removable battery pack, and each key has its own small hole in the casing. It actually looks more mechanically robust than my MacBook pro.

I hoped to find some interesting third party iPhone applications but didn't see anything worth mentioning. I played with a demo iPhone with 1.1.3, it had a bunch of safari based games bookmarked on the second page of icons. Once I got home I upgraded my iPhone, its running fine and the new features work well. They were well leaked a few weeks ago, so there is nothing new to report. The iPod touch picked up the missing set of applications as I expected, but Apple want $20 for the privilege. This is the first example of optional applications, and charging for applications, and I expect the mechanism will get well tested by the Touch then it will be introduced for the iPhone and third party applications.

I have had an AppleTV for six months or so, and the upgraded software looks much better. I don't think I will be renting or buying many movies, but the user interface is better, and the ability to browse both audio and video based podcasts could be interesting. I mostly use my AppleTV for YouTube, playing music and sharing photo albums. One new feature is the ability to pull up photo albums over the Internet from the .mac galleries generated by iPhoto. I've been using .mac for years, and the latest (Leopard) version of iPhoto galleries is very slick and well integrated.

Apple also introduced Time Capsule, which is a network backup server combining a 500GB or 1TB of disk with an Airport. The hardware was not on show as far as I could tell, and it looks similar to the AppleTV package, but with more Ethernet ports and no video port. Personally, I don't want two separate products in this space, I want an AppleTV that operates as a Time Capsule and Airport. I want to backup my laptop and be able to play the music and video directly from the backup copy.

I'm disappointed that there was no new iPhone hardware or Leopard upgrade, and no business oriented iPhone software. I guess that's going to be featured in the next show....

Monday, December 31, 2007

Some New Performance Monitoring Tools

There is a simple and extensible open source C based daemon called collectd that writes to RRD files, an alternative to Orca/procallator for people who don't want the Perl based memory footprint of procallator. I'll check it out on my Gumstix millicomputer sometime.

There is yet another open source full-function monitoring tool called Zabbix that looks similar to Cacti in scope, possibly with more features, and with a SQL database backend. It has a commercial company backing it with support contracts etc, somewhat like the XE Toolkit.

The most interesting commercial tool I saw at CMG earlier this month is a capacity monitoring tool called PAWZ from Perfcap Corporation. The key thing they have worked on is taking the human out of the loop as much as possible with sophisticated capacity modelling algorithms and a simple and scalable operational model. It is very similar in concept to the capacity planning research I was working on and publishing in 2002-2004. The core idea is that you care about "headroom" in a service, and anything that limits that headroom is taken into account. Running out of CPU power, network bandwidth, memory, threads etc. will increase response time of the service, so monitor them all, track trends in headroom and calculate the point in time where lack of headroom will impact service response time. At eBay we used to call this the "time to live" for a service. You can easily focus on the services that have the shortest time to live, and proactively make sure that you have a low probability of poor response time. I'm going to take a closer look at this one...




Sunday, December 30, 2007

Thoughts on iPhone 1.1.3, Macworld, 3G and business users

Reports indicate the the 1.1.3 update is likely to ship before MacWorld in January, it includes an update to Google Maps that adds the same location feature as recently shipped on other non-GPS platforms (basically locating to the nearest cell-tower). It also provides mechanisms to select and arrange the applications installed on the iPhone. This is a key new feature, since the iPhone has been updated by Apple as a monolithic all-or-nothing set of applications so far. It greatly reduces the need to hack into the iPhone to customize it.

The updates themselves are fairly small, and to me it makes sense to get them installed before MacWorld. We will then see announcements at MacWorld of a set of optional applications that users can pick and choose to install on their iPhones. This could include the ability to add the Mail application and other "missing" iPhone apps to the iPod Touch. It could include brand new applications (I'm going to nominate iChat as a candidate yet again...). The public announcement of the SDK is due at MacWorld, with shipping in February, which will open up anyone to build officially sanctioned applications. However key vendors will have been testing the SDK over the last few months so I expect a bunch of third party applications to be announced or ready to ship at MacWorld.

The other leaks and rumors indicate that there is likely to be a second generation iPhone with 3G support shipping in the spring, and announced at MacWorld. This would also support launching the iPhone in Asian markets like Japan, where there is no GSM support.

I also expect that Apple will start to make moves towards business use of the iPhone, with some tools and upgrades provided by Apple, and others by key third parties.

I currently carry a Verizon Blackberry 8703e for work use, and my iPhone for personal use and iPod functionality. In order to use the iPhone as my work phone I need a few key features.

  • Firewall support - the BB is inside the corporate firewall, the iPhone can't access it. We use Juniper Network Connect which is a Java based VPN solution on MacOS/XP.
  • WiFi support - we use LEAP to login to WiFi at work, need support for LEAP on iPhone, it works fine on my MacOS X laptop, should be a simple feature to add.
  • Exchange support - I can't use the IMAP workaround due to firewall issues, properly integrated Exchange email and calendar support is what everyone is asking for.
  • Ideally RIM will port the Blackberry application suite to the iPhone, like they did for the Treo...
The two other biggest missing features are Flash and Java support. I know there are lots of issues with CPU/memory/battery life. Perhaps the next generation iPhone will be based on a more advanced ARM CPU (e.g. the ARM Cortex based Qualcomm Scorpion) with more performance and more memory so it can run Flash and Java apps alongside the existing apps?

We'll find out in a few weeks...
Happy new year.

Friday, December 28, 2007

MacOS X Leopard, iPhone and Stereo Bluetooth Headphones - A2DP

Over a year ago I bought stereo bluetooth headphones on eBay, its a multifunction unit the OMIZ OMS600, and it includes an MP3 player (with MicroSD slot), FM radio, Stereo A2DP headphone and Headset with Mono audio/Microphone. When I tried to configure it on MacOS X Tiger it didn't work because there was no A2DP headphone support.

MacOS X Leopard now supports A2DP and "just works" with this headset. The Mac sees both the Headset (Mono audio/mic for Skype etc) and Headphone (A2DP Stereo) as separate devices. After the usual Bluetooth device wizard setup, simply put the OMS600 into the headphone mode and pick "Use Headphone" from the Bluetooth dropdown menu on the Mac, the Mac's internal speakers mute, and the Headphones play. It worked over a 10 foot range walking around a room, crackled a bit at the limit and dropped the connection if I went too far away.

I don't see the OMS600 for sale any more, but there are plenty of A2DP headphones out there now, and its nice to see that Apple finally got the devices to work, and made it "Just Work" as usual.

The headset mode also works on my iPhone, but the A2DP mode doesn't. I paired the headset with the iPhone and was able to make calls and receive them. When using the FM radio or the MP3 player built-into the headset it paused and resumed for incoming calls. However when the Headphone mode was paired with my Leopard machine the iPhone didn't route calls to the Headset. The OMS600 headset has the microphone built into the left side earpiece with no voice tube or boom down nearer my mouth, and it doesn't pick up very clearly. The noise cancelling Jawbone headset works far better.

I listened to some music on iTunes (Radiohead In Rainbows and Gorillaz D-Sides are my current albums for serial listening) and waited forever for XP to start up in Parallels so I could fire up IE7 and see a Netflix Watch Instantly show about the Pixies reunion tour (called LoudQuietLoud). [Yes I know it would be nice if it worked natively on the Mac, but the studios only approve Windows DRM, and the alternatives all have issues that are taking way too much time to sort out].

Happy new year...

Sunday, December 09, 2007

A. A. Michelson Award Acceptance Speech

Last week I was selected as this year's winner of the A. A. Michelson Award for Lifetime Achievement and Contribution by the Computer Measurement Group. Here is a transcript of my acceptance speech.


Thanks to CMG, the Michelson Award winners who voted to add me to their ranks, and everyone who I have met here since I first attended CMG in 1994. I treat the CMG conference as my annual training class, a chance to mingle with my peers, learn and share the things I have learned.


There are some people who I have met at CMG or while working in this field who have become special friends, not just people I see at the conference each year. Cathy Nolan is not only national CMG president this year, she is president of my local CMG group in Northern California and was also instrumental in the process of getting me nominated for this award. We have all learned a lot over the years from Neil Gunther. Neil and I have jointly presented many training classes so I have had plenty of opportunity to learn his material. He has made queuing theory accessible and useful for very many people. Mario Jauvin has been my CMG conference buddy for many years, and for the last three years we have jointly presented the Capacity Planning with Free Tools workshop. Yefim (Fima) Somin encouraged me to attend CMG in the first place, we met in 1994 while working to port BGS Best/1 to Solaris, and we have kept in touch over the years, even though he doesn't get to come to CMG nowadays. Fima and Henry Newman were part of the Universal Measurement Architecture standards body that I joined in 1995, and I have also kept in touch with Henry. We worked to get some of his ideas (like extended system accounting) to be implemented in Solaris. Bob Sneed worked with me at Sun and is now the main contact point and conscience for CMG at Sun.

Outside CMG there are some very significant people I would also like to thank. Brian Wong is one of my closest friends, responsible for getting me to move to the USA in 1993, and is also author of the Capacity Planning and Configuration for Solaris Servers book. Brian introduced me to Rich Pettit, and we worked together for many years to build and extend the SE Toolkit, the vehicle for most of my performance measurement ideas. Allan Packer was the co-author of my first CMG paper on Database Sizing. He knows everything about database performance and wrote the book Configuring and Tuning Databases on the Solaris Platform. We borrowed Allan from Sun's office in Adelaide Australia for a month or two, and he introduced me to Richard McDougall, who we also borrowed from Adelaide. Allan and Richard both ended up working for Sun's Performance group in California. Richard built the first tools that could measure memory usage, fixed the Solaris 8 memory system, and co-authored the Solaris Internals books. He became a Sun Distinguished Engineer and recently joined VMware as Chief Performance Architect. Jim Mauro is the other author of Solaris Internals, we borrowed him from Sun's New Jersey office a few times before he also joined the performance group full time. Finally Dave Fisk worked with Brian Wong and I at Sun, and became one of my closest friends. He knows more about Unix storage performance than anyone else I've met, and I've called on him many times (he's now a consultant) to help figure out what's really going on.

I would also like to thank my family. My father is a retired Statistics Professor from the University of Hertfordshire (a.k.a. Hatfield Polytechnic) in the UK. He was programming himself in the 1960's and gently encouraged me to tinker with computers from an early age. My son recently graduated with his BSc in Computer Science from the same University, and now works as a project manager at eBay in California. My daughter has just started a degree course in European Literature at Royal Holloway College, London, so I'm no help at all with her assignments :-)  I'm very proud of them. I moved to the USA on my own a few years ago and met my lovely wife Laurel in 2003. I'm very happy that she has also been able to attend CMG with me and meet many of you.


I have always been glad that I studied Physics, the work we do in performance and the diagnosis and measurement of performance leans heavily on my training as a Physicist and I think that pure Computer Scientists are lacking some of the tools they need to understand measurement and performance issues. Since A. A. Michelson was a Physicist this award is especially gratifying for me.


In 1972 my High School had a link to the computer that my Father used at Hatfield Polytechnic. I taught myself BASIC and Algol during recess....


This is what most people know me for, its getting a little out of date now, since the second edition was published in 1998, but a big thank-you to my co-author Rich Pettit and everyone that bought a copy!


The book documented metrics and performance rules, and they were implemented as an SE Toolkit script called Virtual Adrian. I spent a lot of time with all the tools vendors helping them to manage Solaris better, and with Solaris engineering getting them to make Solaris more manageable. This is what justified my involvement with CMG during my time at Sun, and its the legacy that I am most proud of.


To finish, some of the things I have learned over the years that you may find useful...


If you ever try to write a book, you need to pay attention to Cockcroft's Law of Book Writing. It seems counter-intuitive to people who have never attempted to write a book, but it is the most important thing to know if you ever intend to finish a book. For example, you scope a 200 page book and start writing. After you have written 50 pages you re-scope and find your book is now a 300 page book. You write another 50 pages and find you are now writing a 400 page book. The number of pages you have left to write to finish the book is increasing! You have to reduce scope continuously and plot a trend line that slopes downwards if you ever hope to be published.


If you collect information and share it freely, you will become an expert. You become a magnet for questions, issues and information. Some people tightly control their expertise, this is a huge mistake as it gives others incentives to look for alternative experts or to become experts themselves. If you give out your expertise freely, you become the go-to person in your field, and gather many more recommendations from the people you help. Try it!



Some observations that may help you deal with executive level management. The first one is a big problem that Sun and other manufacturers have, because the things they build take many years to go from conception to delivery. If the managers in charge of these projects don't stick around then the project is likely to be tinkered with and delayed further or dropped. Its nice being at Netflix, where the executives have been there for years and the projects take months to deliver. The second observation is that Capacity Planners have a vertical role in the organization, they gather the lowest level metrics from operations and have to present findings to upper management. We love our detailed metrics, but the only metric that upper management should have to deal with is the dollar. If you can present everything in terms of return on investment (spend $100K to save or grow by $1M) then your executive presentations will be much more effective.


What am I working on now? I'm managing a team that develops the Netflix web site. Its a personalized web service and I think that is the future of the computer industry. The scalability and performance challenges are very interesting. I'm also pushing an idea I call "Millicomputing" which could have a disruptive effect on the industry as storage moves from disks (spinning rust) to flash, and the computers move to ultra low power system-on-a-chip devices. I gave a paper on Millicomputing at CMG07 and have a separate blog on the subject.



My paper at last year's CMG was called "Utilization is Virtually Useless as a Metric", and this answer sums it up. Those who ask questions about utilization don't understand that their questions have no meaning so the answers are irrelevant :-)

Thanks again!

Adrian