Sunday, December 21, 2008

Capacity Planning for Cloud Computing

At the recent CMG conference I volunteered to lead a panel session on Capacity Planning for Cloud Computing, and these slides were the result.

Now, this was a conference full of people who already know how to do capacity planning, so they were interested in what was new about Cloud Computing (or even what it was all about from first principles), so the slides do not explain how to do capacity planning in the cloud, they talk about what changes for the capacity planner in this new world.

I'm going to be developing more material in this area over the coming year. I have a head start, in 2002 and 2003 I led projects at Sun where I researched how to do Capacity Planning in Virtualized Datacenters, built some tools, filed a couple of patents, presented papers at conferences and failed to get Sun's N1 project to implement some of it. Well, it seems I was a few years ahead of my time, so I'm going to start by digging out the papers I published at the time.

Solid State Disks - time to give up that iron oxide habit

An interesting note at talks about a 2009 technology that gives flash a million write cycles at the NAND cell level. This adds two orders of magnitude to the "typical" cycle lifetime that is often used as an argument that flash isn't ready for prime time.

In any case this applies to each block on the SSD, and blocks are remapped using write leveling, and writes are cancelled using RAM based SSD controllers, and when the limit is reached, only that block becomes unusable... Some people seem to think that as soon as you do 10,000 writes to an SSD it fails totally, like a head crash on a disk.

I also see that at least one vendor has announced a 512GB SSD in a 2.5" laptop disk format package.

So to reiterate something I've been saying for a long time, spinning rust is dead, and a large number of basic assumptions about how computers behave and the best way to architect them are now wrong. In 2009 SSD's will be faster for read, faster for write, faster for sequential and much much faster for random access, more reliable, more durable, lower power, higher capacity, than discs. Give it another year or so and they will be cheaper as well.

SAN's are now a complete waste of time. There is so much reliable I/O performance available in a single drive, that it makes much more sense to put SSD's in the systems and access them directly. Accessing an SSD over a SAN adds a huge latency and cost overhead. It makes much more sense to use node-to-node replication for critical data (log scraping databases or cluster filesystems).

Monday, November 17, 2008

iPhone Instant Queue Add for Netflix - get it now...

I've released a second version of my very own iPhone app. I also reduced the price from 99c to FREE for the next few days. Please check it out and write a review for me on the app store. More info at

Wednesday, October 15, 2008

Music - Amanda Palmer and Estradasphere

I listen to a lot of music, and the really good stuff gets played over and over again while I'm driving. I also think while I'm driving that I should write a blog post about my favorites but then I forget... until now!

My favorite new music for 2007 was "Palace of Mirrors" by Estradasphere. I found that I never tired of it, and listened to it for months on end. It helps that their music is extremely varied in style and impossible to categorize. You can find out more and listen to music at their web site. I suggest you start with "Smuggled Mutation", there is no such thing as a typical Estradasphere song as they combine just about every musical genre, usually in the same song. However this track shows off their talents very well. There is a live DVD of Palace of Mirrors which is also extremely well done. I have seen them play live several times and they are an excellent live band.

During 2008 Estradasphere collaborated with Amanda Palmer of the Dresden Dolls, and her new album "Who killed Amanda Palmer" has become my favorite pick for 2008. The web site is very interesting, giving a lot of background story details about the making of the album and videos of most of the tracks. The music is distinctively Amanda, but expanded well beyond the Dresden Dolls format, with a much fuller sound on some tracks. The key for me is that the music is very moving and emotional, and can sustain interest for a lot of repeated listening.

I'm also still waiting for Fractal to finish their second album "Sequitur", I've been listening to pre-release versions as they work towards the final cut. I'm not holding my breath, as its taken years to get this far, but its getting very close to being completed.

Monday, September 29, 2008

Gene Saunders RIP

A good friend and well respected colleague from Sun is no more...
More info on Geoff Arnold's blog

For the last few years a group of us have been keeping in touch via email, sharing thoughts and amusing web links. Gene was a key part of that conversation, and I miss him already.

Tuesday, September 23, 2008

Tutorial at LISA08 San Diego Nov 9th

I'm going to LISA '08

I'm giving my usual performance tools and tuning for Solaris/Linux class on Sunday 9th Nov in San Diego.

Monday, August 11, 2008

Looking for a star/ninja/guru developer....

I'm hiring, looking for someone who regularly gets a "wow" response from co-workers:

As a key engineer/ researcher for our web application development team, you will be working on data mining and algorithm development to build the world’s best movie finding software with some of the brightest people in the E-commerce arena. Our goal is to know our customers’ movie tastes well enough to create the perfect store, always presenting amazing titles just for them.

We are looking for a very experienced engineer who can take the lead, researching, modeling, presenting, designing and building algorithms that run in a service oriented architecture to support millions of customers. The successful candidate will be a self-motivated, intellectually curious individual. A Doctorate or experience working in a research environment, work with personalization algorithms, machine learning, adaptive systems, and statistical analysis would be a bonus. Experience with large scale multithreaded Java applications is essential.

This is a hands-on individual contributor role, for example you may need to prototype or model a new concept, communicate it, architect a Hadoop cluster, write the code to load and analyze interesting data and build high performance scalable Java and SQL code to incorporate the results into the web site.

Netflix is a performance culture. Everyone is located in Los Gatos CA, no teleworking, contractors or offshore development. Pay is at the top end of the scale for the SF Bay Area. Netflix is scaling the company by hiring the very best people in the industry. There are no interns or junior staff, minimal need for processes, we hire fully formed adults and give them freedom with responsibility. Netflix is very focused, small enough to be fun and to know everyone and everything that is going on, yet big enough to be interesting, with a big impact in a very entertaining industry.

Tuesday, August 05, 2008

CMG08 Draft Agenda - Las Vegas in December

CMG just published the draft agenda for their annual conference on their website. This year I will be on a panel with other Michelson Award winners discussing Visualization, and probably on a second panel talking about Unix performance. I'm also planning to repeat the half day training seminars on Unix Performance, and Capacity Planning with Free Tools.

See you there!

Monday, July 28, 2008

Skype Downloads Approach One Billion

About eighteen months ago I wrote a blog post about Skype as it approached 500 million downloads. They have continued at around a million a day, and now, nearly 500 days later we are coming up on the one billion mark. I predict at the current rate that the last 42 million downloads will happen in 37 days, so it will pass a billion around September 3rd.

The usual caveats apply, most of the downloads are version upgrades, but there are over 300 million different registered skype names, and concurrent user counts peak at 12 million most days.

At the time I got a nice mention from Skype Journal, and you can find good analysis of the Skype usage levels at Jean Mercier's Skype Numerology blog. As ever, you can track the actual numbers via an RSS feed that Skype provides, or just view them at this graph.

Why does this matter? It matters because this is the kind of critical mass that the rest of the Telco industry and VOIP startups are only dreaming of. I'm about to turn off my long distance and international phone plans and just have local calls on my home phone. The home phone I just ordered will redirect long distance and international calls over Skype, and make local calls on the land line. This will save me hundreds of dollars a year in telco service costs, so it pays for itself in less than six months. There is even a special offer on the hardware right now. I fully expect that in a few years time I will also be making the bulk of my mobile calls over Skype as well.

All I want from the Telco's is reliable unlimited raw bandwidth at a fixed low price, which is a terrifying prospect for the dinosaurs of this industry...

Monday, July 14, 2008

Enhanced headroom plot in R

For some reason I seem to find time to write code in R when I'm on an airplane. The last two trips I made resulted in significant enhancements and debugging of the code for my headroom plot. It started off simple but it now has a lot of bells and whistles, including color coding.

Main changes: the quantile used to remove outliers now only removes outliers that exceed the 95th percentile response time by default. It keeps all the throughput values unless you use qx=True.

In each of the throughput bins used to draw the histogram, the maximum response time for that bin is now calculated and displayed as a staircase line unless you set max=False.

The set of data is now split into ranges and color coded. The times series plot is coded so you can see the split, and the scatterplot shows how those points fall. I have been plotting weekly data at one minute intervals with split=7, which looks pretty good.

I read in some data that has been extracted from vxstat into a csv format at a known URL and plotted it three ways.
I plot the first 2880 data points, picking the read data rather than write, two days at one minute intervals.

> stime <- read.csv(url("http://somewhere/vxstat_response"))
> sops <- read.csv(url("http://somewhere/vxstat_throughput"))

> names(sops)
[1] "DateTime" "vxstat_dg_operationsRead"
[3] "vxstat_dg_operationsWrite"

> chp(sops[1:2880,2],stime[1:2880,2])
> chp(sops[1:2880,2],stime[1:2880,2],q=1.0)
> chp(sops[1:2880,2],stime[1:2880,2],q=1.0,splits=8)

Here is the code that generates the plot.

> chp <-function(throughput,response, q=0.95, qx=F, xl="Throughput",yl="Response",tl="Throughput Over Time",
ml="Headroom Plot", fit=T, max=T, splits=0) {
# remove zero throughput and response values
nonzer <- (throughput != 0) & (response != 0) # array of true/false
y <- response[nonzer]
x <- throughput[nonzer]
# remove outliers, keep response time points inside 95% by default
if (q != 1.0) {
quant <- (y < quantile(y,q))
# optionally trim throughput outliers as well
if (qx) quant <- quant & (x < quantile(x, q))
x <- x[quant]
y <- y[quant]
# make histograms and record end points for scaling
xhist <- hist(x,plot=FALSE)
yhist <- hist(y,plot=FALSE)
xbf <- xhist$breaks[1] # first
ybf <- yhist$breaks[1] # first
xbl <- xhist$breaks[length(xhist$breaks)] # last
ybl <- yhist$breaks[length(yhist$breaks)] # last
xcl <- length(xhist$counts) # count length
ycl <- length(yhist$counts) # count length
xrange <- c(0.0,xbl)
yrange <- c(0.0,ybl)
xlen <- length(x)
# make a multi-region layout
nf <- layout(matrix(c(1,3,4,2),2,2,byrow=TRUE), c(3,1), c(1,3), TRUE)
# set plot margins for throughput histogram and plot it
barplot(xhist$counts, axes=FALSE,
ylim=c(0, max(xhist$counts)), space=0, main=ml)
# set plot margins for response histogram and plot it sideways
barplot(yhist$counts, axes=FALSE, xlim=c(0,max(yhist$counts)),
space=0, horiz=TRUE)
# set plot margins for time series plot
plot(x, main=tl, cex.axis=0.8, cex.main=0.8, type="S")
if (splits > 0) {
step <- xlen/splits
for(n in 0:(splits-1)) {
lines((1+n*step):min((n+1)*step,xlen), x[(1+n*step):min((n+1)*step,xlen)], col=4+n)
# set plot margins for main plot area
plot(x, y, xlim=xrange, ylim=yrange, xlab=xl, ylab=yl, pch=20)
if (max) {
# max curve
b <- xhist$breaks
i <- b[2] - b[1] # interval
maxl <- list(y[b[1] < x & x <= (b[1]+i)])
for(n in b[c(-1,-length(b))]) maxl <- c(maxl,list(y[n < x & x <= (n+i)]))
maxv <- unlist(lapply(maxl,max)) # apply max function to elements of list
#lines(xhist$mids,maxv,col=2) # join the dots
#staircase plot showing the range for each max response

if (fit) {
# fit curve, weighted to predict high throughput
# create persistent chpfit object using <<-
chpfit <- glm(y ~ x, inverse.gaussian, weights=as.numeric(x))
# add fitted values to plot, sorted by throughput
if (splits > 0) {
step <- xlen/splits
for(n in 0:(splits-1)) {
points(x[(1+n*step):min((n+1)*step,xlen)],y[(1+n*step):min((n+1)*step,xlen)], xlim=xrange, ylim=yrange, col=4+n)

iPhone 2.0 upgrade and applications - Pandora

I waited until Saturday evening to attempt an upgrade on my iPhone, and it completed with no problems, so I guess Apple fixed their site and the rush had died down a bit.

I looked through some of the applications and loaded Pandora, eBay, Paypal (for old times sake), Weatherbug, Remote (I also upgraded my AppleTV so this will come in useful) and Facebook. Enough to get me started. I also saw a nice star mapping application, I used to have one of these on my Palm/Treo but it wasn't free, so I decided to leave it for later.

The Pandora application is my favorite. I am an intermittent user of Pandora, as I didn't like being tethered to my laptop to listen. However the iPhone version works fine even over a fairly poor EDGE signal. There are some pauses between songs and the occasional pause during a song, but overall its a nice way to listen to the channels I had already setup, and discover new music that is similar to my favorites. I suspect that this is going to get a lot of usage since I can listen to it on the go.

There are over 500 applications, so plenty of stuff to play with if I get bored.

I applied for iPhone developer program membership when it was announced, and got an email yesterday saying that I have been accepted, so went through the purchase process, and downloaded the final version of the iPhone SDK....

Tuesday, June 10, 2008

Variable Power and Performance in Enterprise Servers

Nice Infoworld article looking at the power savings from variable speed CPUs in Enterprise servers running Windows and Linux.

This feature messes with capacity planning tools and the calculation of headroom, as I described in past CMG papers (Utilization is Useless as a Metric).

Friday, May 16, 2008

Mobile disruptions and going mainstream

VC thoughts via Peter Cranstone

From Vinod Knosla: The mobile phone will be a mainstream personal computer.

From Roger McNamee: Betting on smart phones: The mobile device migration to smart phones from features phones will produce even greater disruption than PC industry moving from character mode to graphical interface.

Pretty much what I've been saying for the last year or two...

Friday, May 09, 2008

Speaking at UK CMG TEC Conference - May 19-21st 2008

UKCMG TEC 2008 is near Northampton in the UK. I'm giving two of the same presentations as I gave at the US CMG, the enterprise version of my Millicomputing talk, and a half day workshop on Unix/Linux Performance.

My old friend Phil Harman is also giving a talk on DTrace, and I'm staying in the UK for a few days after the conference to catch up with friends and family.

Saturday, May 03, 2008

Usenix 08 - Tutorials and Invited Paper - Boston June 23-27

I'm going to USENIX '08

I'm giving the two tutorials that I have been presenting at CMG the last few years, but at Usenix in Boston. The tutorials are the latest update of my Solaris/Linux performance metrics and tuning class, and Capacity Planning with Free Tools.

I'm also giving an invited speaker talk on Millicomputing, and have a nice big time slot, so will cover both the mobile and enterprise aspects in some depth, with time for discussion.

I hope to see you there...

Monday, April 28, 2008

XE Toolkit 1.2 - new portable performance tools from Rich Pettit

Rich just updated the portable Java based XE tookit, it even runs on mainframes now :-)

Rich (with some contributions from me) wrote many versions of the SE toolkit for Solaris. This is his latest project, please try it out, give him feedback and buy a support contract to encourage him...

Cheers Adrian

Subject: [Xetoolkit-interest] XE Version 1.2 Available

XE Version 1.2 is available in pre-packaged form through the Captive
Metrics online store,

New platforms for 1.2:
AIX (5.3 base)
Linux on Power (IBM pSeries & iSeries)
Linux on s390 (IBM zSeries)

Please report bugs to

Changes since Version 1.1

Bug fixes
Solaris locking problems
Performance improvements in mdr
Locale problems in non-US-English Windows installs
Support for AIX (5.3 base)
Support for Linux on Power (IBM pSeries & iSeries)
Support for Linux on s390 (IBM zSeries)
Code restructuring and use of new "snap" utility
and command-line build for Windows
VirtualGuru rotates log files when the day changes
bbClient groovy script for monitoring BigBrother clients

The source code is available at as an SVN tree.
There will be a web site with "tarballs" eventually. It's not
high on the priority list.

The build process uses "snap", a /bin/sh utility written by me that
generates Makefiles on the fly.

Please support future development by purchasing the package.

Thank you for your continued interest.

Richard Pettit
Captive Metrics Software Corporation
Ph: 8-MONITOR-01

Thursday, March 27, 2008

Patent on Performance and Capacity Planning

Here is another patent that I filed while working at Sun.

System and method for generating a probability distribution of computer performance ratios

A system and method for generating a probability distribution indicating the probability of a range of performance ratios between computer systems. Benchmark and/or other testing results are obtained for both computer systems. For each test, a ratio of one system's performance to the other is calculated. The ratios and/or testing results may be weighted. From the performance ratios, a histogram is produced to indicate the probabilities of achieving the various ratios. Based on the distribution of probabilities, a particular performance ratio can be predicted with a corresponding level of confidence, and used for capacity planning or configuring a computing environment.

Patents on Peer to Peer Trading Platform

While working at eBay Research Labs I did some work using Skype that resulted in four patents in the area of P2P trading and reputation systems. I just noticed that they are now public, as of late 2007, and here they are:

Peer-to-peer trading platform

Peer-to-peer trading platform with relative reputation-based item search and buddy rating

Peer-to-peer trading platform with search caching

Peer-to-peer trading platform with roles-based transactions

The seed of the idea came from my good friend Josep Ferrandiz, who still works at PayPal. The ideas were developed with help from Zahid Ahmed who is now working for EMC, and with a lot of help from Neel Sundaresan who manages eBay Research labs. The patents are in all of our names.

I think we came up with some cool ideas, we also stress tested the Skype API's in some novel ways and got some things fixed. The lasting legacy of this work is more robust application to application messaging API's and improvements in Skype4Java.

Sunday, March 16, 2008

Monday, March 03, 2008

BIL - recording an excellent adventure

The closing slide of BIL by Garrett Lisi

Two days of intense expansion of my horizons, a self-organizing conference, and I am left holding a jewel. My contribution to BIL was that I basically did tech-support and audio for all the speakers in the main hall for two days. I brought my audio recorder a good mic and stand, a guitar amp (that was used as a PA speaker) and bags of cables and adaptors. Other BILsters provided a video projector and a karaoke machine for PA. I hooked up the mic and a mini-jack cable for computer audio out, and kept it all running as best I could. I recorded almost 800 minutes of content on my Korg D16 hard disk recorder, which is almost all of every talk. There were other people recording on Saturday, including video, but they had technical problems on Sunday, so we have little or no video of the Sunday talks.

Next I'm going to make a backup copy, then start working on extracting the raw data for other people to process. I'll make mp3's of my favorite talks as I get time. Several TED speakers talked at BIL, including Aubrey De Grey on Saturday and Garrett Lisi, who closed out the conference with a fantastic talk. I chatted to Garrett a bit, very cool...

For the stream of consciousness with pictures that I uploaded during the talk, look at my twitxr page. I will update my twitter account with progress on the audio files. I don't expect to use twitter and twitxr on a daily basis, but I think they are useful for conferences. I'll track progress at Ecomm08 on March 12-14th the same way.

Friday, February 29, 2008

Millicomputing at BIL this weekend

I'm presenting (and generally helping out) at the BIL un-conference this weekend in Monterey. I have posted some slides on my Millicomputing blog, on the subject of "The Future in Your Pocket", that speculates on what could happen over the next few years in the mobile space, and starts a discussion of what we might be able to use it for.

Thursday, February 28, 2008

Hiring a researcher at Netflix

Here is the job specification for a research oriented position working for me at Netflix.

We are looking for a very experienced engineer who can take the lead, researching, modeling, presenting, designing and building algorithms that run in a service oriented architecture to support millions of customers. The successful candidate will be a self-motivated, intellectually curious individual. A Doctorate or experience working in a research environment, work with personalization algorithms, machine learning, adaptive systems, and statistical analysis would be a bonus.

I manage a sister group to the one that does the Cinematch star ratings predictions and hosts the Netflix Prize, we use that data as an input along with everything else we know about movies and customers to come up with the actual lists of movies that are shown all over the site. The research position in my group works on algorithms and mining for new data sources.

Also, Netflix is a great place to work, big enough to be interesting and small enough to be fun.

Tuesday, February 12, 2008

Emerging Communications Conference - EComm March 12-14

I'm presenting at the EComm conference on the subject of Millicomputing on the morning of March 14th. There is a Homebrew Mobile Club meeting at the conference (at the Computer History Museum in Mountainview) the evening of March 12th.

Tuesday, January 15, 2008

MacWorld report: iZap, Apple MacBook Air, iPhone and AppleTV

The picture shows an electric car called the Zap, which someone appears to have decided that it needed a few upgrades to become the iZap. Jet motor, rotor blades and iPod interfaces....

I tracked the MacWorld keynote via the Macrumors site on my iPhone this morning. It worked very well, with updates being pushed every few minutes. Over-all there was some good stuff released, but I think expectations were higher.

Later in the afternoon I dropped in on MacWorld for an hour or so, which was plenty of time to check out the interesting stuff.

The MacBook Air is another hit product, destined to become a geek status symbol. It looks good, feels good to use, and has the option of a solid state disk. I had some hands-on time, and I want one.... They are expensive, but the format is very well designed, and I expect it will come down in price and get new options over the coming years. To make the case strong enough despite being thin, there are more curves in the case (curved metal is much stronger than flat sheet) and it has no large holes in it. So there is no removable battery pack, and each key has its own small hole in the casing. It actually looks more mechanically robust than my MacBook pro.

I hoped to find some interesting third party iPhone applications but didn't see anything worth mentioning. I played with a demo iPhone with 1.1.3, it had a bunch of safari based games bookmarked on the second page of icons. Once I got home I upgraded my iPhone, its running fine and the new features work well. They were well leaked a few weeks ago, so there is nothing new to report. The iPod touch picked up the missing set of applications as I expected, but Apple want $20 for the privilege. This is the first example of optional applications, and charging for applications, and I expect the mechanism will get well tested by the Touch then it will be introduced for the iPhone and third party applications.

I have had an AppleTV for six months or so, and the upgraded software looks much better. I don't think I will be renting or buying many movies, but the user interface is better, and the ability to browse both audio and video based podcasts could be interesting. I mostly use my AppleTV for YouTube, playing music and sharing photo albums. One new feature is the ability to pull up photo albums over the Internet from the .mac galleries generated by iPhoto. I've been using .mac for years, and the latest (Leopard) version of iPhoto galleries is very slick and well integrated.

Apple also introduced Time Capsule, which is a network backup server combining a 500GB or 1TB of disk with an Airport. The hardware was not on show as far as I could tell, and it looks similar to the AppleTV package, but with more Ethernet ports and no video port. Personally, I don't want two separate products in this space, I want an AppleTV that operates as a Time Capsule and Airport. I want to backup my laptop and be able to play the music and video directly from the backup copy.

I'm disappointed that there was no new iPhone hardware or Leopard upgrade, and no business oriented iPhone software. I guess that's going to be featured in the next show....