Cockcroft Headroom Plot - Part 2 - R Version

I kept tweaking the code, and came up with a prettier version, that also has a small time series view of the throughput in the top right corner.



The code for this is

chp <- function(x,y,xl="Throughput",yl="Response",tl="Throughput Time Series", ml="Cockcroft Headroom Plot") {
       xhist <- hist(x,plot=FALSE)
       yhist <- hist(y, plot=FALSE)
       xrange <- c(0,max(x))
       yrange <- c(0,max(y))
       nf <- layout(matrix(c(2,4,1,3),2,2,byrow=TRUE), c(3,1), c(1,3), TRUE)
       layout.show(nf)
       par(mar=c(5,4,0,0))
       plot(x, y, xlim=xrange, ylim=yrange, xlab=xl, ylab=yl)
       par(mar=c(0,4,3,0))
       barplot(xhist$counts, axes=FALSE, ylim=c(0, max(xhist$counts)), space=0, main=ml)
       par(mar=c(5,0,0,1))
       barplot(yhist$counts, axes=FALSE, xlim=c(0,max(yhist$counts)), space=0, horiz=TRUE)
       par(mar=c(2.5,1.5,3,1))
       plot(x, main=tl, cex.axis=0.8, cex.main=0.8, type="S")
}

I also made a wrapper function that steps through the data over time in chunks.

> chp.step <- function(x, y, steps=10, secs=1.0) {
    xl <- length(x)
    step <- xl/steps
    for(n in 0:(steps-1)) {
        Sys.sleep(secs)
        chp(x[(1+n*step):min((n+1)*step,xl)],y[(1+n*step):min((n+1)*step,xl)])
    }
}

To run this smoothly on windows, I had to disable double buffering using

> options("windowsBuffered"=FALSE)

and close the graphics window so that a new one opens with the new option.

The data is displayed using the same calls as described in Part 1. The next step is to try some different data sets and work on detecting saturation automatically.

The Cockcroft Headroom Plot - Part 1 - Introducing R

I've recently written a paper for CMG06 called "Utilization is Virtually Useless as a Metric!". Regular readers of this blog will recognize much of the content in that paper. The follow-on question is what to use instead? The answer I have is to plot response time vs. throughput, and I've been thinking about a very specific way to display this kind of plot. Since I'm feeling quite opinionated about this I'm going to call it a "Cockcroft Headroom Plot" and I'm going to try and construct it using various tools. I will blog my way through the development of this, and I welcome advice and comments along the way.

The starting point is a dataset to work with, and I found an old iostat log file that recorded a fairly busy disk at 15 minute intervals over a few days. This gives me 250 data points, which I fed into the R stats package to look at. I'll also have a go at making a spreadsheet version.

The iostat data file starts like this:

                    extended device statistics              
 r/s  w/s   kr/s   kw/s wait actv wsvc_t asvc_t  %w  %b device
14.8 78.4  183.0 2446.3  1.7  0.6   18.6    6.6   1  21 c1t5d0
 0.0  0.0    0.0    0.0  0.0  0.0    0.0    5.0   0   0 c0t6d0
...

I want the second line as a header, so save it (my command line is actually on OSX, but could be Solaris, Linux or Cygwin on Windows)
% head -2 iostat.txt | tail -1 > header

I want the c1t5d0 disk, but don't want the first line, since its the average since boot, and want to add back the header
% grep c1t5d0 iostat.txt | tail +2 > tailer
% cat header tailer > c1t5.txt

Now I can import into R as a space delimited file with a header line. R doesn't allow "/" or "%" in names, so it rewrites the header to use dots instead. R is a script based tool with a command line and a very powerful vector/object based syntax. A "data frame" is a table of data object like a sheet in a spreadsheet, it has names for the rows and columns, and can be indexed.
> c1t5 <- read.delim("c1t5.txt",header=T,sep="")
> names(c1t5)
[1] "r.s" "w.s" "kr.s" "kw.s" "wait" "actv" "wsvc_t" "asvc_t" "X.w" "X.b" "device"

I only want to work with the first 250 data points so I subset the data frame by indexing the rows with an array (1:250) that selects the rows I want and leaving the column selector blank.
> io250 <- c1t5[1:250,]

The first thing to do is summarize the data, the output is too wide for the blog so I'll do it in chunks by selecting columns.

> summary(io250[,1:4])
      r.s              w.s             kr.s             kw.s        
 Min.   :  1.80   Min.   :  1.8   Min.   :  13.5   Min.   :   38.5  
 1st Qu.: 10.30   1st Qu.: 87.1   1st Qu.: 107.4   1st Qu.: 2191.7  
 Median : 18.90   Median :172.4   Median : 182.8   Median : 4279.4  
 Mean   : 22.85   Mean   :187.5   Mean   : 290.1   Mean   : 4448.5  
 3rd Qu.: 28.88   3rd Qu.:274.6   3rd Qu.: 287.4   3rd Qu.: 6746.6  
 Max.   :130.90   Max.   :508.8   Max.   :4232.3   Max.   :13713.1  
> summary(io250[,5:8])
      wait             actv            wsvc_t           asvc_t      
 Min.   : 0.000   Min.   :0.0000   Min.   : 0.000   Min.   : 1.000  
 1st Qu.: 0.000   1st Qu.:0.3250   1st Qu.: 0.400   1st Qu.: 3.125  
 Median : 0.600   Median :0.8000   Median : 2.550   Median : 4.700  
 Mean   : 1.048   Mean   :0.9604   Mean   : 5.152   Mean   : 4.634  
 3rd Qu.: 1.300   3rd Qu.:1.5000   3rd Qu.: 6.350   3rd Qu.: 5.700  
 Max.   :10.600   Max.   :3.5000   Max.   :88.900   Max.   :15.100  
> summary(io250[,9:10])
      X.w             X.b       
 Min.   :0.000   Min.   : 2.00  
 1st Qu.:0.000   1st Qu.:20.00  
 Median :1.000   Median :39.50  
 Mean   :1.428   Mean   :37.89  
 3rd Qu.:2.000   3rd Qu.:55.00  
 Max.   :9.000   Max.   :92.00  

Looks like a nice busy disk, so lets plot everything against everything (pch=20 sets a solid dot plotting character)
> plot(io250[,1:10],pch=20)
The throughput is either reads+writes or KB read+KB written, the response time is wsvc_t+asvc_t since iostat records time taken waiting to send to a disk as well as time spent actively waiting for a disk.

To save typing, I attach to the data frame so that the names are recognized directly.
> attach(io250)
> plot(r.s+w.s, wsvc_t+asvc_t)
This looks a bit scattered, because there is a mixture of average I/O sizes that varies during the time period. Lets look at throughput in KB/s instead.
> plot(kr.s+kw.s,wsvc_t+asvc_t)
That looks promising, but its not clear what the distribution of throughput is over the range. We can look at this using a histogram.
> hist(kr.s+kw.s)

We can also look at the distribution of response times.
> hist(wsvc_t+asvc_t)
The starting point for the thing that I want to call a "Cockcroft Headroom Plot" is all three of these plots superimposed on each other. This means rotating the response time plot 90 degrees so that its axis lines up with the main plot. After looking around in the manual pages I eventually found an example that I could use as the basis for my plot. It needs some more cosmetic work but I defined a new function chp(throughput, response) shown below.

> chp <- function(x,y,xl="Throughput",yl="Response",ml="Cockcroft Headroom Plot") {
   xhist <- hist(x,plot=FALSE)
   yhist <- hist(y, plot=FALSE)
   xrange <- c(0,max(x))
   yrange <- c(0,max(y))
   nf <- layout(matrix(c(2,0,1,3),2,2,byrow=TRUE), c(3,1), c(1,3), TRUE)
   layout.show(nf)
   par(mar=c(3,3,1.5,1.5))
   plot(x, y, xlim=xrange, ylim=yrange, main=xl)   par(mar=c(0,3,3,1))
   barplot(xhist$counts, axes=FALSE, ylim=c(0, max(xhist$counts)), space=0, main=ml)
   par(mar=c(3,0,1,1))
   barplot(yhist$counts, axes=FALSE, xlim=c(0, max(yhist$counts)), space=0, main=yl, horiz=TRUE)
}

The result of running chp(kr.s+kw.s,wsvc_t+asvc_t)is close...


That's enough to get started.

ps3 Marketplace Research on eBay

Over at Data Mining there is some interesting info on ps3's.

However, there is no need to do manual scraping of
eBay, here is a screenshot from the marketplace research function
that is bundled with my eBay store subscription. For $2.99 for 2 days
access anyone can get at this.

http://pages.ebay.com/marketplace_research/

Skype on Solaris

http://blogs.sun.com/darren/entry/skype_1.3.0.53_on_solaris_via

Solaris has a Linux compatible subsystem called BrandZ for running Linux binaries that don't have Solaris builds (like Skype). Darren figured out how to get the Linux build of Skype to run on Opensolaris.

Thanks to Alec for pointing this out.

10 Things to Know About Skype Ap2Ap Programming

I also posted this on the Skype Developer Wiki

The ap2ap capability is an interesting new network computing paradigm but it is not like a conventional network.

1. end nodes are addressed by skype name, which addresses a person, not a computer
   


2. people can login to skype multiple times, so addressable endpoints are not unique
   


3. skype can go online/offline at will, so there is a concept of "presence" that needs to be managed
   


4. you can only make ap2ap connections to your buddy list or people who you have chatted to "recently"
   


5. both ends of an ap2ap connection have to choose a unique string used to identify their conversation or protocol
   


6. if you quit and restart skype, the first login can persist for a while, so you can get multiple ap2ap connections from a single user, although the ghosts of your previous connections cannot respond to a message. I think is is because you connect to a different supernode each time, and the first one isn't sure if you have really gone away yet
   


7. messages have to be sent as text, so binary objects have to be converted first using something like base64
   


8. the network can behave differently each time you use it, and this non-determinism makes testing difficult
   


9. relayed connections are limited to about 3KB/s, direct ones can run at several MB/s over a LAN
   


10. Skype4Java is cross-platform, but the maximum message size is about 64KB on windows and 16KB on OSX/Linux, and there are several bugs and limitations in the older version of the API library that is used by Skype 2.0 and earlier releases. Use Skype 2.5 or later for the best performance and stability
    

Bloglines, OPML, Blogger and Flock

I aggregate 50 or so blog feeds using Bloglines, its a very useful way to keep track of infrequent blogs in particular, and it strips off the adverts and other decoration from the power bloggers.

I just did some tidying up of my blog list, exported it in OPML format and uploaded it to http://share.opml.org/

This is an interesting way to contribute to the "Top 100" blogs list on that site, and it also has some useful features like seeing who else reads the same blogs, and who has the most similar list of blogs.

I seem to have added to the "long tail" since many of the blogs I read were new to the site, so I'm the only reader. My blog had one other reader "Ian" (hello!) who has a very long list of blogs, that I may poke around in if I get some free time...

Meanwhile, Flock is working well as my cross-platform browser. I'm writing blog entries with it, although I did lose an entry I was writing a week or so ago, when I upgraded Flock and didn't save an almost complete entry first. The new version of Flock appears to automatically save entries every few minutes, but I hate re-writing things, so that entry may not be re-created for a while.

Flock seems to have some issues writing entries to blogger.com at the moment. I'm not using the updated blogger.com, but Flock fails to write the blog entry most of the time, then randomly works. I gave up and used cut and paste to post this entry directly....

Pandora Prog Channel

I've been trying Pandora on and off for Internet music for a while, and attended a talk by CEO Tim Westergren last week, which got me to try it again. They are continuously improving their algorithms for choosing music, and I was trying to make a channel that would serve me interesting new music alongside some of my favourite experimental "Prog Rock" bands. It seems to be working much better, and I keep tuning the channel by skipping tracks that I don't like and giving thumbs up to the ones I do. The nice thing is you can listen to my channel, even though you don't get exactly the same songs as I do, there should be an interesting mix of King Crimson, Zappa, Estradasphere, and many other bands playing music you won't hear often. Its easy to make your own channel (it takes less training to make a more mainstream channel) and its the best way I've found to discover completely new music.

http://www.pandora.com/?sc=sh59488715848528731

Enjoy...

Blogged with Flock

CMG06 Conference - Reno December 3-8

As usual I'll be attending the Computer Measurement Group conference in Reno Nevada this December. I've attended every year since 1994, and its the place where I get an update on the state of the art in Performance Management, and get to mingle with my friends and peers who work on Capacity Planning.

This year I'm presenting three times:

1. Sunday morning 3 hour seminar on Capacity Planning with Free and Bundled Tools. This is a repeat of last years talk, presented jointly with Mario Jauvin, who covers the Windows OS and Networking related areas. I cover Solaris, Linux and the system oriented tools.
2. Wednesday morning conference paper titled "Utlization is Virtually useless as a Metric!". Regular readers of this blog will recognize much of the content of this paper, which gathers together all the ways in which your measurements can be corrupted by virtualization.
3. Thursday morning I'm giving a 3 hour training course called the Unix/Linux CMG Quick Start Course, which is part of a new feature for CMG and is based on the training classes in performance tuning that I have given for many years.

Early bird discounted registration is open until October 13th. The sunday seminars are an extra cost item, but the Thursday morning training classes are included in the regular conference fee. This is the only place I'm planning to give public training classes, and since I'm at the conference all week its a great opportunity to discuss performance and capacity issues in person. I hope to see you there...

technorati tags:cmg, solaris, performance, capacity, training, seminar, linux

Blogged with Flock

Updated Ad Setup

I just changed to the flash based ad sidebar. It lets you pick different keywords without reloading the page by clicking on the top tab.

I also continued to focus on Technology Books, picking some specific categories to try and exclude the certification and training books that I find less interesting. I then excluded Microsoft and mcse as keywords, and ended up with Cisco certification so I excluded them as well, and it looks like a more reasonable selection now.

Blogged with Flock

Comments on Web Traffic

This blog gets about 50 visitors a day, and most new visitors arrive as the result of a google search for my name, Thumper/ZFS or the SE toolkit. There is a very small number of yahoo and msn searches. The other main source of traffic is the Sun Community blogging site, which links to Sun Alumni's blogs including this one. A few weeks ago I got a lot of traffic from Sun, and I think I traced it back to a blog entry from Jonathan Schwartz, who talked about his General Counsel's blog, and Mike Dillon the GC mentioned the Sun Community blog site. This doubled my traffic for a week or two.

The other recent change is that I stopped showing adverts from ctxbay, which was created by an eBay developer as a side project, won a prize, but never really worked well enough to be useful. I've replaced them with the official eBay in-house AdContext system, which is being beta tested. AdContext looks at your page content, and matches keywords it with popular items from eBay. It can be configured to exclude certain keywords (in my case I exclude "Adrian", which was causing problems for ctxbay), and you can choose certain categories or stores to pick items from. I've picked Technology Books and some Storage hardware categories. I'm going to experiment with different formats and constraints, to see how well it works.

There are three formats, text only, pictures (which I started with) and flash (which scrolls multiple adds into the same amount of screen space). I'll switch it to flash when I get around to it...

technorati tags:sun, thumper, traffic

Blogged with Flock

Wally's Items...

http://www.dilbert.com/comics/dilbert/archive/dilbert-20060821.html

Sounds like a good plan....

technorati tags:dilbert

Blogged with Flock

Web vs. Skype, a paradigm shift

The essential characteristics of the http based web are that by default everyone is anonymous, and everyone can get to everything. Its "free" and the trend is for the parts that are not free and anonymous to move in that direction. For example, you can now buy stuff on eBay Express without having to sign up for an eBay account, and there are fewer newspaper sites requiring paid subscriptions, since they are losing audience to the free sites.

However, the essential characteristics of the Skype peer to peer network are the opposite of the Internet. Everyone has a clear identity and no-one can get to anything without asking for permission or being invited. I think this truly a different paradigm for building systems.

Everyone on Skype is plugged into the public key encryption infrastructure (PKI) which provides a secure identity as well as secure communications between peers. However, to communicate with other peers you need to know them and have permission. For me the most interesting capability on Skype is the application to application messaging API (ap2ap) that enables a new class of distributed applications that leverage the social network formed by the mesh of Skype contact lists.

The upshot of this is that some things that are easy on the Internet are difficult on Skype, and vice versa. There is a temptation to take something that we know works on the web, and try to make something similar on Skype ap2ap, but that is pointless, just use the web! Look for things that really don't work well on the web, or look for web-based systems that connect a few people but need an expensive back-end or don't scale. This is the start of something interesting....

technorati tags:skype, ap2ap, scalability

Blogged with Flock

Solaris Internals and Performance 2nd Edition

Richard and Jim have finally finished their updated book and got it published. Rush out and buy a copy! I just listened to a podcast where they talked about it and mentioned that you can get it for 30% off from http://www.sun.com/books. Strangely, my own Sun Performance and Tuning book isn't listed there, although my BluePrint books on Capacity Planning and Resource Management are.

I was also amused to see that in the slide deck they use to launch the book they reference Adrian's Rule of book writing (book size grows faster than you can write).

Congratulations!

Now do I have time to start another book? I'm not sure... maybe.

technorati tags:Solaris, performance

Blogged with Flock

Sun ZFS and Thumper (x4500)

I was one of the beta testers for Sun's new x4500 high density storage server, and it turned out pretty well. I was able to hire Dave Fisk as a consultant to help me do the detailed evaluation using his in-depth tools, and it turned into a fascinating investigation of the detailed behavior of the ZFS file system.

ZFS is simple to use, has lots of extremely useful features, and the price is right (bundled with Solaris 10 6/06 or OpenSolaris). However its doing lots of clever things under the hood and it behaves like nothing else. Its far more complicated to predict its performance than any other file system we've looked at. It even baffled Dave at first, he had to change his tools to support ZFS, but he's got it pretty well figured out now.

For a start, its a write anyware file system layout (WAFL) which is similar in some ways to a NetApp filer. This means that random writes are batched up, sorted by file, file system etc. and every few seconds a big burst of sequential writes commits the data to disk as a transaction. Since sequential writes to disk are always much more efficient than random writes, this mean that it gets much more performance per disk than UFS/VxFS etc for random writes.

The combination of the x4500 and ZFS works well, since ZFS knows that the firmware on the 48 SATA drives in the x4500 have a write cache that can safely be enabled and flushed on demand. This greatly improves performance and fixes an issue that I have been complaining about for years. Finally a safe way to use the write caches that exist in every modern drive.

Its actually easier to list the things that ZFS on the x4500 doesn't have.

• No extra cost - its bundled in a free OS
  
• No volume manager - its built in
• No space management - file systems use a common pool
  
• No long wait for newfs to finish - we created a 3TB file system in a second
• No fsck - its transactional commit means its consistent on disk
• No rsync - snapshots can be differenced and replicated remotely
• No silent data corruption - all data is checksummed as it is read
• No bad archives - all the data in the file system is scrubbed regularly
• No penalty for software RAID - RAID-Z has a clever optimization
• No downtime - mirroring, RAID-Z and hot spares
• No immediate maintenance - double parity disks if you need them
• No hardware failures in our testing - we didn't get to try out some of these features!
  

and finally, on the downside

• No way to know how much performance headroom you have
  
• No way to get at the disks without taking the top off the x4500
  
• No clustering support - I guess they couldn't put everything on the wish list...

The performance is actually very good, and in normal use its going to be fine, but when we tried to drive ZFS to its limit, we found that the results were less consistent or predictable than more conventional file systems. Some of the issues we ran into are present in the Solaris 10 6/06 release, but when the x4500 ships it will have an update to ZFS that includes performance fixes to speed things up in general and reduce the impact of the worst case issues, so it should be more consistent.

We've put ZFS on some of our internal file servers, to see how it goes in light usage. However, it always takes a while to build up confidence in a large body of new code, especially if its storage related. If we can add this one to the list:

• No nasty bugs or surprises?
  

Then ZFS looks like a good way to take a lot of cost out of the storage tier.

I'm interested to hear how other people are getting on with ZFS, especially mission critical production uses.

technorati tags:ZFS, Solaris, x4500, Thumper, Ortera

Blogged with Flock

IEEE Conference Paper

I attended the IEEE E-Commerce conference in San Franscisco. The conference is known as CEC06/EEE06 and some other acronyms. It was a very interesting academic oriented event with a few hundred people from all over the world, I made some good contacts and learned some new stuff.

My own paper was about how I built a large scale simulation of a peer to peer network using a very efficient architecture based on the Occam language. I used to write a lot of Occam about 20 years ago, and it seemed appropriate to the problem I wanted to solve. I think most people are baffled by the language, but I like it. Unlike most recent languages where everything is an object with types and methods, in Occam everything is a process with protocols and messages. The other difference is that Occam was designed to run fast on a 10MHz CPU and on todays CPUs it is extremely fast and small compared to recent languages like Java.

What I found at the conference was that most of the simulation frameworks people were using were run overnight to generate results. My own example simulation of 1000 nodes ran for about three seconds to produce an interesting result.

The full paper can be obtained from http://doi.ieeecomputersociety.org/10.1109/CEC-EEE.2006.81

This is the official URL, and IEEE charges non-members for downloads.

technorati tags:IEEE, Occam, simulation

Blogged with Flock

CPU Power Management

AMD PowerNow! for the Opteron series of server CPUs dynamically manages the CPU clock speed based on Utilization. The speed takes a few milliseconds to change, and it is not clear exactly what speeds are supported, but one report stated that the normal speed of 2.6GHz would reduce to as low as 1.2GHz under a light load. This report also shows CPU detailed configuration and power savings. http://www.gamepc.com/labs/view_content.asp?id=opteron285&page=3

The problem with this for capacity management is that there is no indication of the average clock rate in the standard system metrics collected by capacity planning tools. PowerNow! is described by AMD at http://www.amd.com/us-en/0,,3715_12353,00.html and drivers for Linux and Windows are available from http://www.amd.com/us-en/Processors/TechnicalResources/0,,30_182_871_9033,00.html. In the future, operating systems may be able to take the current speed into account, and estimate the capability utilization, but the service time is higher at low clock rates, so we will still see some confusing metrics.

The current PowerNow! implementation works on a per-chip basis, and Opteron’s have two complete CPU cores per chip that share a common clock rate. In a multiprocessor system made up of several chips, each pair of processing cores could be running at a different speed, and their speed can change several times a second.

Our basic assumption for well behaved workloads, that mean service time is a constant quantity, is invalidated in a very non-linear manner, and utilization measurements will also move in mysterious ways....

eBay AdContext Contextual Adverts

I've been running contextual ads on this site for a few months, using the CTXbay service that won an eBay developers program award. I don't think I've had many people click through, since the ads aren't very relevant, and seem fixated on the word "Adrian".

Now that eBay has announced its own AdContext service is on the way, I'm planning to replace CTXbay with the official service as soon as I can get access to it. The eBay service has access to a lot more information and is quite customizable. I asked about it at the Developers Conference and was told that I could set a default category and control what kind of items appear.

eBay Wireless WAP Access | by Adrian Cockcroft | June 12th, 2006

I was at the eBay developers conference showing some proof of concept prototypes of new mobile applications, and I found that hardly anyone knew that eBay already has a WAP based mobile version of the site. It loads in seconds on any phone that has any kind of web browser, but there is no automatic redirect from the main eBay site. You should bookmark this on your phone's web browser:

http://wap.ebay.com

You can also use this site to backend other mobile applications. If your own code helps a user find an item on eBay then you can form a URL that contains the item id and go directly into the official eBay WAP based site. It handles user login, MyEbay, watchlists etc. The main problem with the wap site is that the search functionality is too simplistic. The prototypes we were showing (no, you can't access them, you should have been there...) are aimed at fixing the finding experience.

Part 3: Disruptive Innovation viewed as a Maturity Model | by Adrian Cockcroft | June 6th, 2006

This time I'll take a more abstract view of a maturing market as each phase evolves, and refer to the development of in-home movie watching as an example.

1. An emerging market is characterized by competition on the basis of technology. Early adopters like to play with new technology and are able to cope with its issues. Many different products are competing for market share on the basis of "my features are better". Think of the early days of the VCR, with VHS vs. Betamax. In a mature market, few people worry about features, most VCR or DVD players have the same feature set and very good picture quality at a very low price. If you want to be sure you get a good one, you are most likely to buy using brand name (e.g. Sony) rather than poring over detailed specifications. Margins are low, but volume is high and margins can be better if you won the brand battle.
   
2. The next phase in the market is characterized by competition on the basis of service. Think of the video rental store as a service. You visit the store and pay rental according to how much you use the service. As an emerging service, anyone could setup to rent videos and DVDs. As the market matured, larger stores with a bigger selection and more centralized buying power provided a better service, and video rental chains such as Blockbuster took over the market. Again, the power of a dominant brand became the primary differentiator as the service market matured.
3. The third phase in the market is the evolution of a service into a utility. A utility provides a more centralized set of resources, and a regular subscription or monthly bill. It can provide similar services, but in a more automated manner. NetFlix is my example of a utility based DVD provider service. You pay a monthly fee which encourages steady consumption, and NetFlix have automated the recommendation system, which replaces asking the counter clerk in a video rental store for advice. The recommendations are the result of many peoples opinions, so are likely to be less biased and better informed, but the most important difference in the utility approach is that it doesn't need people to provide the service directly to the customer. This makes it fundamentally cheaper. Many traditional services were transformed into utilities by the arrival of the Internet, which allows consumers to access information based utilities in a generic and efficient manner. The network effect benefit of having a large user base also causes dominant brand names to emerge. NetFlix leads mindshare in this space, despite attempts by BlockBuster to copy their business model, NetFlix can grow faster with fewer people as a pure utility.
   
4. The final phase in the evolution of a market occurs as the cost of replication and distribution of the product approaches zero. For digital content the end customer already has a computer and an Internet connection. There is no additional cost to use it to download a movie. A central utility such as YouTube can use a mixture of advertising and premium services (for a minority of power users) to offset their own costs. Peer to peer systems distribute the load so that there is no central site and no incremental cost in the system. The only service that is needed is some kind of search, so that peers can find each other's content to exchange it. PirateBay is primarily a search engine, and search engines become dominant when the brand gets well known, and they find what you are looking for because they have a comprehensive index.

So the evolution of a marketplace goes from competing on the basis of technology, to competing on service, to competing as a utility, to competing for free. In each step of the evolution, competitors shake out over time and a dominant brand emerges.

To use this as a maturity model, take a market and figure out whether the primary competition is on the basis of technology, service, utility or search, and consider whether a dominant brand has emerged in that phase. The model should then indicate what the next step is likely to be, so you can try to find the right disruptive innovation to get you there. Good luck!

Part 2: Moving Pictures - disruptive innovation from the Cinema to PirateBay | by Adrian Cockcroft | June 3rd, 2006

Lets look at the history of movies. The initial technology to capture and replay moving pictures was developed around 100 years ago, and the initial competition between inventors went through its first transition when movie theaters became established and began to settle on a standard form of projector. The inventors who had alternative camera/recording/projector technology died out. Consumers wanted to go see movies and the movie industry formed to provide content for that market.

The next innovation was to be able to watch movies at home on film, then there were movies on TV. The movie theaters had far bigger screens, better sound and color but the technology at home gradually caught up in features and reduced in cost, and a market transition to home viewing occurred. The total market size for equipment bought to watch movies at home is huge. Its important to note that the primary vendors in each phase of the market are different. The movie theater business is very different to the home video equipment supplier business. The early battles in the home were over the standard formats, famously Betamax failed to win over VHS for video tape, and there are continuing battles over DVD formats, but Sony is a dominant brand name in a crowded market for home video equipment.

The next innovation was video rental, and Blockbuster ended up as a major player in this market, with presence on every high street. However, that presence became unnecessary as Netflix shipped DVD's directly to consumers and took over a large share of the market.

Finally, video is available directly over the internet , its being viewed on PC's rather than TV sets, anyone can create and upload it, and YouTube is this year's hot market leading name in this space for all kinds of short videos. Its also trivially easy to take a full length movie or TV program and share it using one of the many BitTorrent services, and a growing proportion of movies are being watched for free, to the consternation of the movie industry.

The PirateBay site in Sweden was recently shut down and charged with copyright violation, but it appears that a significant proportion of the population of Sweden were users and they got upset as they had got used to exchanging content for free. After three days the site came back up, hosted in Holland, and with even more users due to the publicity.

Unlike YouTube, BitTorrent sites such as PirateBay don't host the actual content, they just connect individual users who exchange content, they don't need to provide storage or bandwidth, just a searchable database of small index files that configure the BitTorrent transfer between a large number of seeders that have some or all of the file already, and leechers who want to get the file, and who can in turn become seeders.

The publicity gained as a side effect of trying to shut down the PirateBay site may even have the opposite effect of cementing the PirateBay brand as a market leader and accelerating growth in this space.

Every step in this history involves a disruptive innovation. There is a fundamental reduction in cost, offset by a large increase in unit volume, which has often increased the overall revenue using a new way to monetize the market for moving pictures. Each time the previous market leader is left behind (often kicking and screaming) as the new larger market emerges. Each time a new brand captures the attention span and trust of the consumer, and dominates the market.