Some New Performance Monitoring Tools

There is a simple and extensible open source C based daemon called collectd that writes to RRD files, an alternative to Orca/procallator for people who don't want the Perl based memory footprint of procallator. I'll check it out on my Gumstix millicomputer sometime.

There is yet another open source full-function monitoring tool called Zabbix that looks similar to Cacti in scope, possibly with more features, and with a SQL database backend. It has a commercial company backing it with support contracts etc, somewhat like the XE Toolkit.

The most interesting commercial tool I saw at CMG earlier this month is a capacity monitoring tool called PAWZ from Perfcap Corporation. The key thing they have worked on is taking the human out of the loop as much as possible with sophisticated capacity modelling algorithms and a simple and scalable operational model. It is very similar in concept to the capacity planning research I was working on and publishing in 2002-2004. The core idea is that you care about "headroom" in a service, and anything that limits that headroom is taken into account. Running out of CPU power, network bandwidth, memory, threads etc. will increase response time of the service, so monitor them all, track trends in headroom and calculate the point in time where lack of headroom will impact service response time. At eBay we used to call this the "time to live" for a service. You can easily focus on the services that have the shortest time to live, and proactively make sure that you have a low probability of poor response time. I'm going to take a closer look at this one...

Thoughts on iPhone 1.1.3, Macworld, 3G and business users

Reports indicate the the 1.1.3 update is likely to ship before MacWorld in January, it includes an update to Google Maps that adds the same location feature as recently shipped on other non-GPS platforms (basically locating to the nearest cell-tower). It also provides mechanisms to select and arrange the applications installed on the iPhone. This is a key new feature, since the iPhone has been updated by Apple as a monolithic all-or-nothing set of applications so far. It greatly reduces the need to hack into the iPhone to customize it.

The updates themselves are fairly small, and to me it makes sense to get them installed before MacWorld. We will then see announcements at MacWorld of a set of optional applications that users can pick and choose to install on their iPhones. This could include the ability to add the Mail application and other "missing" iPhone apps to the iPod Touch. It could include brand new applications (I'm going to nominate iChat as a candidate yet again...). The public announcement of the SDK is due at MacWorld, with shipping in February, which will open up anyone to build officially sanctioned applications. However key vendors will have been testing the SDK over the last few months so I expect a bunch of third party applications to be announced or ready to ship at MacWorld.

The other leaks and rumors indicate that there is likely to be a second generation iPhone with 3G support shipping in the spring, and announced at MacWorld. This would also support launching the iPhone in Asian markets like Japan, where there is no GSM support.

I also expect that Apple will start to make moves towards business use of the iPhone, with some tools and upgrades provided by Apple, and others by key third parties.

I currently carry a Verizon Blackberry 8703e for work use, and my iPhone for personal use and iPod functionality. In order to use the iPhone as my work phone I need a few key features.

• Firewall support - the BB is inside the corporate firewall, the iPhone can't access it. We use Juniper Network Connect which is a Java based VPN solution on MacOS/XP.
• WiFi support - we use LEAP to login to WiFi at work, need support for LEAP on iPhone, it works fine on my MacOS X laptop, should be a simple feature to add.
• Exchange support - I can't use the IMAP workaround due to firewall issues, properly integrated Exchange email and calendar support is what everyone is asking for.
• Ideally RIM will port the Blackberry application suite to the iPhone, like they did for the Treo...

The two other biggest missing features are Flash and Java support. I know there are lots of issues with CPU/memory/battery life. Perhaps the next generation iPhone will be based on a more advanced ARM CPU (e.g. the ARM Cortex based Qualcomm Scorpion) with more performance and more memory so it can run Flash and Java apps alongside the existing apps?

We'll find out in a few weeks...

Happy new year.

MacOS X Leopard, iPhone and Stereo Bluetooth Headphones - A2DP

Over a year ago I bought stereo bluetooth headphones on eBay, its a multifunction unit the OMIZ OMS600, and it includes an MP3 player (with MicroSD slot), FM radio, Stereo A2DP headphone and Headset with Mono audio/Microphone. When I tried to configure it on MacOS X Tiger it didn't work because there was no A2DP headphone support.

MacOS X Leopard now supports A2DP and "just works" with this headset. The Mac sees both the Headset (Mono audio/mic for Skype etc) and Headphone (A2DP Stereo) as separate devices. After the usual Bluetooth device wizard setup, simply put the OMS600 into the headphone mode and pick "Use Headphone" from the Bluetooth dropdown menu on the Mac, the Mac's internal speakers mute, and the Headphones play. It worked over a 10 foot range walking around a room, crackled a bit at the limit and dropped the connection if I went too far away.

I don't see the OMS600 for sale any more, but there are plenty of A2DP headphones out there now, and its nice to see that Apple finally got the devices to work, and made it "Just Work" as usual.

The headset mode also works on my iPhone, but the A2DP mode doesn't. I paired the headset with the iPhone and was able to make calls and receive them. When using the FM radio or the MP3 player built-into the headset it paused and resumed for incoming calls. However when the Headphone mode was paired with my Leopard machine the iPhone didn't route calls to the Headset. The OMS600 headset has the microphone built into the left side earpiece with no voice tube or boom down nearer my mouth, and it doesn't pick up very clearly. The noise cancelling Jawbone headset works far better.

I listened to some music on iTunes (Radiohead In Rainbows and Gorillaz D-Sides are my current albums for serial listening) and waited forever for XP to start up in Parallels so I could fire up IE7 and see a Netflix Watch Instantly show about the Pixies reunion tour (called LoudQuietLoud). [Yes I know it would be nice if it worked natively on the Mac, but the studios only approve Windows DRM, and the alternatives all have issues that are taking way too much time to sort out].

Happy new year...

A. A. Michelson Award Acceptance Speech

Last week I was selected as this year's winner of the A. A. Michelson Award for Lifetime Achievement and Contribution by the Computer Measurement Group. Here is a transcript of my acceptance speech.

Thanks to CMG, the Michelson Award winners who voted to add me to their ranks, and everyone who I have met here since I first attended CMG in 1994. I treat the CMG conference as my annual training class, a chance to mingle with my peers, learn and share the things I have learned.

There are some people who I have met at CMG or while working in this field who have become special friends, not just people I see at the conference each year. Cathy Nolan is not only national CMG president this year, she is president of my local CMG group in Northern California and was also instrumental in the process of getting me nominated for this award. We have all learned a lot over the years from Neil Gunther. Neil and I have jointly presented many training classes so I have had plenty of opportunity to learn his material. He has made queuing theory accessible and useful for very many people. Mario Jauvin has been my CMG conference buddy for many years, and for the last three years we have jointly presented the Capacity Planning with Free Tools workshop. Yefim (Fima) Somin encouraged me to attend CMG in the first place, we met in 1994 while working to port BGS Best/1 to Solaris, and we have kept in touch over the years, even though he doesn't get to come to CMG nowadays. Fima and Henry Newman were part of the Universal Measurement Architecture standards body that I joined in 1995, and I have also kept in touch with Henry. We worked to get some of his ideas (like extended system accounting) to be implemented in Solaris. Bob Sneed worked with me at Sun and is now the main contact point and conscience for CMG at Sun.

Outside CMG there are some very significant people I would also like to thank. Brian Wong is one of my closest friends, responsible for getting me to move to the USA in 1993, and is also author of the Capacity Planning and Configuration for Solaris Servers book. Brian introduced me to Rich Pettit, and we worked together for many years to build and extend the SE Toolkit, the vehicle for most of my performance measurement ideas. Allan Packer was the co-author of my first CMG paper on Database Sizing. He knows everything about database performance and wrote the book Configuring and Tuning Databases on the Solaris Platform. We borrowed Allan from Sun's office in Adelaide Australia for a month or two, and he introduced me to Richard McDougall, who we also borrowed from Adelaide. Allan and Richard both ended up working for Sun's Performance group in California. Richard built the first tools that could measure memory usage, fixed the Solaris 8 memory system, and co-authored the Solaris Internals books. He became a Sun Distinguished Engineer and recently joined VMware as Chief Performance Architect. Jim Mauro is the other author of Solaris Internals, we borrowed him from Sun's New Jersey office a few times before he also joined the performance group full time. Finally Dave Fisk worked with Brian Wong and I at Sun, and became one of my closest friends. He knows more about Unix storage performance than anyone else I've met, and I've called on him many times (he's now a consultant) to help figure out what's really going on.

I would also like to thank my family. My father is a retired Statistics Professor from the University of Hertfordshire (a.k.a. Hatfield Polytechnic) in the UK. He was programming himself in the 1960's and gently encouraged me to tinker with computers from an early age. My son recently graduated with his BSc in Computer Science from the same University, and now works as a project manager at eBay in California. My daughter has just started a degree course in European Literature at Royal Holloway College, London, so I'm no help at all with her assignments :-)  I'm very proud of them. I moved to the USA on my own a few years ago and met my lovely wife Laurel in 2003. I'm very happy that she has also been able to attend CMG with me and meet many of you.

I have always been glad that I studied Physics, the work we do in performance and the diagnosis and measurement of performance leans heavily on my training as a Physicist and I think that pure Computer Scientists are lacking some of the tools they need to understand measurement and performance issues. Since A. A. Michelson was a Physicist this award is especially gratifying for me.

In 1972 my High School had a link to the computer that my Father used at Hatfield Polytechnic. I taught myself BASIC and Algol during recess....

This is what most people know me for, its getting a little out of date now, since the second edition was published in 1998, but a big thank-you to my co-author Rich Pettit and everyone that bought a copy!

The book documented metrics and performance rules, and they were implemented as an SE Toolkit script called Virtual Adrian. I spent a lot of time with all the tools vendors helping them to manage Solaris better, and with Solaris engineering getting them to make Solaris more manageable. This is what justified my involvement with CMG during my time at Sun, and its the legacy that I am most proud of.

To finish, some of the things I have learned over the years that you may find useful...

If you ever try to write a book, you need to pay attention to Cockcroft's Law of Book Writing. It seems counter-intuitive to people who have never attempted to write a book, but it is the most important thing to know if you ever intend to finish a book. For example, you scope a 200 page book and start writing. After you have written 50 pages you re-scope and find your book is now a 300 page book. You write another 50 pages and find you are now writing a 400 page book. The number of pages you have left to write to finish the book is increasing! You have to reduce scope continuously and plot a trend line that slopes downwards if you ever hope to be published.

If you collect information and share it freely, you will become an expert. You become a magnet for questions, issues and information. Some people tightly control their expertise, this is a huge mistake as it gives others incentives to look for alternative experts or to become experts themselves. If you give out your expertise freely, you become the go-to person in your field, and gather many more recommendations from the people you help. Try it!

Some observations that may help you deal with executive level management. The first one is a big problem that Sun and other manufacturers have, because the things they build take many years to go from conception to delivery. If the managers in charge of these projects don't stick around then the project is likely to be tinkered with and delayed further or dropped. Its nice being at Netflix, where the executives have been there for years and the projects take months to deliver. The second observation is that Capacity Planners have a vertical role in the organization, they gather the lowest level metrics from operations and have to present findings to upper management. We love our detailed metrics, but the only metric that upper management should have to deal with is the dollar. If you can present everything in terms of return on investment (spend $100K to save or grow by $1M) then your executive presentations will be much more effective.

What am I working on now? I'm managing a team that develops the Netflix web site. Its a personalized web service and I think that is the future of the computer industry. The scalability and performance challenges are very interesting. I'm also pushing an idea I call "Millicomputing" which could have a disruptive effect on the industry as storage moves from disks (spinning rust) to flash, and the computers move to ultra low power system-on-a-chip devices. I gave a paper on Millicomputing at CMG07 and have a separate blog on the subject.

My paper at last year's CMG was called "Utilization is Virtually Useless as a Metric", and this answer sums it up. Those who ask questions about utilization don't understand that their questions have no meaning so the answers are irrelevant :-)

Thanks again!

Adrian

CMG07 and the A. A. Michelson Award

From the Computer Measurement Group website:

Each year, CMG considers exceptional individuals who have made significant contributions to our profession over their entire professional careers as possible recipients of the A.A. Michelson Award.

Albert Abraham Michelson, for whom the award is named, was known for his outstanding technical accomplishments in measuring the speed of light as well as for his role as a teacher and inspirer of others. CMG presents this lifetime achievement award to a single individual to recognize and encourage the same combination of technical excellence and professional contributions found in only an exceptional few.

The recipient is nominated and chosen by the past winners, and it has been granted to one person every year since 1974. This year I was very honored to be the recipient! It was announced on Monday 3rd December, at the start of the CMG07 conference. I gave an acceptance speech, that I have summarized into a blog post.

Thanks to everyone who has supported me over the years, reading the blogs, papers and books; coming to training classes; the engineers at Sun who fixed my bugs and added the features I lobbied for to make Solaris more measurable; and the friends, mentors and mentees that I have worked with for many years.

Cheers!
Adrian

Jobs at Netflix - One down, one to go...

I have been working on hiring for the last month or so. We found and hired one good match and had a bunch of near misses, so I have revised a reissued job description that is clearer about what we are looking for. I need one more senior Java/SQL developer with experience of large scale consumer web site development. We are looking for someone who has a lot of experience in product development, may have spent some time in management or team lead roles, but still "has their head in the code". We run small agile teams, you get to architect and code new product features yourself and deliver to the site every few weeks.

My group develops core personalization algorithms that are used to generate lists of candidate movies and filter them down to the top few to be shown in any particular block on the site. We work on the pages grouped under the main "Browse DVDs" tab at www.netflix.com. We collaborate with a sister group that collects and predicts star ratings and runs the Netflix Prize, they work on pages under the "Movie's You'll Love" tab.

The business logic is written in Java, pulling its data out of Oracle and via middle tier services. The front end presentation layer (javascript/CSS etc.) work is done by engineers working for Bill Scott, Director of UI Engineering, who used to work at Yahoo! where he was chief AJAX evangelist. He is also hiring....

Development is very rapid, agile and iterative. Features go through rigorous statistical A/B testing, and have to show a significant benefit before all the users get to see them. There is a great deal of freedom to try out ideas and an extremely analytical approach to picking the winners.

Netflix is also something of a social experiment, the company culture is unique and very employee oriented. There is minimal process, a huge reliance on personal judgement, and zero tolerance for antisocial behaviors. What would normally be a fat binder of HR policies has been summarized into a few lines: "Act in Netflix's best interest" and the vacation policy is "take some". If you don't have the personal judgement to do the right thing, we say goodbye...

Web Engineering consists of a relatively small number of senior people working very efficiently and productively. We like to hire the best "stars" we can find and build up "bench strength" like a championship winning sports team. We are looking for some more star talent...

Here is the opening: http://jobs.netflix.com/DetailFlix.asp?flix1916 

I'm hiring one engineer at this time - apply at the above link or find me in LinkedIn or FaceBook to discuss.

Parallel Application Performance and Tuning with OpenMP

I just had dinner with my old friend Ruud van der Pas, who works for Sun's developer tools organization. He is a world class expert in making parallelized applications scale, and recently gave a talk with a very amusing performance conversation.

Ruud is an advocate and expert in OpenMP, and has just published an excellent book on the subject. "Using OpenMP". Published by MIT Press. The work was done together with Barbara Chapman and Gabriele Jost. The ISBN numbers are ISBN-10: 0-262-53302-2 and ISBN-13: 978-0-262-53302-7.

Even if you aren't using OpenMP, the concepts for performance and scaling are the same for everyone...

Computer Measurement Group - CMG07 - San Diego - December 2-7

As usual I will be presenting at CMG07. I have attended every year since 1994, and its the place where I keep track of new developments in the performance management tools industry, as well as learning and sharing with my peers.

CMG moves around the US, last year it was in Reno NV, before that in Orlando FL.

On Sunday 2nd December there is a workshop day. I'm co-presenting (with Mario Jauvin) a half day tutorial on Capacity Planning and Performance Monitoring with Free Tools (for the third year). We updated our slides a bit more this year to track the latest versions of the free tools.

During the week I'm giving a new paper on Millicomputing, that introduces the concepts I have been discussing on my Millicomputing blog, and talks about some of the management challenges of managing large numbers of small computers.

I'm also giving a half day training class on Unix/Linux Performance Management, which is based on the materials I developed over many years in my books and talks. It is still somewhat biased towards Solaris since I know that better, but I have been adding more on Linux and AIX as that is the world I currently live in at work (Netflix runs RedHat Linux and AIX on IBM P-series) and in my Millicomputing sideline (I run Ubuntu on my development host and linux 2.6 on my gumstix).

The conference contains a wide range of content, there is:

• a large vendor exhibition showing the latest tools, vendor presentations and user group meetings to talk about their next steps.
  
• everything from beginner level introductions to deep mathematical theory.
• coverage of networking, storage, CPUs, application level performance.
  
• discussion of Linux, Solaris, AIX, Windows, Mainframe and all combinations
• panel discussions on virtualization, service oriented architectures, scaling large sites

I hope to see some of my readers there. I used to give training classes for Sun on a regular basis, and this is now the main venue for me to present updates versions of those classes.

First look at Android and implications for the Homebrew MyPhone

Google's new Android SDK is a lot more than a Linux distribution, its a complete and documented application framework with developer tools support. The primary language for application development is Java, with a special runtime called the Dalvik virtual machine that is designed to run efficiently on a battery powered device.

The SDK is 55MB, there is available Eclipse support.

Sample applications include Google Maps, mail, webkit based browser and the usual Phone and PIM stuff. The user interface seems to be based on a new set of Java classes, but there is also direct access to the display in C/C++ using SGL for 2D and OpenGL ES for 3D, and audio and video players for things like mp3 and H.264.

From a first pass through the docs, a lot of work has gone into this already. It looks as if it should be portable
to a homebrew phone without much difficulty, and from then on, we are likely to be leveraging a much larger community than the other open alternatives. It appears that the current state of the OpenMoko project is that they have made some progress and have a somewhat usable toolkit written in Python, but that C/C++ development based on GTK+ is painful, and documentation is scarce. It looks to me as if Android has taken the same basic inspiration as OpenMoko, but with a much more professional execution, and with a Java based developer platform.

We will be discussing this at the Homebrew Mobile Phone club meeting on Wednesday. We are holding the meeting at Mozilla's offices and will also be discussing their work on a mobile version of the Mozilla browser.

Leopard review, iPhone 1.1.2 release and the Google Phone

There is an excellent in depth review of Leopard at Ars Technica, it talks about the Core Animation feature which seems like a very powerful way of specifying device independent animation effects that will work on a desktop or (eventually) on an iPhone.

Now that Apple has got Leopard out, their next move is the UK launch of the iPhone on November 9th, and as I have mentioned before, there is some additional localization needed for this and it appears that this is going to be the main feature of iPhone 1.1.2 release. The UK press is starting to get its hands on the UK product, which is running 1.1.2. The main features are a fairly complete set of worldwide localized languages, a closing of the hole that is currently being used for hacking into the iPhone, and some local features related to the UK deal with O2 and The Cloud for WiFi access, and the UK iTunes music store. There doesn't seem to be any new applications in this release.

There has been a lot of speculation about Google releasing a set of phone applications or perhaps a complete phone OS. They have been working closely with Apple, so there may be some additional/optional iPhone applications coming directly from Google. However, there is currently no support for Java on the iPhone, and some of what Google has been doing is building a suite of Java based mobile applications, so this could be seen as a broadening of Google's mobile support. If Google decides to support a fairly generic mobile Linux (like OpenEmbedded) then it opens up their applications to OpenMoko and the homebrew mobile club's myPhone. What we have today from Google is three mobile applications:

Google Maps on the iPhone (which is probably a native re-write in Objective-C). Google Maps for everyone else (which is written in Java), and I think Google Maps for PalmOS also appears to be a native application. I've used this on Treo, Blackberry and iPhone. Its obviously based on the same backend web services but the user interaction is a little different on each.

Gmail reader, which I have used on Treo and Blackberry appears to be a Java based client. I'd like to see this on the iPhone, so I'm hopeful for a full function iPhone Gmail client.

YouTube viewer on iPhone. This looks more like an Apple application, running against the Google/YouTube web services, and required coordination to get the encoding standardized on H.264. It also supports the AppleTV. I suspect that Google will have a YouTube viewer for other platforms, probably written in Java. It could also be an mp3/music player.

The web browser is a problem that the iPhone has solved, but everyone else has a very fragmented approach, caused in part by lack of memory and CPU power on most phones. There are some open source projects that have got going recently to develop Mobile Mozilla, and there are more mature products like Opera. If Google has been working on its own mobile web browser, then they have been keeping very quiet about it.

Google's office suite is based on AJAX front ends and a back end set of web services that handle presentations, documents and spreadsheets. A mobile front end application could make these much more usable, and I would suspect an iPhone version and/or a more generic Java version could be in the works. The main problem with Google's existing online office suite is that it can't be used when there is no data connection. It would be much more useful to allow some subset of useful operations to work while disconnected.

iPhone SDK, Leopard and 1.1.2

As I suspected, Apple plans to release an SDK for native apps on the iPhone next year. The official word is on Apple's Hot News site. They appear to have pre-announced the SDK which is likely to have its official announcement at MacWorld in January, and they say it will ship in February. I'm sure that this was the plan all along, and the developer frenzy has managed to flush out a pre-announcement to give some guidance to the market, which I think is a good move at this point.

There has also been speculation that the iPhone "runs Leopard". I think that the truth is likely to be a bit more subtle than that. There has to be a master code base for MacOSX, and efficient engineering management practices should try to minimize branches in that code base. The master code base has a release branch called "Leopard" and another one called "iPhone", but the differences between them should be as small as possible. Bug fixes and features get checked back into the master code base whenever possible. So the work put into Leopard is leveraged for the iPhone, and we should see some of the feature set that was announced for Leopard transfer to the iPhone.

I think there is a good chance that Apple will release iPhone 1.1.2 immediately after they release Leopard, and that some Leopard features will be included. One obvious feature would be localized versions of Mail and other apps for the iPhone and iPod Touch, bringing the iPod Touch more in line with the iPhone's application set. There is a new and upgraded iChat in Leopard, and it would also make sense to release this for the iPhone/Touch.

I also noticed that the DTrace analysis/monitoring framework from Solaris 10 is included in Leopard. This is very cool, it provides a huge upgrade in observability for performance analysis of both desktop and server applications.

WWJD - What Would Jobs Do...

Somewhere in Apple there is a product team with a plan. We now have observed several steps unfolding in this plan, so I'm going to put myself in their place and try to make sense of the plan so far and where it could be going.

I have *no inside knowledge* of anything going on at Apple, this is all conjecture, OK?

Product teams have four big things that can be traded off against each other to some extent.

1. The release date
2. The available resources
3. The feature set at launch
4. The product quality at launch

For the iPhone, they waited until they had working prototypes in January, and set a release date for June that they kept to. The product quality at Apple has to be very high, and I think they did a great job with the iPhone at launch. However, products always take longer than planned, and we know that Apple delayed the launch of MacOSX Leopard so that they could put extra resources into the iPhone, so it was probably slipping at that point. The one thing that I think they were able to trade down to get a high quality product on the release date was the feature set. This is why I regard the iPhone as an unfinished product. To get a high quality product, you can simplify testing by having fewer features and fewer options in those features. This is the Minimum Marketable Features (MMF) approach I have mentioned before. Some simplifications at launch:

• US Market Only - no internationalization
• Basic camera support, no video capture (not even demo'd at the launch event)
• Relatively few preferences and options

Apple appeared to schedule bugfix and minor features releases shortly after launch (1.0.1 and 1.0.2), while concentrating development and testing to include internationalization for the worldwide rollout of the sister product, the iPod Touch and the international iPhone with 1.1.1. The Starbucks feature of the iPhone music store has its own roll-out issues, so it makes sense to add that feature as a major update rather than on initial launch.

I expect that Apple thought they had locked down the phone well enough in its first release and were surprised by the speed and extent of application and SIM based hacking. Adding digital signatures to prevent hacking in the initial launch version would have also made the final stages of debug and testing harder, so it makes sense that they didn't do it. Once Apple realized that they needed to lock down the phone to take back control of their product feature rollout plan, they  had to find extra resources to develop and test the extra security features. This had to steal resources from something else, so I expect that some features originally planned for 1.0.2 and 1.1.1 have been delayed.

Opening up a product for developers is a lot of work if you do it right. There are developer specific tools, APIs that need to be stabilized, extended and documented; training classes and developer support organizations to put together. We already know that Apple was pulling resources from Leopard development just to get the iPhone out of the door. We also know that Apple already has a developer support organization, but that organization already has its hands full trying to get Leopard out of the door as a high quality product with a lot of new features that need to be documented etc. Leopard ships this month, so Apple's developer support organization should now have some spare resources to focus its attention on doing the work required to do proper high quality iPhone developer features and documentation. Based on this analysis, the earliest I would expect to announce developer support is MacWorld in January, and that plan could have been set over a year ago. I also think Apple underestimated the demand from iPhone developers, but had no spare resources to do anything about it other than emphasizing Safari based applications.

The iPod Touch has a smaller set of applications that have been fully internationalized. The extra applications on the iPhone are the US specific ones. I still think this is a transitional problem, and I expect that a minor update in October or November will upgrade the iPhone to have more extensive internationalized support, and that the non-phone-specific applications will all end up on the iPod Touch as well. If I was planning the release calendar, I would bring out one new application in each release, along with a set of less exciting upgrades like internationalization.

There are three empty slots in the iPhone user interface. I think they will all be filled by the time we get to January's release. One will be the Finder, so that users can store and browse files and applications, one could be an iWork launcher to highlight some of Apple's own business oriented applications, and January's icon could be related to installing and managing third party applications, hopefully including Flash and Java.

Its possible that iChat could merge with the SMS tool, and I would expect video support to be a feature of the camera tool. I would like a terminal with ssh support, but that's not mainstream enough to get its own icon....

Of course, Steve calls the shots, but what do you think the three remaining icons will be used for? Let's collect some suggestions in the comments.

iPhone 1.1.1 and the iBrick

This blog just got mentioned by a Wired News posting about the iPhone. Other than some random spellings of my name I think it picked up on my points quite well.

I haven't hacked my iPhone, and it upgraded to 1.1.1 cleanly. The new features and upgrades are welcome, but the upgrade process didn't clear out the ever increasing amount of "Other Memory". I had resorted to removing videos from my iPhone to make space and avoid out of memory errors. You can see 1.6GB in the display below.



I went through the settings, cleared out deleted mail and safari history and it made no difference. I deleted all the content and it made no difference, so I re-loaded the iPhone from scratch, and it dropped to 14MB after I had put my content back.



There are some posts on the iPhone support forums about this problem, but no mention of an official fix from Apple. One user claims that it could be related to Google Maps, which I use quite often. I'm going to watch this "other memory" usage to try to figure out which apps are increasing it.

I haven't seen a clear explanation of what is going on inside the iPhone that makes hacking applications very different from hacking the carrier, and from my experience building myPhone I can hopefully clarify this issue.

There are two CPUs inside just about all phones of any type. One is part of a GSM modem module that interfaces to the phone network, the other is the application CPU that runs the user interface (e.g. OSX on the iPhone). For the Gumstix Goliath a Siemens MC75 module interfaces to the SIM card, and the GSM antenna directly, and has USB, serial and audio interfaces. The way all these GSM modems are managed is by sending it command sequences that would be familiar to any old-timers, its a derivative of the Hayes AT command set. Inside the iPhone is a similar module. The GSM modem has a low speed low power CPU with its own flash memory. The service providers don't like having people mess with this firmware, because it controls the wireless signaling, power levels and cellular communication protocol directly, and they have to be certified for connection to the GSM network. For our homebrew phones we buy Telit or Siemens modules off the shelf and don't mess with their internals, and they just work. Thats why we can build usable homebrew phones.

Since this firmware interfaces directly with the SIM card, it can contain code that locks the phone to a particular service provider. The iPhone's GSM modem firmware is locked to AT&T. In order to modify this firmware the unlock applications such as iPhoneSIMFree had to take advantage of a debug feature or security flaw to load modified firmware into the GSM modem. Since the GSM modem is a standard component, hackers were probably able to find out its supplier and part number and get hold of a lot of documentation on how it works.

With the 1.1.1 release it appears that Apple closed off some security holes that were being used to hack into the iPhone, but also digitally signed the code inside the phone. This means that any attempt to modify OSX or the GSM firmware breaks the signature, and since the CPUs have hardware support for digital signatures, its extremely difficult to get around this form of protection. Its now several weeks since 1.1.1 was released on the iPhone and iPod Touch and it hasn't been broken into yet. The hackers have to find and exploit a completely new security flaw to get into the application CPU first, then find a way into the GSM modem. It is then easy for Apple to figure out how they are getting in, and fix it in their next release. With Apple releasing monthly updates, there isn't a viable business proposition based on hacking iPhones.

I still think Apple has a lot of unreleased iPhone software in the pipeline and will open up to developers when it has pushed out a few more updates. I think the ideal audience for an iPhone developer launch is MacWorld San Francisco in January. The highest density of startups and developers is in the Bay Area, and its the anniversary of the iPhone launch. Thats as near as I'm getting to a prediction....

In the meantime, the developers who want something to do could join the OpenMoko project, help build applications for it and maybe even port the Linux based OpenMoko to the iPhone hardware. Thats what I will be working on when I get myPhone up and running....

What happened to the iPhone, and what comes next...

After the launch event last January, there was a lot of repetitive and shortsighted commentary about the iPhone, and I wrote a blog post about the way I thought Apple's strategy made sense and what the next few moves would be.

I think I got it broadly right, their strategy did make sense, and still makes sense to me. The iPhone was launched with features designed for the consumer marketplace, and redefined the smartphone category, ignoring the pundits who said it couldn't succeed as a smartphone without a business oriented feature set, it outsold all the other smartphones by an order of magnitude, and the competition is now scrambling to come up with products that bear any kind of comparison with the iPhone features and ease of use. The hardware is relatively easy to duplicate, but the software is years ahead of anything else.

One prediction I made was that Apple would release an iPod version this summer, and that just came true, with the iPod Touch. The hardware omits the phone features, adds more flash memory and uses a slightly smaller form factor, but it completely leverages the software investment which is the most important part of the iPhone. This WiFi based iPod is also completely internationalized, which is its primary software advance on the iPhone. It's also clear to me why the software updates for the iPhone haven't delivered much yet, the team was working hard on the iPod Touch, and it takes a lot more work than you may expect to internationalize and test a worldwide product. This foundation also sets up the iPhone for its own worldwide rollout. The iPod Touch will create clear consumer demand for the iPhone around the world that will get better deals for Apple as it works its way around the world's Telcos.

The iPod Touch omits Google Maps, Stock Quotes etc, which are USA market specific. I expect that when you sync an iPod Touch in a specific country Apple will install whatever unique applications are available in that country, so US consumers could get a quick upgrade to match the iPhone application set. The Mail application is also missing, while it could be provided in a generic form, the iPhone version is setup to make it easy to connect to US based mail providers, so it could be part of a localized application bundle.

Another prediction I made was that a 3G iPhone would follow for the European market. The cut in price of the original iPhone creates an empty price-point at $599, which could be filled by a 3G capable iPhone with 16GB of flash and possibly GPS. Some commentators have suggested that this could be announced at Apple Expo in Paris on Sept 25th, and I think that makes sense. The 3G phone could be thicker to accommodate a larger battery, in the same way that the iPod models vary thickness in the product range.

The iPhone is a *platform* which is in its infancy. I'm looking forward to a lot of software upgrades which will broaden its market appeal. As an unfinished product, it also makes complete sense to me that there is no encouragement for developers. For many platform based products (from Windows to eBay) the developers go into business to extend the feature set, but the platform vendor ends up competing with their developer community as they bundle more features into the platform. It makes sense for Apple to release the bulk of the applications that they are working on in-house and do deals behind closed doors with major vendors such as Google (and I suspect Adobe and even Microsoft), where they can build a coherent set of features into a roadmap that completes the basic platform. *Then* open up the platform to the developer community to innovate from a much richer baseline.

The current hacker based developer community knows it is living on the edge, so expects to be disrupted each time Apple releases something. I'm sure they will hack the iPod Touch within a few days, but the key for Apple is to carry this initial enthusiasm into a real developer program, while keeping the number of hacked systems to a minimum. One way that Apple could make life difficult for the hackers would be to release software updates very frequently.

Some more predictions - what would a completed product look like?

1) Flash and Silverlight - I'm sure that Adobe has come under intense pressure from all sides to get Flash onto the iPhone. It's a matter of time. We are also seeing Microsoft take on Adobe with Silverlight, and if Microsoft could establish Silverlight as a viable platform on the iPhone, then they would also be in a better position for Windows Mobile platforms and any future Zune based iPhone competitors. Either way, this would provide a much more open way (compared to the bundled YouTube application) to stream animation and video to the iPhone and iPod Touch from any web site.

2) Camera - the current iPhone camera looks barely finished. The viewfinder ripples if you move, has few features and takes a long time to grab a picture. Its clearly a software based camera, and the ARM based hardware used in the iPhone usually contains a lot of accelerators for image and video capture and compression which do not appear to be in use in the current version. My guess is that a future software upgrade will move to a hardware accelerated camera with video capture support and features like digital zoom.

3) Instant Messaging/VOIP/Video - current solutions use a web page or hacked application to do call-back to the iPhone over the regular phone network. This requires centralized hardware resources that cost real money. The free way to do chat, voice and video over the network is on a true peer to peer basis, and I expect Apple to produce iChat for the iPhone, and eventually perhaps iChat AV to include video. A call from an Apple laptop or desktop system with a camera could show up as video on any iPhone, but a new iPhone model with a forward facing camera would be needed to act as a true portable two-way video phone. I also suspect that a faster CPU would be needed so this is likely to be a longer term prospect. It should also be possible for developers to port other messaging applications such as Skype.

4) Business Applications - the most requested being Microsoft OutLook/Exchange support at the same level as we have come to expect from RIM Blackberry's. I don't think full integration will come directly from Apple, but there is likely to be a food fight amongst third party developers to win this market. Business orientated smartphones are going to suffer. The main casualty will likely be the Palm Treo in the short term, and the RIM Blackberry and the Windows Mobile based systems in the longer term. I doubt that any of these players will be able to get their platform software into the same space as Apple fast enough to hold onto this market. If RIM is smart, they will do an iPhone port of their application and hold onto their dominant position with the backend infrastructure for corporate messaging, while gradually losing ground to iPhones in the handheld device space.

In January I said I would wait before I got an iPhone, but it was too good, I wanted one, and my wife bought me one when they first came out. It redefines the market, and the US based phone market was ripe for a serious upgrade. I also have a work-provided Verizon based Blackberry and I carry both. When better business oriented email and calendar integration arrives, I'll return the Blackberry.

Some of you may be wondering about my homebrew phone. Its still a work in progress, and I expect to have it up and running shortly. It took a few turns to get everything on the Gumstx Goliath board working properly.

Gumstix Goliath - myPhone system board

The Gumstix Goliath open phone hardware is now available for order. There are versions with and without the Global Positioning Systems (GPS) support.

Features:	GSM/GPRS/EDGE, GPS, analog audio in/out, LCD, touchscreen controller, USB host port, 3D-accelerometer, NiMH battery charger (batteries not included) Provides full- speed USB access to the Siemens MC75 GPRS/EDGE module, the u-blox NEO-4S GPS module and one external USB device.
Connection:	60-pin Hirose connector to verdex motherboard only
Dimensions:	105.5mm x 67.3mm

I've placed my order (I already have a Gumstix Verdex CPU board), and I'm still working on myPhone case design for this board and its 4.3" touchscreen LCD. Unlike the iPhone, the hardware and software is completely open, you can do anything you like with it, and thats also the problem, the software is extremely basic at the moment. However, its about the journey, not the destination, I'm going to have a lot of fun as we build this device.

iPhone, the good the bad and no ugly bits

I got my iPhone in the mail after 10 days, beating the 2-4 weeks delivery estimate. I've used it for a week and have some opinions I'd like to share.

The biggest suprise is just how good video looks. Everyone I show it to goes "wow". This is something you have to see in person. The video quality for "video iPod" mode and for "WiFi YouTube" mode is basically perfect. When running on EDGE, YouTube downloads a lower quality version of the video that has visible artifacts, and looks more like typical video on a PC quality (e.g. similar to my Slingbox). In any case this is a seriously good video player, and I happily watched several complete TV shows on it.

The built-in speaker is also surprisingly good, considering the limitations of a mobile device. I left the iPhone lying on the table and used it as background music.

The user interface is easy, no-one has any problem using it immediately, and typing on the keyboard is much easier and more accurate than I expected.

My main disappointment is with Email. It works, but I've got used to the Gmail client application on the Treo and Blackberry. The generic pop client just doesn't work well enough since it doesn't understand archiving and threading, and the safari browser version is inconvenient and slow when using EDGE. I really hope that the Gmail application is in the works for a future update, it seems likely, given the relationship between Google and Apple, and the Google Maps application that is already included.

For use at work, it can't yet replace my Blackberry. The iPhone VPN doesn't work with the VPN where I work. The LEAP security mode we use for wireless is also unsupported, although it works fine on my MacOS laptop. Using a simpler guest wireless setup I did see over 3Mbits/s in a speed test which is very impressive for a mobile device.

I can use Safari to get at Outlook Web Access, and http redirect based secure access to internal web pages worked fine. I have had no crashes or problems with Safari, trying it out on several sites.

For Netflix, heavy use of "mouse-over" in the user interface design is an issue since the touch screen doesn't generate mouse-over events. Although Safari on iPhone supports JavaScript, some work is needed to figure out how to use JavaScript effectively given the different input behaviors. In the meantime, the basic operations of browsing for movies and adding to the queue work fine, but rating a movie doesn't work since the star-setting method uses mouse-over.

For detailed insight into the iPhone, go read RoughlyDrafted, its by far the most thoughtful and interesting blog on the subject.

Global Classified Ads - Kijiji

eBay recently took the final step with their Kijiji classified advertising brand of launching it in the USA. There has been quite a lot of commentary, most of it fails to point out what matters about Kijiji. eBay also holds a small stake in Craigslist, which also has sites around the world, but there is one very significant difference that no-one seems to have commented on:

Go to http://rome.craigslist.org - there are a handful of entries, but the most important thing is that the site is still in English. Check out http://paris.craigslist.org - same thing.

Now try http://roma.kijiji.it and http://paris.kijiji.fr - notice the difference? Its localized and there are a decent number of items listed. Now see how many times you can figure out http://[city].kijiji.[country] (remember to use the local name) and see just how many cities and languages there are. Remember to try http://shanghai.kijiji.cn too... There are also a few countries where eBay bought a local brand - http://www.gumtree.com for the UK, and http://www.loquo.com for Spain.

So its clear that eBay has spent the last few years building up a global localized classified advertising business that has wide coverage and a lot of listings. They have had English-speaking Canada for a while (they also have French Canadian), so adding the USA must have been a small amount of extra work and cost, and there was a well crafted and debugged product ready to roll.

It may take a while to catch on in the USA, but its not likely to go away. Its free for the users, and can generate revenue by using services like banner ads and Google Adsense to monetize the page views. Kijiji is already doing quite nicely in my opinion....

[Disclosure: I used to work for eBay, I know some of the people who created Kijiji, but I have had nothing to do with Kijiji since it launched in 2005, and this is all public information]

Must. Buy. iPhone. Must. Buy. iPhone....

OK, I give in. Its Apple at their best, insanely great, detailed craftsmanship and "it just works" usability. However I didn't want to get in line at a store, so I logged in to the Apple online store and made sure my one-click ordering was setup right and enabled, and at 6pm when the store re-opened I got my order in as fast as the site would refresh. It was a little slow, but as far as I could tell, the store was up and running and handled the peak load, so some extra congratulations to the team that runs the store. Delivery was stated to be 2-4 weeks, I should be near the front of the queue, so we will see what happens.

What about the open phones? Well, OpenMoko is running late, but is just about to ship developer phones for real, however the version that will have WiFi is their second revision that is months away, and their software is still being developed. However Gumstix are also very close to getting their new "Goliath" phone kit out, and that is a much more flexible design, where I can add WiFi etc. without replacing anything else, and I can wrap my own case design around the 4.3" LCD. The software is rudimentary, but we intend to port the OpenMoko packages to the Gumstix based "myPhone", and tinker with it to do whatever we want. In the meantime, I will soon have a phone that "just works" better than anything else....

Updates: AdSense, FON and Millicomputing

The Ads are making a bit more sense now...

There is a global commercial WiFi service that has good coverage in places like Hotels and Airports called Boingo. They aggregate services that are not branded under their own name, but if you have a Boingo account you can use the service. Last month they announced a deal to include FON in their network, more than doubling the number of access points to a few hundred thousand. This month they announced an annual flat rate plan for $39 per month. Very cool! Global WiFi access at a low flat rate.

I'm posting a series of updates at Millicomputing.com that describe a detailed design for an Enterprise server Millicluster. Its a radical improvement in terms of performance per watt and performance per dollar for some kinds of applications.

What the iPhone doesn't do, why not, and why I still want one...

There is a very nice posting on the iPhone at RoughlyDrafted, taking a thoughtful look at all the coverage, and pointing out double standards in many of the negative commentators who are putting down the iPhone for not being "enterprise ready".

Back in January, I wrote a summary of thoughts on the iPhone from the perspective of typical product development practices. That post got a lot of traffic, it seemed to be very popular in Germany, where there is a great tradition of quality product design (I can't think of a German designed product that isn't well engineered).

Overall, with the additional advance information that has been released so far I don't think I was too far off the mark. I did expect that support for Adobe Flash and Java would be included, but they both appear to be missing. This is going to be the biggest pain, as there are a lot of web sites that use Flash or Java applets. Given time I expect these issues will be addressed.

I talked about Minimum Marketable Features (MMF) which are described in a book called Software by Numbers, written by my good friend Mark Denne. Its a philosophy of development that optimizes product features by releases over time. Many of the iPhone's shortcomings are explained by an initial focus on the needs of people who have never used a "smartphone" before, (rather than a focus on the business market) and a close integration with other Apple products that support an "insanely great" combination of product features.

For integration with non-Apple products, there is a two pronged strategy. One is to provide basic file viewers for Microsoft Word and Excel documents that are common email attachments. The other approach is to leverage Google's web based application suite within the Safari browser, which now includes Word, Excel and Powerpoint, Gmail etc, and also integrate with the Google maps application on the iPhone itself.

For the question of whether the iPhone can be used in corporate environments, there are two issues - virtual private network (VPN) access to get at internal web sites, and email integration with Microsoft Exchange. There have been some hints that the standard MacOSX VPN functionality is included, and Apple states that syncing is supported with Outlook/Entourage contacts and calendars. For actual Exchange email access, the Exchange server would have to support IMAP or POP (most aren't setup this way) or the Outlook Web Access client could be used in the Safari browser on the iPhone.

Safari is key. It is not only the way to manipulate documents and corporate email, its also the designated "API" for new iPhone applications to be developed. Apple stated that Safari will have access to many of the internal functions of the iPhone, so that the primary programming language for developers becomes Javascript. For all the people with existing mobile applications this is a pain, as they want to have J2ME to get them ported quickly. However, the iPhone is a huge leap forward in user interfaces, and to provide the consistent look and feel that magnifies the ease of use, Apple are enforcing Safari as the only framework for extending the iPhone. This makes perfect sense as a product strategy. It has upset many developers, but the laser-like focus on ease of use greatly increases the iPhone's perceived value and total addressable market. More users will be willing to pay more, increasing Apple's sales. Later on, perhaps Apple will focus more on the "traditional" smartphone market, but for now they are simply going to redefine what that market is.

How many people have both iPods and cellphones in their pockets? This fall, how many students will turn up at college with an iPhone in their pocket?

My work machines are a MacBook Pro and a Blackberry. I'm building the homebrew myPhone design (more on that later), but I think I'm getting tempted by the user interface, I want an iPhone....

FON: World WiFi Domination?

Now if there ever was a company working on world domination, it would be FON. They aim to build a global network of Wifi hotspots by selling subsidized routers that have custom firmware. The routers have a private secure WPA based network for you to use, and a public network that has a signup page, for anyone else to use. There is also a control that limits how much bandwidth the public can use. They have a few hundred thousand users, and are now the largest WiFi network by far, but the location of their base stations is much more random and less useful than T-Mobile HotSpots.

The incentive in the short term is that you can get a high quality router at a very low price. FON has been giving routers away to seed markets and to surround Starbucks stores. The incentive in the longer term is that if you have a FON router at home, you can use any FON router you find on your travels for free.

I've had a FON router for some time. I met CEO Martin Varsavsky when he gave a talk at eBay last year, and got one of the initial Linksys units with custom firmware. When their own hardware solution came out it had a much nicer feature set, so I upgraded to La Fonera as they call it. I've been using it as my primary home router for about six months now, and its an excellent design. Its been much more reliable than the other low cost routers I've owned (Netgear and Linksys) and works as well as the much more expensive Apple Airport Express that I've had for a long time.

FON just released a booster antenna "La Fontenna", it is designed to give 6.5dB of extra signal strength and project further out than the standard antenna. The idea is that you mount it on the window facing the street, and it covers a larger area. I just got one, and while it seems to work fine, it doesn't appear to be giving me as much signal strength or range as I expected. I wandered around outside with my MacBook using iStumbler's WiFi monitoring widget to track signal strength, and it showed a bit more gain than my Airport. I think I need to experiment a bit more with positioning La Fontenna.

So next time you need a router, don't just buy a normal one, join Fon and support world domination!

Ads that don't make sense

For some reason the ad bar on this site has a fixation about the b-word. I just removed the word bl*g from the template in two places, and I'm avoiding saying it in this posting. I really think Google should be able to find something more interesting to show ads about, so lets see if this makes any difference....

Jobs at Netflix: Capacity/Performance and Storage Admin

Netflix is a great company to work for, there are lots of clever friendly and happy people here.
I just hired an engineer for my own group, but we have a lot more openings:

We are looking for a performance and scalability oriented Java/Oracle developer
Senior Performance and Scalability Engineer
http://jobs.netflix.com/DetailFlix.asp?flix1651

My good friend Ed Wustenhof just joined Netflix to manage the System Administrator group, he's got two open positions, a storage oriented one:

Sr Unix Administrator - Storage Focus
http://jobs.netflix.com/DetailFlix.asp?flix1699

and a Capacity Planning and Performance Tuning one that I've copied the full info for below.

Senior Unix Administrator - Capacity Planning & Performance Tuning

http://jobs.netflix.com/DetailFlix.asp?flix1698

RESPONSIBILITIES

Design, Plan, Deploy and Support all aspects of Netflix's server and storage environment with a focus on Performance Tuning & Capacity Planning.

The SA team at Netflix is a crucial component to the success of the day to day operations of the company. We are a highly motivated and capable team that is seeking a new member that can take a key role in the provisioning and support of our unix/linux servers while ensuring performance stays within thresholds.

Being the second line of support and subject matter expert, you will take individual responsibility for resolving system, storage and application issues reported via the Netflix Operations Center. You will be expected to resolve tactical issues while providing structural solutions. Your area of expertise is performance tuning and capacity planning.

Part of the position will require travel to and from the data center (15-20min or ~10 miles), sometimes multiple times/week. On-call responsibilities, night and weekend projects are also to be expected.

The candidate will often be the technical lead on short term projects. They will be the Subject Matter Expert with its associated responsibility. Some level of project planning and coordination of resources is to be expected. This includes vendor and contractor coordination.

We are looking for someone with the following technical experience:
•Deep experience of Unix with at least one of the following operating systems: AIX, Linux or Solaris.
•Expert in capacity planning and performance tuning.
•Strong experience in web based environments and supporting such architectures such as Java, Tomcat, Apache and other web applications.
•Knowledge of IBM enterprise/midrange systems and storage virtualization is a plus.
•Strong coordination and analytic skills.
•Project planning experience and the ability to manage multiple projects simultaneously.
•10+ years IT experience with at least 8 years as System Administrator.
•BS or MS degree in Computer science or equivalent experience.

Exposure to the following highly desirable:
•Experience with Symantec/Veritas product like VxVM and Netbackup
•Strong expertise with SAN-based storage/application required.
•Large Scale (5Tbyte +) Storage solutions deployment experience.
•Unix shell and Perl programming.

Using Occam to Simulate p2p Web Services

About a year ago I gave a paper at an IEEE E-Commerce conference that described how I used the latest pi-calculus enhanced version of the old Inmos Transputer language Occam to build a very efficient massively threaded simulator for large scale p2p web services that ran on my laptop.

I blogged this at the time, but The paper itself is subject to the rules IEEE sets out for peer-reviewed conference papers and you need to be a member or pay a fee to get it from http://doi.ieeecomputersociety.org/10.1109/CEC-EEE.2006.81

However, I wanted to share this more widely and have just posted a pdf of the presentation slides for my IEEE paper. Here are the first few slides, and a description of the Tools section:

Simulation of Skype Peer-to-peer Web Services Choreography Using Occam-Pi

Abstract
Complex web services are very difficult to test and verify before deployment on a large scale.
A semantically equivalent in-memory simulation can be built using Occam-pi that runs up to 100,000 nodes about a million times faster than real life.
Rapid protocol development and detailed behavioral analysis using simulation supports the development of robust services that can scale.
The implementation of a simulator that models centralized web services and application to application messaging over the Skype Peer to Peer network is described.

Overview

• The Landscape
• The Problem
• The Tools
• The Solution
• Implementation
• Running the Simulation
• Conclusion
• References

The Tools – Pi Calculus

• WS-BPEL, SSDL and WS-CDL are based on Pi-Calculus
• Pi-Calculus is based on Communicating Sequential Processes (CSP)
• Pi-Calculus provides a formal model of parallel message based computing
• Occam-Pi
• The Occam language is based on CSP, and has been extended to add the Pi-Calculus extensions to form the Occam-Pi language.
• The primary implementation of this language is known as KROC, the Kent Re-targetable Occam Compiler
• KROC is freely available from the University of Kent at Canterbury, UK
• Runs on Intel architecture Linux, MacOS X, and Microsoft Windows/Cygwin platforms. Older versions exist for SPARC, PPC etc.

The Tools – Occam Language Constructs

• The constructs that are used in Web Services choreography map directly to Occam language constructs
• The Occam language has direct support for sequential, parallel and alternate processing blocks, complex protocols, and channel based communications
• Occam-Pi adds more dynamic constructs to the language
• Mobile channels - pass a channel end over another channel
• Mobile processes - suspend a process, pass it over a channel and resume it in a new context
• Dynamic process forking with barrier synchronization
• Rigorous Occam-Pi Compile-time Checks
• Processes or expressions are not allowed to have any “side effects”
• Syntax and usage of all protocols, data and constructs is checked
• Occam is designed to allow very comprehensive static analysis

The Tools – Occam Runtime Characteristics

• There is no need to use XML message encoding or namespaces
• since the compiler can check that a protocol is being communicated correctly
• At runtime, the Occam-Pi language is fast small compiled code
• with its own threading model, in a single process
• The Occam-Pi runtime detects and reports deadlock
• including the line number in the code at which each process was stalled
• Occam-Pi is very efficient
• All communication takes place in a single address space at memory bus speeds
• Basic process creation takes 20 nanoseconds on an 800MHz PC
• Basic channel communications takes 70 nanoseconds
• Compared to typical web services transactions over the internet these transactions are about a million times faster
• The language is also very compact, and one hundred thousand to one million threads can be created within a 2 GByte address space.

Works Cited

• [Cygwin] “GNU + Cygnus + Windows”, http://www.cygwin.com/
• [GraphML] “Graph Modeling Language “, http://graphml.graphdrawing.org/
• [Hoare78] C. A. R. Hoare. “Communicating Sequential Processes.” Communications of the ACM 1978.
• [Hoare85] C. A. R. Hoare. Communicating Sequential Processes. Prentice Hall, 1985.
• [KROC]. “Kent Retargetable Occam Compiler” http://www.cs.kent.ac.uk/projects/ofa/kroc/
• [Occam2] Inmos Ltd, Occam 2 Reference Manual. Prentice Hall, 1988.
• [Milner99] Robin Milner, “Communicating and Mobile Systems: the Pi-Calculus”, Cambridge University Press, 1999.
• [Skype]. “Skype Developers API” https://developer.skype.com
• [SSDL] Webber et al. “Asynchronous Messaging between Web Services using SSDL.” IEEE Internet Computing Vol10, No1 2006.
• [WS-BPEL]. “OASIS Web Services Business Process Execution Language” http://www.oasis-open.org
• [WS-CDL] “Web Services Choreography Description Language”, http://www.w3.org
• [yEd] “Graph visualization and editing tool”, http://www.yworks.com/

Aftermath is finally released...

After many years of listening to this song evolve from an initial concept to its final released form, its now time to share it with the world and find out what everyone else thinks of it. Its a moving reaction to the events and aftermath of 911, sounds like nothing else, and has an odd but catchy hook.

Its available on Fractal's site at fractal.bebo.com and via the iJigg widget below.

Orb Swarm Rolling Robot Video at Maker Faire

Update: this is part of the Orb Swarm project. They intend to build several orbs and take them to events like Burning Man.

Workplace Myths - Career Advice

Some very good career advice on Guy Kawasaki's blog today. The Nine Biggest Workplace Myths, by Penelope Trunk. I've spent a lot of time mentoring people and helping with career development and this is pretty much what I've been telling people for many years.

Its not easy to follow the advice, but it does give a clear vision of what to aim for, and how to know when you get there...

See you at Techshop at Maker Faire this weekend

I've been using Techshop to make cases for my homebrew phone on their 3D Printer, and I signed up for annual membership since I have a few other projects to work on (e.g. TIG welding some modifications to my Fiat 124 Abarth Rally replica :-)

I'm going to be at Maker Faire in San Mateo this weekend, helping explain the 3D printer to attendees at the Techshop pavillion, and hanging out with Matt on the SVHMPC booth. I'm on duty from 2pm onwards both Saturday and Sunday for Techshop so I should be easy to find and I'd be happy to chat about how to make mobile phone cases or millicomputing or whatever.

I'm hiring a senior engineer at Netflix

In my new role, I manage a small group and I have an open headcount for a senior engineer

The full job specification is at http://jobs.netflix.com/DetailFlix.asp?flix1642

Here's what we are looking for:

RESPONSIBILITIES:

• Designing and developing the Netflix web application and middle-tier architecture focused on personalization software.
  
• Full software life-cycle ownership of key strategic business logic from inception through production.
  
• Conceiving, articulating and implementing projects in coordination with technical and non-technical coworkers for product enhancement and improvement.

QUALIFICATIONS:

• Experience with high-traffic, highly scaling, multi-threaded software design.
  
• 7+ years of web application development experience. (Java a plus)
  
• Experience with Oracle or equivalent database system.
  
• Flexibility to shift priorities based on changing needs of the business.
  
• A Bachelors or Masters in Computer Science or equivalent engineering.

Characteristics of Success at Netflix:

• Strong customer orientation, both internally and externally.
  
• Strategic thinking both technically and business-wise.
  
• Ability to execute successfully on business strategy.
  
• Self motivated / self starter.
  
• Personal drive to achieve world class software development.
  
• Desire to work in a fast-paced, evolving, growing, dynamic environment.
  
• Love of technical challenges and the sense of pride it brings to solve them.
  
• Ability to work well in a cross functional environment.

Netflix and Silverlight and Joost

Very interesting demo of Microsoft Silverlight as a Netflix application from MIX07.

This seems like a nice, high performance multiplatform way to build web based user interfaces. It should give Flash some competition...

I'm handing back eBay's Dell D600 with Windows XP and getting a MacBook when I start at Netflix, it will be nice to have XP running only when I need it, and when the Silverlight version of Netflix web video ships, it will support Mac.

Joost has also released a new beta build, however I don't have a machine I can run it on during this transition. They don't support my old PowerBook G4 yet....

Jobs at Netflix

A few people have asked me if there are interesting jobs available at Netflix. I haven't started there yet, but they do seem to be looking for senior developers who have a clue about performance.

Here are some job descriptions, let me know if you apply to them...

http://jobs.netflix.com/DetailFlix.asp?flix1449

http://jobs.netflix.com/DetailFlix.asp?flix1576

http://jobs.netflix.com/DetailFlix.asp?flix1440

Leaving eBay to join Netflix

I recently gave notice to eBay and start at Netflix on May 7th. I've had a lot of fun working at eBay Research Labs, but I'm making a strategic move to a smaller company (about a tenth of the number of people as a technology organization) which makes it easier to take a broader role and develop skills and experience in new areas. I will be directing a few senior engineers who develop part of the Netflix web site as my primary role. Its an exciting new challenge!

Mobile Phones at Maker Faire

The Silicon Valley Homebrew Mobile Phone Club is going to be featured at Maker Faire this May 19/20th. One of my phone design pictures was used on the site :-)

I've also been continuing to develop the general purpose millicomputing concepts and have been documenting them on my companion blog. I have bought a Gumstix module for use in a phone, and I'm working on benchmarking it at the moment.

Load Average Differences Between Solaris and Linux

A lot of people monitor their servers using load average as the primary metric. Tools such as Ganglia colorize all the nodes in a cluster view using load average. However there are a few things that aren't well understood about the calculation and how it varies between Solaris and Linux.

For a detailed explanation of the algorithm behind the metric, Neil Gunther has posted a series of articles that show how Load Average is a time-decayed metric that reports the number of active processes on the system with a one, five and fifteen minute decay period.

The source of the number of active processes can be seen in vmstat as the first few columns, and this is where Solaris and Linux differ. For example, some Linux vmstat from a busy file server is shown below.

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 4 43      0  32384 2993312 3157696  0    0  6662  3579 11345 7445  7 65  0 27

The first two columns show the number of processes that are in the run queue waiting for CPU time and in the blocked queue waiting for disk I/O to complete. These metrics are calculated in a similar manner in both Linux and Solaris, but the difference is that the load average calculation is fed by just the "r" column for Solaris, and by the "r" plus the "b" column for Linux. This means that a Linux based file server that has many disks could be running quite happily from a CPU perspective but show a large load average.

The logic behind the load average metric is that it should be a kind of proxy for responsiveness on a system. To get a more scalable measure of responsiveness, it is common to divide the load average by the number of CPUs in the system, since more CPUs will take jobs off the run queue faster. For disk intensive workloads on Linux, it may also make sense to divide the load average by the number of active disks, but this is an awkward calculation to make.

It would be best to take r/CPU count and b/active disk count then average this combination with a time decay and give it a new name, maybe the "slowed average" would be good?

SEtoolkit and XEtoolkit releases

The SEtoolkit was developed in 1993 by Rich Pettit, and I used it as a way to prototype many new tools and ideas over the years. Its a Solaris specific performance tool scripting language that supports very rapid development of new tools. The SEtoolkit has been widely deployed as the Solaris collector for the popular system performance monitor Orca. Rich gave up development of the SEtoolkit a few years ago, put the code into open source under GPL, and its now available via sourceforge, where it is being maintained by Dagobert Michelsen. A bug in the SEtoolkit was causing it to crash when used with complex disk subsystems, and this has now been fixed in the SE3.4.1 release (April 10th, 2007).

Meanwhile, Rich has been trying to make a multi-platform (Solaris, Windows, Linux, FreeBSD, OSX, HP-UX, AIX) version of SE for a long time, and finally gave up trying to implement his own language, and based his latest development, the XEtoolkit, on Java 5. The first full release XEtoolkit 1.0 came out on April 15th, 2007. The code is released and supported under both open source and commercial licenses, by Rich's new company - Captive Metrics. The GPL license allows full free use of the provided tools, and development of new and derived tools that are also contributed to the community. The commercial license allows custom XEtoolkit development for proprietary tools, with a higher level of support.

The XEtoolkit 1.0 release doesn't support HP-UX or AIX, but AIX support is coming soon. I encourage you to try it out, give Rich some feedback and make it worth his while to continue. He's one of the very best programmers and performance tool architects I've ever met....