Refining a risk KPI dashboard in SharePoint

This is a continuation of my article from yesterday, where I showed how to take a preexisting SharePoint list and use it to create KPIs and from those a dashboard indicating a level of risk. The plus of the technique I used was that it was very easy to implement and required no changes to the list. But the algorithm was pretty naïve. It simply used the count of items (such as system changes or help desk tickets) in a given time period to indicate an arbitrary threat level.

Today while discussing it with my fellow director (I'm over app dev, he's over IT), we decided that it was in fact too simple, because not all changes are created equal. Some of our tickets are actually informational only and have no risk whatsoever ("Remember the new naming standard when standing up new servers.") Some changes have minimal impact ("I installed the Powershell feature on server foo.") And some are very wide-ranging and inherently risky ("I took the A/D domain into Windows 2008 mixed mode over the weekend.") Given that, we wanted to come up with a way to "weight" the list items and then have the dashboard use those weighted values to better indicate an overall risk level.

After some discussion we decided the requirements of the weighting scheme were:

• A "high" risk level item should immediately trigger a red stoplight for the day. In other words, if there is a high risk item performed, then at least the day's KPI should go to red.
• Lots of little changes can also end up adding up to a higher level of risk, just because a lot of variables, each seemingly low impact, are changing at once. So if a given number of low or medium items reached a given point score then the risk level would be raised.
• We felt that weekly and monthly risk levels were not "linear," i.e., they weren't gotten just by multiplying the daily limits by 7 and 30. The "change velocity" for a given day may be acceptable, but if the same number of changes were kept up over a week or a month then they represented a lot of change for that time period and deserved closer attention.

The following explains how I implemented the above. It presumes you read yesterday's article and isn't going to cover that ground again.

The first thing I did was modify the list by adding two columns. The first column was called Risk Level and I made it mandatory that the user had to fill it in. It is a "choice" column, which can be represented in SharePoint as a drop down. To create such a column:

1. Navigate to your list.
2. Click on Settings|List Settings.
3. Under Columns click on Create column.
4. Fill in a column name, such as Risk Level.
5. Choose Choice for the column type.
6. Fill in the Description field.
7. Click Yes on Require that this column contains information.
8. Enter your carriage return separated list of choices. I used Info - No Risk, Low, Medium and High.
9. Set a default value. I chose Medium. Most changes in production have some risk.
10. You can choose whether to add the column to the default view or not. I did.
11. Click OK.

Here's a screenshot:

[Click on all images to enlarge.]

So now we have a nice, constrained set of options for the user to choose from. But our KPIs are going to need numbers to work with for weighting. So I created another column, called Risk Factor. It is a calculated column whose value will be determined by the value in Risk Level. To create it:

1. Click on Create column again.
2. Fill in the Name field. I chose Risk Factor.
3. Choose Calculated as the type.
4. Fill in the description.
5. In the Formula field use something similar to the formula I give below.
6. Indicate that the data type returned from the formula will be a number. Set the number of decimal places to 0.
7. Choose whether to add it to the default view. I did not.
8. Click OK.

The formula I used relies on SharePoint's IF function:

=IF([Risk Level]="None - Info Only", 0, IF([Risk Level]="Low", 1, IF([Risk Level]="Medium", 2, IF([Risk Level]="High", 8, 0))))

Basically that assigns a risk weighting of 0 to informational-only items (and unknown items via the inner-most embedded "else" value), 1 to low impact items, 2 to medium and 8 to highly risky items.

Here is the screenshot:

Then I changed the views. Yesterday I said you didn't need to care about what columns were in the time-based views you used to show the last day, week and month, but now you do because the view is going to have to include the risk weight (Risk Factor) so that the KPI can work with it. Therefore I altered all the time-based views to include both Risk Factor and Risk Level (the latter was not strictly needed, but it helped me confirm that Risk Factor was getting set appropriately based on Risk Level's value). Under List Settings click on each view in order and perform the following:

1. Add Risk Factor and Risk Level as columns in the view.
2. Click OK.

Obligatory screenshot follows:

Now we're ready to change our KPIs to use the Risk Factor column. Go to your KPI list and for each KPI hit the down arrow to get the context menu and choose Edit Properties and perform the following:

1. Under Value Calculation change the radio button from Number of list items in the view to Calculation using all list items in the view.
2. From the drop down for calculation type choose Sum.
3. From the column drop down choose Risk Factor (in our list Risk Factor was the only numeric column, so it was the only column available here).
4. Leave Better values are set to lower.
5. Set your green light and warning level limits. See the table below for ours. Remember that these limits are for "less than," i.e., hitting the value changes the risk level.
6. Click OK.

Screenshot:

Here are the limits we chose to start with:

Risk Level	Green	Amber	Red
Past Day	<5	<8	>=8
Past Week	<19	<24	>=24
Past Month	<48	<60	>=60

Since we can easily change them at any time simply by editing the KPIs we're going to monitor for a while and tweak as we go. You don't want your dashboard to always be all green lights because that's misleading, and you don't want it to always be all red lights, either, because that just means everyone ignores them. So expect a bit of trial and error as you find the limits that realistically model your shop's risk level. We also decided that if an item was entered as low or medium impact and then actually ended up having a high impact after the change we are going to go and change the item's Risk Level after the fact so that it will be reflected in the dashboard for the day and more importantly the week and month KPIs.

We're almost done. Since we added those columns to the list, we have to populate the existing list items with them. I went to the list and chose the Action to edit it in datasheet view, which gives an Excel-like interface. I then quickly went through the rows and set the drop down for each item's Risk Level. The one warning I'd give about doing it this way is that if you have a lot of list items it's going to take someone a while to work through them all, so maybe you only want to make the changes to the last month's worth (since any before that won't show up on the KPIs anyway), and also if anyone has alerts set on the list to fire for each list item as it's changed they are going to get a lot of alerts. So you may want to remove alerts and then add them back after you're done.

Finally, here are the results on the dashboard on our department's home page:

I hope this will be useful to someone out there. We think it is going to be very helpful for us not just in monitoring risk on an ongoing basis, but also in scheduling when changes happen. For example, on a week that's already running amber or red we may choose to defer a major change until the following week, just to let everything stabilize first.

A simple risk KPI list in SharePoint

There are quite a few examples of how to set up "key performance indicators" (KPIs) in SharePoint, but I thought I'd give a quick little tutorial on how to use a SharePoint list that you may already have in existence, such as an IT change log (which we've recently started using) or help desk tickets, to put up a "risk dashboard" for your IS department's site. The thinking being that the more major system changes (or help desk tickets) there are occurring over a given period of time, the higher the risk that something is wrong or might go wrong. Consider this something like a "threat level" indicator.

There are four basic ways you can feed KPIs in SharePoint - from SharePoint lists, from a SQL Server Analysis Server OLAP database, from an Excel spreadsheet and from manually entered values. We'll deal with the first one in this example. I am assuming you have some SharePoint list on your site where every entry in the list documents your department's or organization's exposure to a "risk point" - major system changes, help desk tickets, customer complaints, whatever. We'll call it the "risks list" here. The good news is that you will not need to change that list at all to use the technique here, other than to add some new views to it.

The first thing you will need to set up on your site is a KPI list to hold your KPIs. That's a trivial exercise:

1. Site Actions|Create.
2. Choose KPI List from Custom Lists.
3. Fill in the Name and Description fields and choose whether to display the list on the Quick Launch and click Create.

The second step you will need to do is to create some new views on the "risks list." Each view will be for a different time window. For example, we're going to create views that show the new risks for the last day, week and month. Go to the "risks list" and do the following:

1. Settings|Create View.
2. Click on a view under Start from an existing view. It really doesn't matter which one.
3. Give the new view a meaningful name, like "Past Day" for the view to expose the last day's worth of risks. Don't make it the default view.
4. Under the Filter settings click on Show items only when the following is true:.
5. In the Show the items when column drop down choose a date column from the list - we had one called "Date," interestingly enough, although the built-in "Created" or "Modified" columns would work, too.
6. Then choose is greater than or equal to from the drop down.
7. Then fill in the formula field with [Today]-1. Important note: There are hits on the Internet saying that for that formula to work there can be no spaces around the minus sign.
8. Click OK to save the view and test that it does what you think it does.
9. Repeat for weekly and monthly views, changing the 1 in the formula to 7 and 30, respectively.

Again, the contents of what the views display aren't important, since all we're going to use the views for are to get counts of the items meeting their criteria. Here's a sample of how the Filter section should look for the "Past Day" view:

[Click all images to enlarge.]

Next, go to the KPI list you created and do the following:

1. Click on New.
2. Fill in the KPI's Name, Description and Comments fields.
3. Change the List URL field to point to the "risks list." There's a handy browse button to the right of the field to help you get the right URL.
4. Change the View drop down to pick the right view for the given KPI (e.g., the "Past Day" view you created above for the KPI that is going to measure the risk for the past day, etc.)
5. Under Value Calculation choose Number of list items in the view.
6. Under Status Icon Rules change the Better values are drop down to "lower."
7. Fill in the two goals fields for when to display the green light and when to display amber (red is automatically displayed when the amber limit is passed). Click OK.

Here's a sample of the relevant parts of the "Past Day" KPI item:

For now I set up our goals as follows:

Threat Level	Green	Amber	Red
Past Day	<= 2	<= 5	> 5
Past Week	<= 8	<= 20	> 20
Past Month	<=24	<=60	> 60

Those limits are just guestimates for now and can be changed easily enough by modifying the KPI properties. Note that they are not linear on purpose (i.e., the "Past Week" goals are not simply the "Past Day" goals multiplied by 7). My thinking was that while a couple of major system changes a day may not be out of the ordinary, if we were making that many a day for a whole week it would be, and if we were doing that many a day for a whole month it would be extraordinary. And again, I'm not saying the number of changes are "bad," simply that they elevate the level of risk exposure, so the KPIs are there to remind everyone when a lot has changed in the environment in a short period of time, in case suddenly the users start experiencing a lot of odd technical issues.

Here's how the KPI list looks after the three KPIs are created:

While you can certainly view the KPIs by navigating to the KPI list, the last thing to do is to put a KPI "dashboard" somewhere more noticeable so the whole department sees it. I put ours on the IS department's home page. To do that, simply:

1. Click on Site Actions|Edit Page.
2. Click on Add a Web Part in the appropriate zone on the page.
3. Check the Key Performance Indicators Web part under the Dashboard section.
4. Click on the Open the tool pane link in the Web part once it is on the page.
5. Fill in the Indicator List field by pointing to your KPI list (use the handy browse button to the right of the field).
6. Fill in the other fields to have the Web part display and act as you wish and click OK.

Here's the finished result:

In summary, this post shows how you can take an existing SharePoint list you may already have that tracks "potential trouble items" and turn it into a handy KPI dashboard. I hope you found it helpful.

Remote control parenting

We had promised our 13-year-old daughter access to the Internet on her last birthday. She already gets access at her dad's house and when she visits her grandparents, but I kept dragging my feet. For one thing, her computer is old and slow and underpowered (remember when 256MB of memory was a lot?) and can barely run Windows Me, let alone a newer OS nor anti-virus software, so putting it directly out on the Interpipes would be like dropping a goldfish into a piranha tank. And for another, I don't trust any sort of parental controls that live on the computer itself, because kids are smart and can figure out their way around most of them if they try hard enough. Sometimes it's as simple as shoulder-surfing Mom or Dad while they make a change to the computer.

So then I got the bright idea of having her access the Web in the same way I do all my work - via a remote desktop session. I figured I could have her remotely login to my Ubuntu server and then use the Web from there. One of the biggest things I like about that is that I will be able to "watch over her shoulder" virtually, and she is going to know that, too. I honestly don't plan on watching her every move (she's a good kid), but I will check in from time to time, and I do want her to know I could be watching at any time. That, combined with the computer being in a public area with the screen visible to all ought to be enough.

So today I finally got around to setting it all up. There were many fits and starts, but basically it all boiled down to the following:

1. Installing Xubuntu on a spare under-powered box (but with 384MB of RAM, not just 256!) we had laying around. I had to do this because it turns out her machine's network adapter is hosed. I chose Xubuntu because it runs the Xfce window manager which is more friendly to machines with "lightweight" resources. I put the Xubuntu box under her desk with her other machine and hooked up a spare KVM to both of them so she can share the same keyboard, monitor and mouse with her Windows machine, where she has a lot of her school work, games, etc.
2. Configuring the "new" Xubuntu machine so that her id is a "normal" id with no sudo or other admin privileges.
3. Configuring the Xubuntu machine with a static IP address, a default gateway of my Ubuntu server (which doesn't do routing) and no DNS servers. That makes it pretty hard to get out of the local area network, and even if she does, with no DNS servers configured in the networking on her machine, it makes finding anything on the 'net rather difficult. Especially since the router is set to block her machine by MAC address as well. :o)
4. Installing TightVNC server on the Ubuntu server.
5. Adding her id as a normal, non-privileged id on the Ubuntu server.
6. Adding the following command to her .bashrc file on the server:
   
   /usr/bin/vncserver --geometry 1024x768 --depth 16 --alwaysshared :2
   
   Now whenever she wants to surf the Web (and she's going to have to ask) I will simply Alt-Ctrl-F2 over to another console session and login as her and it will start the VNC server on display :2. That --alwaysshared parameter is important, since that's what enables me to login to the same desktop and "look over her shoulder."
7. Setting up the Remote Desktop Viewer (vinagre) on the Xubuntu box to remember the VNC password to the server VNC session.
8. In the Sessions settings on her Xubuntu login I added the following:
   
   /usr/bin/vinagre --fullscreen 192.168.xxx.yyy:2 (xxx.yyy = the address of the server)
   
   Now whenever she logs into the Xubuntu box the VNC client will pop up in full screen mode. She can still get out of it, but she can't go off machine if she does (see above), so there's not much reason to.
9. [Added on November 16, 2009] And because nothing's ever easy and the keyboard mappings weren't right due to an apparent bug in Gnome running under VNC I then applied solution #3 from here and that worked (after stopping and restarting the VNC server).

It's just that easy! :o)

Here's a screen shot of Remote Desktop Viewer on the Ubuntu server seeing her session from the Xubuntu desktop:

There's one downside to using VNC and that's no audio. I will probably switch it all over to another protocol at some point in the future so she can get sound while playing Flash videos and whatnot, but for now I think she'll be happy enough.

Why the Web is so fun

We stumbled across the following at work today while testing a new Windows Server 2008 R2 image for use as our new terminal servers. One of the Web-based software products we use lost its menu when opened in IE 8 on that machine, accompanied by a Javascript error. Changing the compatibility view settings made no difference whatsoever. IE 8 comes built-in on R2, so it's not like we can revert to IE 7 through a simple uninstall of IE 8, either. I did some digging and here is what I found.

The vendor's application is built in ASP.NET. Looking at the .aspx page that builds the menu, I noticed there was conditional HTML being built depending on whether the software had been installed to use Java or Javascript to build the menus (I think at one time in the past they used Java and then decided that was too heavy-weight just for something like menus and reverted to Javascript, but left both in there for backwards compatibility). In the HTML that gets excreted if they use Java, there is an OBJECT tag with a bunch of PARAM tags embedded in it passing values needed for the menu. All well and good.

But if the site was installed not to use Java, then the HTML that gets built changes the OBJECT tag to a DIV (while still using the same "Applet1" id attribute - don't you just love legacy code?) and keeps all the PARAM elements embedded within the DIV. Got that? It's no longer an OBJECT, but a DIV, but they still embed PARAMs in the DIV. Later on in that page they then do the following:

var els = Applet1.getElementsByTagName('PARAM');

Then they process through the PARAM elements in Javascript and use the values in the NAME and VALUE attributes to build up the menus. But there's a problem. In the past, IE apparently would simply go, “Huh, a PARAM in a DIV - that’s different,” parse it and add it into the DOM and move on. IE 8 says, “Huh, a PARAM in a DIV – that’s retarded,” and drops all the PARAMs on the floor (since browsers historically ignore elements they don’t understand). Hence, els.length === 0 after the above call, but of course they're not checking for that and blithely go accessing elements by subscript in the els node list, which then yields the to-be-expected "object undefined" error.

I built the following little test page to demonstrate:

<html>
<head>
  <title>DOM Test</title>
</head>
<body>
  <div id='MenuItems' visible=false style='display:none;'>
      <param name='foo' value='1' />
      <param name='bar' value='2' />
      <param name='xyzzy' value='3' />
  </div>
  <div>
      <span><b>Count of <i>&lt;param&gt;</i> elements found:&nbsp;</span><span id='Results'></span></b>
  </div>
  <script language=javascript type="text/javascript">
      var items = MenuItems.getElementsByTagName('param');
      Results.innerHTML = items.length;
  </script>
</body>
</html>

The results? Firefox 3.5.1, IE 7 and Opera all displayed 3 in the Results span. IE 8 and Chrome(!) displayed 0. I didn't test Safari because I didn't want to bother with reinstalling it. But the Chrome result surprised me. I looked through the HTML and DOM standards for a bit and didn't see any standards language using MUST around where PARAMs should live, although it does indicate that PARAM elements are supposed to go inside APPLET and OBJECT elements (duh). But it doesn't say MUST. Obviously as with so many things W3 related, there is room for interpretation, argument and cross-browser differences. Which is what makes the Web so fun! :o) My reasoning on why IE 8's compatibility view didn't make a difference is because it seems aimed mostly at rendering, not necessarily on DOM compliance.

So I guess the moral of the story is the new! improved! more standards-compliant IE 8 is not going to accept a lot of the stuff that IE used to accept in the past when it comes to the DOM. Be careful out there.

Configuration Resource Manager

My friend Clint Edmonson has been threatening to give a talk on "Never start with a blinking cursor" for some time. I've heard him give the highlights of his thinking on this over lunch a few times and I agree with him 100%. Basically, the gist of it is that with all the modern application platforms for collaboration, blogging, customer relations, content management and what not, anyone who starts a new application by opening up Visual Studio to a new project with a blinking cursor is wasting their (and their employer's) time and money. They are going to end up rebuilding a bunch of things that already exist in all such application platforms. Things like security and user management. Scalability. Import and export. Logging. Configuration. Lots of mundane little features which, each taken by themselves may not be so hard to do (although they are hard to do right), but when added up together are just an orgy of wheel-reinvention.

Why bother? Don't you have a business domain problem to solve? Spending time working on anything other than the business problem when writing business applications is what I contemptuously call "graduate research assistant projects." Resume enhancers. Satisfying as mental masturbation and giving an illusion of some sort of progress ("We are about a week away from getting the logging subsystem finished"), but not of much real worth. The business is waiting on a solution, not on you figuring out how to build a UI with the CTP release of the .NET Framework 6.0's telepathic user interface (code-named "Mindmeld").

I've been thinking about this a lot since Clint originally brought it up. And here's the thing - programmer's eschew most such platforms as the basis for their application infrastructure because most are Web-based not just for presentation but for setup and configuration, and you can do an amazing amount of work and produce a lot of functionality simply by sitting in front of a browser for half a day and pointing and clicking. Which is really, really productive. But it is also really, really boring. I mean, we all went to school and learned our chops by programming, right? So if we build on top of such a platform, where's the programming? Where will the interesting bits be?

Trust me, friend, they'll still be in there. There will be plenty of hard business problems to solve no matter what, and you can actually work on solving them well if you don't spend the first three days/weeks/months of your project dicking around with building your own workflow engine, or whatever.

The way my mind works I don't get many flashes of brilliance; few sudden epiphanies. No, I think about things, ruminate on them (my boss is very used to me saying, "Let me sleep on that and tell you in the morning"). I park them in my subconscious and let them ferment while various inputs come in, some which don't seem obviously related, and then when it's ready, there will be this quiet little thought of, "You know, you could do it this way." And I've learned to trust those quiet little thoughts. Listening to them, nurturing them, proving them out and then selling them to others are pretty much the crux of any value-add my career has ever had for my employers and their customers.

So let me tell you about my latest quiet little thought. One of the things we wrestle with at work is the ongoing configuration of a suite of applications that have grown up over the past five years. During that time various techniques have been used for storing configuration values, based on what technologies were available and popular at the time, how much time there was to implement the given application being worked on, who was doing that implementation and their skill level and what not. So we've ended up with a mishmash of configuration stores and access techniques - some in web.config files, some in standalone XML config files in a shared directory, some in various app-specific databases, some in a database specifically created to hold configuration data across the enterprise, and I'm sure the registry's got a few values tucked away in it, too.

Most (the vast majority) of this configuration data is business-oriented, yet much of it doesn't have even have a simple CRUD application in front of it so that business users can make configuration changes themselves. Instead, the IS department is always being called in to open some obscure XML config file and change a value that the business specifies, but may not know how to specify it in the syntax the file and program requires, requiring an email exchange with the vendor's tech support to find out. Maintaining business configuration values is not a good use of high-cost technical resources, but we are forced to simply because we're not going to let the user's hack up XML files or edit values directly in the database.

What is clearly needed is a uniform platform for holding configuration data. That platform should at a minimum supply the following:

• A single, cross-enterprise configuration data store accessible by all applications.
• A user interface that is capable of editing all the configuration data and that can be used by a business end user, including all the basic CRUD operations.
• A robust security model that includes not just edit and read capabilities but also security trimming so that a user can't even see that a given configuration entity exists if they aren't allowed access to it.
• The ability to search across large configuration data sets to find the entries of interest for editing (with security trimming on the search results, too).
• The ability to model complex configuration data entities, far beyond BBNVPs (Big Bags of Name/Value Pairs).
• The ability to access the configuration data via both SQL and Web services in a secure manner.
• The ability to import and export configuration data. The imports for when originally setting up the system with default values from the vendor or when migrating settings between a lab and production, the exports for migrating between systems and sending configuration sets to the vendor for support issues.
• Having the data store already built into the nightly backup and maintenance procedures, with absolutely no extra effort required by the IT staff to ensure its recoverability and performance.
• And finally, it should require absolutely NO CUSTOM CODE to achieve any of the above. None. Zero.

And my quiet little thought? It was that I already know of just the platform in our shop for accomplishing all of that. And if you are a certain type of Microsoft-based shop, you may have it in-house already, too.

[Warning: I am going to commit blasphemy now, and half of the programmers who ever read this will stop reading right after the next sentence and never come back, convinced I am a madman.]

Simply put, my quiet little thought was that all configuration data should be put into custom entities inside Microsoft CRM (assuming you have MS CRM). By doing so, you get EVERY SINGLE ONE of the bullet points above, built-in from the start. Just from a few hours of clicking in a browser window. The only code you then have to write is code to consume the configuration data via SQL or Web services, but hey, you were going to write that code, anyway, and if you're smart you've already abstracted your code's access to configuration data, so the only thing you're going to have to change is the lowest level data provider in that code. That's it.

To be fair, one of the applications I mentioned above had already started using such an approach for holding a lot (not all) of its configuration values in CRM, but it was already a CRM-based application with lots of custom CRM entities to begin with. But it hits me as fairly original that if you have MS CRM in your shop then it should be the repository for all configuration information for all of your apps, with the exception of a single connection string or Web service URL buried in your (Web|App).config file for how to get to it in the first place. Or perhaps something like that connection information simply goes in your machine.config file and be done with it.

Purists are going to argue that's not what CRM is for. So what? One of my favorite quotes lately is Voltaire's, "The perfect is the enemy of the good." In half a day I can roll out a cross-enterprise configuration data store complete with a secured business user interface, and one that will require not a single extra bit of work from IT to install it, maintain it and back it up. It won't cost a thing. And by doing so I didn't reinvent a single wheel. I just consumed what I already had in a new and different way, and moved on so I can get to the business at hand, which, like it or not, is solving business problems, not writing cool new configuration storage mechanisms.

And besides, who says it's not what CRM is for? CRM is as much an application platform as any other I can think of. We've built heavily on top of it, in fact, and yet looking at what's been done, I would say we've only leveraged 25% of its capabilities, if that. In this case, my title is my point - CRM doesn't just have to stand for "Customer Relationship Management." It can stand for "Configuration Resource Manager," too. And probably lots of other "CRM" abbreviations I don't have the time to generate just to be cute.

So even if you don't have MS CRM in-house, even if you're a UNIX shop, this post is meant to challenge you into thinking about "Never start with a blinking cursor." What do you have in your shop that you can already use to take care of something mundane, like configuration data, or task assignment and management (not even talking about something as complex as workflow), or calendaring, or whatever? The next time an application you're working on needs such "utility" functionality, don't think about writing it yourself. And don't think about buying a tool, either. I am betting that you already have something that would work Just Fine. And that will save you a lot of effort in the long run. Effort you can then use to be a hero and deliver five more business functionality use cases in this version, because you didn't have to spend unnecessary time building such utility code from scratch. Or maybe just delivering what was promised ahead of schedule, because you didn't dick around writing Yet Another Configuration Infrastructure.

Thanks to Clint for his ideas and great lunch conversations on this! I have plagiarized them and put them out here as much as a goad to get him to write more about it (because he has much more to say than I have written here) as to share my bit of insight on how one such existing platform can be used to implement just such a concept.

Put away childish things

Most developers I know who have any influence whatsoever on picking out their own hardware, either by being able to spec it at work or by buying their own, are always proud of having the latest and greatest tower or laptop. They then proceed to fill it up with a database server, a web server, a source code control server if they're working on their own, plus a modern, heavyweight IDE, not to mention all the typical stuff one puts on a box such as anti-virus software, an office suite plus some games for when there needs to be a break from debugging that wily crash in the middle of some heinous multi-threaded code. They get their box all set up, tweaked and tuned to their liking - Consolas font installed, desktop theme to their liking, and on it on. It can take days to get it all right.

And then their hard disk crashes.

Or worse, that state-of-the-art box not so much screams now, as cries. It doesn't run, it crawls.

Might I suggest that none of this is a productive use of a modern, 21st century developer's time, nor of end user consumer-level hardware resources?

So what do I use instead? I'm lucky - I work in a shop where servers are 90+% virtualized now using VMWare products, and we'd like to get to more of a 97% level. So we understand all the "chewy goodness" of VMs. Portability. Scalability. High availability. The ability to clone a new box from an old one at the press of a button. Snapshots and the ability to roll them back if something goes awry, or to commit them if the upgrade goes well.

And guess what - everything that makes virtual machines make sense for production or lab boxes also makes them make sense for development environments. So I had IT set up a development server image running on one of our beefy production VMWare ESX boxes. The image is well spec'ed, so it runs well (way better than even when I had a dedicated Dell PowerEdge server in my office), even with all the typical cruft a developer adds to a machine in what seems to be our never-ending attempt to slow it down. Plus, it is backed up (can you say that about your dev machine? Really?), along with all the other production boxes, so I don't have to worry about a hard disk crash, or really any hardware failure, since VMWare's high availability tools allow it to be migrated to other ESX servers when there are hardware issues.

And if I want to clone a new image to take it in a different direction, like installing the latest and greatest betas to play with while still having a stable dev environment for maintaining the current production code base that's easy, and even better - I don't have to do it! I just ask the guy in charge of VMs to do it for me, and once he tells me it's up (usually in less than an hour) I log on and hack it up to my heart's content.

To access my dev image I simply use a remote desktop client, either from the laptop that sits permanently on my desk serving as a three-monitor display station, or from my other laptop which I take everywhere with me. I don't worry about accessibility to my dev image when I am at home or on the road - I can VPN in over wireless or either of the two 3G air cards I have, and the remote desktop client is tuned to be responsive enough it is usually just as good as being locally connected. When describing this to one friend he worried, "What if I can't get connectivity?" But I have never been anywhere where one of the two air cards didn't work - at least not anywhere where I wanted to be logged into work, anyway.

In the end, both my laptops have become nothing more than "display stations," dumb bricks that have a browser and a remote desktop client installed and not much more. If one of them suffers a hardware failure I am out...nothing. Absolutely nothing. I can pick up a spare laptop out of our inventory and just keep working. NO DOWN TIME AT ALL. And the thing with end-user hardware is it will fail, sooner or later. So why invest a lot of time and effort building such boxes into something they were never meant to be, i.e., a real, powerful server? I think mostly because developers like to futz around with such stuff and show it off to each other - "Dude, check out what I have running on my laptop." But that's just pointless dick swinging. Your time is better spent being productive as a developer, not doing low-dollar IT grunt work. If you want to customize your dev environment, fine. Just make sure it's running somewhere where you aren't responsible for backing it up, worrying about hardware failures (or upgrades!), security or whatever else.

If you are working as a developer in a shop that supports virtualization of any flavor and you haven't made the cutover to putting your dev environment on a real machine (or hey, "machines," plural - images are cheap), you're just playing with yourself. Sure, you may still want to convince your boss that you NEED a high-end desktop or laptop, simply so your games and videos will play well. But when it's time to work, it's time to put away that toy hardware and get some real power by developing on an unreal (virtual) server. Get out of the local machine maintenance business and let your IT folks worry about whether it's available any time you need it. You'll be glad you did.

Bandwidth is for the birds

There's an old computer saying, "Nothing beats the bandwidth of a station wagon loaded with x driving down the highway at 60MPH," where x is the high-capacity storage media of the day. I first heard it back in the mid-1980s as "Nothing beats the bandwidth of a station wagon loaded with reels of tape driving down the highway at 60MPH." Today you could replace x with DVDs or high-capacity USB memory sticks. And depending on where you live on the planet, you can now apparently also replace "station wagon" with "carrier pigeon" and still beat your ISP.

You may remember the April Fool's RFCs for IP over Avian Carriers, first described here, then updated with Quality of Service here. Well, apparently a South African firm took that idea to heart and raced a carrier pigeon with a 4GB memory stick strapped to its leg against its ISP to transfer a large amount of data over 60 miles. The pigeon won handily by over an hour.

To be fair, according to the ISP, "Several recommendations have, in the past, been made to the customer but none of these have, to date, been accepted." But I think that if a company is going to go to the trouble of pulling off a publicity stunt like this there are probably some underlying issues that are going to remain even if they accepted the recommendations (which, for all we know may be too costly or simply wrong).

Anyway, it gave me a chuckle. Hmmm, I just noticed that there's no RFC for IP over Serial Routed Transcontinental Equine Carriers. And there's plenty of time before next April, too!