The Puf Principle

Block font

2010-06-13T05:14:00.003+01:00

Many people seem to doodle while they're in meetings. Doodling is a nice way to distract yourself with something, while ensuring that you can still follow what is being presented. I guess that's why those meeting centers hand out those incredibly thin notebooks.

My doodles typically consist of writing down words that have to do with what is being presented. Now before you say "that is called: taking notes" - I take my notes in elaborate fonts.

One of the fonts I came up with this time is a 'block font'. And since the meeting was rather long, I decided to create all letters.

I should apologize to the speakers, because I wasn't paying too much attention near the end of their presentation. Coming up with all letters in a single style is hard work.

I'm quite happy with most of the letters, but there's some that I really can't get to look decent. The I and the J are especially painful. Is there somebody with real designer skills that can show me how to do those character properly in this style?

What I use my iPad for

2010-04-10T19:03:00.009+01:00

It's been a week since Apple released their latest revolution: the iPad. By now every major site has reviewed it (most positive), but unsure what the device is meant for. I got my iPad on the launch date, so have had a week to play around with it and show it off.

In the past week the number one question from everyone has been: what do you use your iPad for? As with any new device type, the most common use-cases are not immediately clear. I don't think Apple ever thought the appstore would be the greatest attractor of new users to the iPhone. The iPad is no different: it's a completely new beast and it'll take some time for people to figure out what to do with it.

In the past week, I've used my iPad primarily for these three things:

Games

Surfing

In addition I've actually also spent quite some time showing the iPad off to various people. But that is of course a use-case that will be less relevant over time. Until that happens though, I'd advice every iPad owner to buy the apps in the iWorks suite. They show off the serious side of the iPad best for the moment.

Games

Not surprisingly the iPad makes a great device for games. For me it has been mostly casual games so far, but I might try out some more serious games (red alert, need for speed) soon.

My favorite games so far have been Harbor Master, Rush Hour and Labyrinth. All three seem to have been made exactly for this type of device, but with Rush Hour the playing pieces feel a bit too big. Maybe they're ahead of the pack and already made their game for the smaller iPad that is rumored to come next year.

Surfing

The iPad comes with Apple's latest Safari release, which is perfect for surfing the web. It doesn't allow tabbed browsing, which I think is a miss. But you can keep multiple pages open and switching between them is relatively elegant. When you shut down/restart Safari the pages take a bit of time to reload, which I think should not be needed on a device with 16GB or more of memory.

The iPad will not soon replace your laptop as your main device for surfing. Although it is totally usable, it is just not made for using at your desk. Maybe that changes when I get a docking station later this month, but for the moment the laptop wins the battle for my desk.

Where the iPad really shines is on the couch. I had some discussion with friend who'd say they use their laptop on the couch too. Maybe they do, but it's not the same thing. Whether I'm talking to visiting friends or are watching TV, the iPad is always within immediate reach. You can just leave it turned on all the time, it turns off the screen automatically and never gets hot nor does it ever make any noise. A laptop would simply not fit comfortably in the same seat as me or be waiting on the armrest of my chair just in case I need to look something up. Hmm... I think I've seen that actor before, let's look on IMDB quickly - no need to wait for a commercial break.

Email

My number three usage has been reading and responding to email. The on-screen keyboard is quite good, especially in landscape mode. Although as Apple claimed the keys are almost full-size, it does take some time to adapt to them. Without tactile feedback it is very easy to touch a few keys without noticing. So what I see most people do very quickly is arch their fingers a bit more to avoid those accidental key presses.

Reading and writing email works like a charm. Switching between multiple mailboxes takes a bit of tapping, but nothing too bad. The iPad really came to my rescue this week when my laptop started acting up. Instead of having to fall back to my iPhone or having to steal somebody else laptop, I could switch to the iPad and write reasonably lengthy responses.

So those are my top 3 use cases of the past week: gaming, surfing and mailing. There were also some things that I didn't use my iPad for, even though there is great software available and people had predicted those use-cases.

Not for reading

Apple tries to position the iPad as the ultimate eBook reader. I was already skeptical about this before I got my iPad, simply because I've grown to love my Sony Reader over the past half year. The one thing I love about the Sony is the eInk screen.

Although the iPad's screen is gorgeous, for readability it doesn't compare to eInk. Try reading intensely for more than a few minutes and you'll notice the difference. The iPad might look fresher and cleaner, but reading an eInk screen is simply less stressful for the eyes than reading on the iPad.

Another thing holding the iPad back as an eBook reader is it's size and weight. I often read my Sony in bed, lying on my back holding it above me. I challenge anyone to do that with the iPad for more than a few minutes. It's simply too heavy and too clunky.

That said: the iBook software on the iPad looks great. Apple was kind enough to include an illustrated book of Winnie the Pooh, which perfectly shows the type of material the iPad was made for: colorful books with large fonts and pictures. I can easily see how the iPad might be a kids favorite reading device, even though I will stick to my Sony Reader for lengthy reading sessions.

Where the iPad had proven useful is for reading articles. Whenever I want to read an article from a website, I've always had to convert it for reading on my Sony Reader. Although it's not a huge amount of work, I love using Instapaper for the iPad. With a single click on my laptop, I transfer any article from the web over to Instapaper on the iPad - where it is automatically reformatted into easy reading format. I really wish a similar program/feature existed for my Sony Reader though, because I do miss the eInk screen when reading on my iPad.

Not for video

I've also not been watching much video on my iPad. I think I should say "not yet" here, because there seems to be no reason not to use the iPad for watching youtube or video's bought from iTunes.

Since the iPad doesn't support flash, many video sites may not work. But hopefully this is just a temporary nuisance until all sites do what youtube did: re-encode their video's to work in H.264 or whatever the format is.

When I did watch video, it looked great on the screen. At time the wireless network at some places couldn't keep up with the video, which is an unfortunate consequence of the HD resolution that the iPad supports.

Who estimates the stories in Scrum?

2010-03-04T19:49:00.000+01:00

A few years ago we did a project with an outsourcing company. We gave them the current source code of a product, we gave them the backlog for the new version and we asked them to start implementing that backlog top to bottom.

Since the external team was new to both the product and the Scrum process, I decided to also do some rough estimates myself. Given my experience with both the product and the process, those estimates could serve as a rough baseline.

The progress

The remote team worked on the project for 5 sprints. During that time we saw an interesting trend.

As the team finished each sprint, some of their work was accepted and other work wasn't. Work was sometimes rejected because of its quality, but often it was rejected because it simply didn't do everything that the product owner had expected. In those cases the team put another story on the backlog for the missing parts and provided a fresh estimate for the "new" work. They also re-estimated the existing stories based on the newly gained insight of the previous sprint.

So over time the team's backlog grew in size. That is not uncommon, but look at the burnup chart:

burnup 1: based on team estimates

Based on this burnup chart, the project is in serious trouble. The team seems to be doing great, delivering more work with every sprint. But even though more and more work is delivered every sprint, the team is not getting closer to the goal.

The fifth sprint here is really disastrous: the team stopped increasing delivering more work. Did they burn out? How are we ever going to get the project out the door?

For comparison let's look at the data based on my estimates - that the team never saw and didn't commit to:

burnup 2: based on estimates by an external stakeholder

On my end, for every new story the team added to the backlog I checked if I had considered that part in my original estimate. If so, I split the story points I had originally estimated over the two stories. If the new story was functionality I had not expected in my estimate, I would provide a fresh estimate - also increasing the backlog in my part. As you can see that happened only once, but the increase was substantial (about 10%).

In my chart too the team seems to be burning out a bit in sprint 5. But it doesn't seem half as bad as in the firsts burnup chart. They are still getting closer to the goal.

And in both charts the team seems to be about 20% done with the total amount of work.

Analysis

So what's the difference between the two charts and estimators. From my perspective there are two major differences: the stability of the progress and who made the estimates.

How stable is the progress

The charts below show the velocity per sprint as derived from the burndown charts above:

velocity 1: velocity from sprint to sprint in burndown 1

velocity 2: velocity from sprint to sprint in burndown 2

The burnups don't really have a lot of data. But if you look at the first velocity chart, you can see that sprints 1 to 4 show a somewhat exponential growth in velocity (1.3x, 1.4x, 1.8x).

The scope/goal line in the corresponding burnup chart (burnup 1) shows a constant growth, mostly because I don't have the exact data anymore.

So at some point the two lines in burnup 1 are going to intersect, but it is pretty difficult to determine where they'll intersect with the naked eye.

The second burnup doesn't have this growth in velocity and the scope increase is about 10% over 5 sprints.

It is a lot easier to see where this project is going to end. And isn't scrum all about transparency and easy to use charts and data?

Who provides the estimates?

The second question I posed above is who provides these estimates. With the whole hype about lean development I learned that one way to optimize output is to eliminate waste. Anything that isn't delivered to the customer is waste and should be eliminated.

Who needs these estimates? The customer? I don't think my customer cares about estimates. He cares about getting as much software as possible for his/her money. In fact, while the team was providing these estimates they could also have been building more software.

So is it the team that needs these estimates then? After all without those estimates, how do they know what they can commit to? Well... the team does need to know how big a story is before they can commit to it. But they only need to know that at the start of each sprint. And only for the stories that they think they might do in that sprint. So although the team needs to know the size of each story it commits to, it doesn't need to know the size of all stories at the start of the project. Nor do they need to re-estimate the stories (another form of waste).

So the only person that actually needs those estimates, is the guy drawing the charts. In this case that was me, an external stakeholder who is not a part of the team. In many cases it will be the scrum master, who needs those estimates to give his stakeholders some view of the progress towards the overall goal. In other cases it will be the product owner, since he is most interested in seeing his return on investment.

Conclusion

My suggestion: let the guy who needs them come up with the numbers. And if you don't feel comfortable, do a reasonable guess. And if you don't even feel comfortable guessing, set all stories to the same size. In the end it doesn't really matter too much and you'll allow the team to focus on what really matters: building working software.

Java Particle Systems part 2: adding some life

2009-08-25T08:04:00.005+01:00

The frame below shows a demonstration of the particle system we are about to build. Move your mouse through it to have some fun. If it doesn't work for you, click this link to open the same demo in a popup window.

Introduction

A few months ago I wrote an article that demonstrated how to create a very simple particle system in less than 50 lines of JavaScript code. That article has slowly been finding its way into the Google search indexes and by now it's becoming obvious that I'm not the only one who thinks particle systems deserve more attention and JavaScript is the perfect language to experiment with them.

Back then I was quite pleased with being able to implement a fully functional particle system in less than 50 lines of (quite readable) JavaScript code. But I think we can agree that the particle system was not very exciting. Although the system contained all three required ingredients, the end result was a bit "static". From games and other places where particle systems are used, we are used to them being a lot more "alive".

So that's the goal for this article: to add some life to our previous particle system, while still keeping it small. In this case I've increased my self-imposed limit to a 100 lines. A generous doubling from our previous system, so we'd better have some real interesting "life" to show for the increase.

We'll add three behaviors to the particle sytem.

Gravity

Grow then shrink

Mouse control

Lot's of work to do, so let's get started with gravity straight away.

Gravity

In our first particle system, the particles would fall at a constant speed. The fall speed of each particle was determined at random when creating the particle.

      dy: Math.random() + 0.5

And then we just updated the y position every time:

      particles[i].y = particles[i].y + particles[i].dy;

I always find it very interesting to add some gravity-like behavior to systems I create. So how about making the particles fall at an increasing speed?

      p = particles[i];
      p.y += p.dy;
      p.dy += 0.1; // simulate gravity

So we increase the particle's y velocity every time we update its position. I realize that to make this work like real gravity I'd have to know the particle's mass and do a multiplication. But for this simple example just increasing the dy is already convincing enough.

Since the particle will now drop at an ever increasing speed, it is nice to make it go up a bit at first:

      dy: -1 + Math.random()

So the particles will first move up a bit and then start falling down at an increasing speed. If all goes well, they'll move in a nice arc-like pattern. Again there is no real math or physics reason for these values, these just looked best to me. If you feel the need to make the system more physically correct, just download the source and go ahead. Let me know if you find an interesting combination.

Grow then shrink

Another thing I wasn't too happy with in the original particle system is the static size that the particles have. Each particle is created to be 3px in width and height. By the way: there was a bug in the original code such that the sizing didn't work in Internet Explorer. As my team often tells me: "you need to test more with IE Frank!" My bad. Let's hope I got it right this time.

I changed the code to shrink the particle a bit in each update:

      p.s -= 0.1

This of course requires the particle to have a size when we create it:

      p.s: 3.0

With this setup the particle will slowly shrink from 3 pixels width and height to 0. Once the size of the particle reaches 0, we can kill it:

      if (p.ttl-- > 0 && p.s > 0) {
         updateParticleElement(p);

This leads to a working particle system, but somehow it's still a bit boring when all the particles start and shrink at the same speed. So let's make the size change random and instead of just shrinking, let's make the particles grow at first and then shrink until they die out.

For the initialization the change is simple:

      p.ds: Math.random()+0.1,

This gives the particle a growth speed that is guaranteed to be positive. So instead of adding 0.1 each time we update the particle, we add the ds:

      p.s += p.ds;

If we leave this unchanged, the particle will grow forever - or at least until it's time to live runs out. To ensure the particle also starts shrinking at some point we give it a maximum size:

      if (p.s > 5) p.ds *= -1;

When the particle reaches its maximum size, we just flip the sign of the growth speed - so it becomes a shrink speed. After this the particle will keep shrinking until it reaches size zero at which point it dies out.

By now I'm not really sure if the ttl property is still needed, as the particles already have an implicit time to live and get their randomness from the growth/shrink speed. But I left the ttl in as a safeguard against my own mistakes when manipulating the size. I wouldn't want to end up with an infinitely living and growing particle by mistake.

If you would reconstruct the particle system at this point it is already quite lively and dynamic. Particles seem to be attracted by the bottom of your screen and particles seem to have a real life cycle, where they grow and shrink to die out. But this particle system is still a spectator sport. Let's see what happens when we give the user some control over what happens on screen.

Mouse control

As our last feature we want to give the user some control over the behavior of the particles. Specifically I want to accomplish two things:

particles originate close to the position of the mouse cursor

particles move roughly in the direction that the mouse is moving

To accomplish this we need to add four variables to the system.

      mx and my - the position of the mouse
      dmx and dmy - the direction and speed of the mouse's movement

These variables are global to the script, so they are not stored for each particle.

These variables are updated whenever the user moves the mouse, by hooking the onmousemove event of the window. The mousemove handler is a bit tricky to read at first. But since the behavior of the particles only depends on the four variables mentioned here, I will not explain the mousemove handler of the code further.

Now how do we use these variables when creating and updating a particle? The variables are actually only used when we create a new particle:

      x: mx,
      y: my,
      dx: dmx + Math.random(),
      dy: dmy + Math.random() - 1,

As you can see the code remain quite similar to how it was. We've just substituted our new "mouse dependent" variables for the previous hard coded and random values. We still make the dx and dy a bit random, since otherwise all particles would go in the same direction.

The updating of the particles doesn't have to change for this behavior. The mouse just allows the user to control the position and the direction of the initial movement. After its creation each particle behaves as the rules of our system dictate, which ensures the particles still behave similar even with al their differing properties.

Summary

So in the first article we created a simple particle system in less then 50 lines of JavaScript code. The system was fully functional, but felt a bit static after you'd been looking at it for a while.

In this second article we've added life to our particle system by adding by:

making the environment a bit more "physical" with our simple gravity emulation

making the life cycle of the particles more complex, by making them grow and then shrink and die out

allowing the user to control the particles, by hooking their position and movement up to the mouse movement

All in all our particle system has now gotten quite complex. But weighing in at 90 lines, it is still well under my self-imposed limit of a 100 lines of code. I did my best to keep the code readable again and even added some constants to make it easier to change and debug the particle system.

Now go ahead and play with the system. I find it's great fun to see how far up you can throw the particles. Remember: this is controlled by moving the mouse, so start pushing and pulling your mice!

The code


// constants
var NUM_PARTICLES = 50;
var FPS = 20;
var PARTICLE_LIFETIME = 4; // in seconds
var ANIMATION_LIFETIME = 300; // in seconds

// global variables
var container;
var particles = [];
var framecount = 0;
var mx = 20; // default origin, will be set to mouse position once the mouse moves
var my = 20;
var dmx = 1.0; // default movement, will be set to mouse movement once the mouse moves
var dmy = -1.0

function onMouseMove(e) {
 var IE = document.all?true:false
 var nmx, nmy;
 if (IE) { // grab the x-y pos.s if browser is IE
  nmx = event.clientX + document.body.scrollLeft
  nmy = event.clientY + document.body.scrollTop
 } else {  // grab the x-y pos.s if browser is NS
  nmx = e.pageX
  nmy = e.pageY
 }
 dmx = (nmx - mx) / 4;
 dmy = (nmy - my) / 4;
 mx = nmx;
 my = nmy;
}

function createParticle(container) {
 var p = {
  elm: createParticleElement(),
  x: mx,
  y: my,
  s: 0.0, // size
  dx: dmx + Math.random(),
  dy: dmy + Math.random() - 1,
  ds: Math.random()+0.1,
  ttl: PARTICLE_LIFETIME*FPS
 };
 container.appendChild(p.elm);
 updateParticleElement(p);
 return p;
}

function createParticleElement() {
 var elm = document.createElement('span');;
 elm.style.border = '1px solid blue';
 elm.style.position = 'absolute';
 return elm;
}
function updateParticleElement(p) {
 p.elm.style.width = p.elm.style.height = p.elm.minWidth = p.elm.minHeight = Math.floor(p.s) + 'px'; // doesn't work on IE
 p.elm.style.left = Math.floor(p.x) + 'px';
 p.elm.style.top = Math.floor(p.y) + 'px';
}

function updateParticles() {
 for (var i=particles.length-1; i >= 0; i--) {
  var p = particles[i];
  p.s += p.ds;
  if (p.s > 5) p.ds *= -1;
  p.x += p.dx;
  p.y += p.dy;
  p.dy += 0.1; // simulate gravity
  if (p.ttl-- > 0 && p.s > 0) {
   updateParticleElement(p);
  } else {
   container.removeChild(p.elm);
   particles[i] = createParticle(container);
  }
 }
 if (framecount++ < ANIMATION_LIFETIME*FPS) {
  setTimeout('updateParticles()', Math.floor(1000 / FPS));
 } else {
  alert("All done. Reload to watch again.");
 }
}

window.onload = function() {
 container = document.getElementById('particles');
 for (var i=0; i < NUM_PARTICLES; i++) {
  particles[i] = createParticle(container);
 }
 
 setTimeout('updateParticles()', Math.floor(1000 / FPS));
 document.onmousemove = onMouseMove;
}

Download it here.
And the corresponding HTML:


<html>
    <head>
        <title>JavaScript Particle System</title>
        <script type="text/javascript" src="particles2.js"> </script>
    </head>
    <body>
        <div id="particles" style="width:640px; height: 480px; overflow: hidden"> </div>
    </body>
</html>

Download it here.

Seven rules to keep your twitter followers

2009-07-25T20:50:00.001+01:00

With Jon's recent "top 40 list of CMS gurus" I suddenly got a lot of additional followers. And I added quite a few of those people to my twit-list too. From the first few days it looks like I might start unfollowing some soon, but there's definitely a few that are a great addition to the list. While scrolling through recent tweets, I sought what makes tweets interesting for me and things that I really don't like. So here is the list (in random order) of do's and don'ts to keep my as your follower on twitter.

1. Yet another link retweeter

"RT @cmswire Open Source for America: Change We Can Believe In? http://bit.ly/r9sj6 #opensource #oss"
"RT @cmswire 39 More Ways to Get Your CMS Twitter Fix http://bit.ly/YiV9I"
"RT @OpenText #OpenText Completes #Vignette Acquisition http://cli.gs/bUvzsN"

There's nothing wrong with retweeting something interesting you've heard somebody say. But if all you tweet is other people's tweets, I'd much rather listen to those other people.

2. Your opinion matters to me

Personal tweeting is fine. I love to hear what friends and family are doing. But from people that don't fall into those categories, I don't need:

"Ottawa Organic Market on Bank.... "

Don't just tell me that you're at an organic market in whatever city. If we're not emailing or calling each other at least once a week, chances are I don't care that much about your whereabouts. Sounds harsh? It's not meant to be harsh.

Because I apparently care enough about your opinion to follow your tweets. So how about you tell me your opinion on that market.

"the Ottawa organic market is really the best"

3. Challenge me

Even better, don't just tell me your opinion. Tease me to give you mine. It's OK to do a bit of trolling as long as long as you don't mind getting trolls back too.

"the Ottawa organic market is really the best, certainly beats the one in Amsterdam"

Now that's a great way start to a discussion. Because I really, really LOVE the organic market in Amsterdam.

4. Don't forget I'm eavesdropping

One of the worst things you can do to me on twitter is to start a 1-on-1 conversation with somebody where you forgot that I'm still there reading along.

"Sorry? Barbecue where? Are you kidding me?"

What barbecue are you talking about? Where is it? Am I invited too? If you don't want to let me know these things, why are you sending your private message as a public tweet?

Remember: if you want to have a private conversation, there is always still email or Google talk. Or even better dm in twitter.

But if you feel like you want to share your conversation with the world, give me enough context.

5. Give me context

If talking to someone in a public tweet, make it clear who you're talking to.

"@john123 glad you liked the post!"

It certainly beats the tweet about that barbecue. Because here I can at least check the other part of the conversation in john123's tweet stream. But still, giving me a bit of context goes a long way.

"@john123 glad you liked the post! http://is.gd/1JyHY"

One click and I'm at the post. But then again, why not just retweet the original and add your comment to it:

"glad you liked the post! RT @john123 good analogy on how to mix persuasive content with relational data in modern CMS. http://is.gd/1JyHY"

Phew... 138 characters, that's cutting it close. But it has as much context as you can add to your tweet.

6. Tell me something I can use

A friend was apparently enjoying a milkshake a bit too much and had to share it with the world:

"Just had the best strawberry milkshake ever"

I like reading it, because I always love to hear about good food - especially from someone whose taste buds I trust.

But how am I going to be able to do something with that? Tell me where you had that milkshake. Believe me, I'd love to know - even if it is a three hour drive.

"had the best strawberry milkshake ever at Mill Creek near Mineral, NoCal"

Honey start the car, we're going for a drive!

7. I don't need more spam^H^H^H^H marketing

Corporate tweeting is hot these days. There are many corporate twitter accounts that are easily recognizable, because they consist of the company's name. But there are also plenty of innocent-looking names that generate a remarkable number of tweets that simply reiterate how great product xyz is or how well product abc would solve the problem I'm (not) having).

I don't have a problem with these accounts. We're all luckily free to tweet whatever we want. But if you'd like to keep your followers: keep your marketing to content ratio below 20%.

8. I read your blog already

Some people see link tweets as the natural progression of feed readers. I don't. I still have a healthy 100 or something feeds that I follow, just because they regularly show content that I like to read.

In other words: if you have a blog that is of interest to me, chances are it's already in my Google reader. So if all that you are tweeting is links to your blog posts, following you is not really adding any value for me. Now where's that unfollow link?

Mind you: there's nothing wrong with tweeting about your own posts. In fact that's what I'll do once I click the publish button for this post too. But if you want to keep me as a follower, be sure to add some original content into your tweets too.

And yours?

Those are the rules that sprang to my mind as I was scrolling through the rapidly growing stream of tweets. Do you have any rules of your own? How can I make sure that you don't click the "unfollow" link on me?

Safari 4 on Windows crashes when you provide incorrect credentials

2009-07-06T20:38:00.002+01:00

Sorry for the long title of this post. But it is the most accurate description of a problem we're having in a project at the moment. Safari 4 on Windows seems to crash in what I perceive to be a common use case. Here are the minimal steps that I came up with for reproducing the problem:

create a new web site in IIS

enable integrated authentication on this web site

connect to the new web site with Safari 4 for Windows

provide incorrect credentials

At this point Safari 4 crashes on my system.

Providing correct credentials allows us to continue. But in the application my team is building, we will often encounter another authentication problems later on.

Safari pops up a log on box when we open a page from the same web site in an iframe. No matter which credentials we provide, we can't continue.

I've been searching whether this is a known problem for a while now. But I haven't found a single page that points out the problem we encountered here. Hence this simple write-up.

Has anyone had similar problems with Safari on Windows? If so, is there any way to work around the problem? Because as it is, we can't really support our application on Safari 4 on Windows. :-/

Particle systems in JavaScript

2009-05-16T18:56:00.001+01:00

Particle systems have always held some magic for me. Or maybe I should explain it more clearly: anything that resembles physics in a game-like fashion has always held some magic to me. In this post we'll look at how to create a simple particle system in JavaScript.

Pod racers

A few years ago when I wrote a small 3D pod racer for my own amusement. If you don't know what pod racers are, have a look at this video:

My game looked nothing like that of course. I'm a developer, not an artist. And besides I spent lots of time playing with the physics of the pod: where are the thrusters on my pod? How much force do the thrusters give? Does the force get exponentially bigger when the pod gets closer to the track? Fun stuff.

I also added a particle system to my pod. After all, those thrusters are bound to give off some exhaust fumes and this was a great way to give an indication of speed. And of course it was a great way for me to get started on particle systems.

As it turns out particle systems are really quite simple things. I had a blast playing with the parameters and figuring out how to create a most interesting effects.

Of course the pod racer was never finished and the source code has long been lost. That's why today we'll create a completely new particle system. In less than 100 lines of JavaScript. Granted it will only be 2D this time. But believe me, I've already been staring at the flow of particles longer now than it took me to write the script.

parts of a particle system

Let's get started with the three things that make up a particle system:

an array of particle data structures

a function that updates all particles each frame

some form of lifetime management for the particles

Parts of a particle

First comes the particle data structure. What makes up a particle? If you've ever seen a particle system in its bare essentials, you'll have noticed that it consists of multiple small elements on your screen that each move on their own account, yet in a predictable and similar way.

This already means quite a few things. A particle apparently has a location on the screen. And since it moves, it must also have a direction.


var particle = {
  x: 0.0,
  y, 0.0,
  dx, 0.5,
  dy: 0.5
}

So this is a particle that is at position (0,0) and moves half a pixel to the right and bottom every time.

Creating an array of these objects is of course really simple.


function createParticle() {
  return {
    x: 0.0,
    y, 0.0,
    dx, 0.5,
    dy: 0.5
  }
}

var particles = [];
for (var i=0; i < 10; i++) {
  particles.push(createParticle());
}

But of course these are really boring particles. They're all the same. Let's make the particles unique by adding a bit of randomness:


function createParticle() {
  return {
    x: 0.0,
    y, 0.0,
    dx, Math.random() + 0.5,
    dy: Math.randon() + 0.5
  }
}

var particles = [];
for (var i=0; i < 10; i++) {
  particles.push(crateParticle());
}

So this creates 10 particles. They all start from the same location on the screen, yet move out in a different directory.

With that out of the way, let's move on to the behavior of the particles.

Particle behavior

As I said in the previous section, particles move in a predictable way. So we'll need a function that updates the position of each particle every frame.


function update() {
  for (var i=0; i < particles.length; i++) {
    particles[i].x = particles[i].x + particles[i].dx;
    particles[i].y = particles[i].y + particles[i].dy;
  }
}

And of course we'll need to call this function periodically:


setInterval("update()", 50);

Calling the update function every 50ms results in a 20 frames per seconds. This will give us quite smooth animation, without being too taxing for todays browsers/PCs.

lifetime management

We're almost done here. But if we would start this particle system now, the particles would just move away from each other indefinitely. That won't be very interesting to look at.

What we need to add is some way for the particles to die and for new particles to be born. The simplest way to do this for now is by giving each particle a lifetime counter at birth and just decrease that with every update.

Let's first update our createParticle function:


function createParticle() {
  return {
    x: 0.0,
    y, 0.0,
    dx, Math.random() + 0.5,
    dy: Math.random() + 0.5,
    ttl: Math.floor(Math.random()*50) + 50 // time to live in frames
  }
}

We now defined that each particle will live between 50 and 100 frames. With a framerate of 20 frames per second, that will give the particles a lifetime of between 2.5 and 5 seconds.

Let's use this new property in our update function:


function update() {
  for (var i=0; i < particles.length; i++) {
    particles[i].ttl = particles[i].ttl - 1;
    if (particles[i].ttl > 0) {
      particles[i].x = particles[i].x + particles[i].dx;
      particles[i].y = particles[i].y + particles[i].dy;
    } else {
      particles[i] = createParticle();
    }
  }
}

See the nice little trick we do here? We don't delete the particle, we just replace it with a new one. Of course the new particle will start at the origin again and will have its own direction and speed.

This way we will always have the same number of particles in our system.

visualize the particle system

I was not completely honest at the start here. There is actually one more step we have to take before we can see our particle system in action. After all what good is a particle system if you can't see it.

Up until now the particle system we've created is completely agnostic to the output system. If you use JavaScript for scripting your DirectX game, you might render these particles onto a DirectDraw surface. If you feel really "Linuxy" you might render them onto a 24x40 character display. But for our purposes here, we'll render the particles using HTML.

I'll just show the modified fragments here. For the complete script, refer to the end of the article.

We visualize each particle as a simple HTML span with a blue border:


function createParticle(span) {
  return {
   elm: span ? span : createParticleElement(),
        ...
  }
}
function createParticleElement() {
  var elm = document.createElement('span');
  elm.style.border = '1px solid blue';
  elm.style.position = 'absolute';
  elm.style.width = elm.style.height = '3px';
  container.appendChild(elm);
  return elm; 
}

When updating the particle, we also have to update its span. When the particles dies, we re-use the span for the new particle:


    if (particles[i].ttl > 0) {
      ...
      particles[i].elm.style.left = Math.floor(particles[i].x) + 'px';
      particles[i].elm.style.top = Math.floor(particles[i].y) + 'px';
    } else {
      particles[i] = createParticle(particles[i].elm);
    }

demo

The frame below shows a demonstration of the particle system we just built. If it doesn't work for you, click this link to open the same demo in a popup window.

next steps

As you can see this particle system is about as simple as it can be. Real particle systems in games and simulations are of course much more complex. But the basic ingredients stay the same:

an array of particle data structures

a function that updates all particles each frame

some form of lifetime management for the particles

If there is interest in it, I would like to make this post the first of a series where we investigate particle systems. If that is the case, the next step will be to make the particle system a bit more interesting by adding some "physics-like" behavior to it.

the finished script

This is the complete script needed to run this particle system. If you count them, you'll see that it is only 47 lines of code. So you can create an (admittedly simple) particle system in less than 50 lines of code.


var container;
var particles;

function createParticle(span) {
  return {
    elm: span ? span : createParticleElement(),
    x: 0.0,
    y: 0.0,
    dx: Math.random() + 0.5,
    dy: Math.random() + 0.5,
    ttl: Math.floor(Math.random()*50) + 50 // time to live in frames
  }
}

function createParticleElement() {
  var elm = document.createElement('span');
  elm.style.border = '1px solid blue';
  elm.style.position = 'absolute';
  elm.style.width = elm.style.height = '3px';
  container.appendChild(elm);
  return elm; 
}

function update() {
  for (var i=0; i < particles.length; i++) {
    if (i == 0) console.log(i+' '+particles[i].ttl);
    particles[i].ttl = particles[i].ttl - 1;
    if (particles[i].ttl > 0) {
      particles[i].x = particles[i].x + particles[i].dx;
      particles[i].y = particles[i].y + particles[i].dy;
      particles[i].elm.style.left = Math.floor(particles[i].x) + 'px';
      particles[i].elm.style.top = Math.floor(particles[i].y) + 'px';
    } else {
      particles[i] = createParticle(particles[i].elm);
    }
  }
}

window.onload = function() {
  container = document.getElementById('container');
  particles = [];
  for (var i=0; i < 10; i++) {
    particles.push(createParticle());
  }

  setInterval("update()", 50);
}

Download it here

The corresponding HTML:


<html>
    <head>
        <title>JavaScript Particle System</title>
        <script type="text/javascript" src="particles.js"> </script>
    </head>
    <body>
        <div id="container" style="width:320px; height: 200px"> </div>
    </body>
</html>

Download it here

mstsc /console doesn't work anymore

2009-04-17T16:40:00.008+01:00

Summary: instead of calling mstsc /console /v:servername you should now use mstsc /admin /v:servername.

This post should really go under the category: what were they thinking?

In our company we use a product that for licensing reasons really wants to make sure that only a single instance of the application is running. One of the things the developers have done to ensure this, is not allowing the application to be started on a remote desktop session. Otherwise: you could install the program on a single machine and let multiple users initiate remote sessions on that machine and this violate the license agreement. Now I'm not saying I agree with this type of licensing setup, but the software is pretty good and we want to make sure we're not doing anything illegal with it.

So what we came up with was setting up a server with the software and then asking developers needing the tool to connect to the console session on that server using Windows' Remote Desktop Connection. The console session is (from the little that I understand of it) the first session that is started on a machine. So even on a terminal services server, there is only one console session. Connecting to it is quite simple from a command prompt:

mstsc /console v:machinename

The chances of two of our users needing access to the tool at the same time are pretty slim, so this setup actually worked for us quite well for the last year or so. Until last week that is.

Suddenly I was getting reports from one of my team members that he couldn't start the tool due to a licensing error. He checked and double checked, but seemed quite sure that he was using the correct command line. I walked up to the server myself and tried to start the tool - it started without problems. So I left the tool running for him and told him to try again. He did... and said the tool wasn't there when he connected!

Anyone with enough experience with multi-session machines knows that this means the remote desktop was in another session than the one I started the application in. But how could that be? My team member clearly told mstsc to connect to the console session.

A few Google searches later, I found the answer. Now keep in mind: the problem occurred and was solved last week. But just writing about it here, still makes me angry - very angry with some people at Microsoft.

What happened is that a team at Microsoft found out that there was a security issue with the way terminal services worked. So they changed the logic of console vs. administrative sessions for security reasons. Nothing wrong with that - I appreciate security as much as the next guy.

So they now separate the real session 0 from the first GUI session. Sounds good. I am no security expert so I don't know if it helps, but I'll take their word for it. But they didn't stop at that. Somebody then also decided that the /console option of the remote desktop client should not be called /console anymore. After all you might not be always be able to connect to the console anymore - on newer servers it would connect you to the first, typically administrative GUI. So they relabeled the "console" session to the "administrative" session. Relabeling is not wrong - I appreciate clarity as much at the next guy.

So instead of calling mstsc /console /v:servername you should now use mstsc /admin /v:servername. All clear - nothing wrong with that.

All of this so far is fine and can be explained to anyone. But this of course leaves one decision to be made: what do you do with the existing /console option? What do you do with all those existing scripts that might already use mstsc /console in the expectation that they end up connected to the console session?

And this is where they did something that I don't understand, that I really, REALLY, REEEAAAALLLY don't understand. They decided to in some cases start ignoring the /console option. So with some versions of the remote desktop client, using mstsc /console will end you up in the console/admin session. And in other versions of the remote desktop client, using the same command will end you up in a completely normal session.

Even after a week of thinking about it, I still don't understand why somebody would decide to silently start ignoring the /console a command line switch that users might depend on. The switches of a command line tool are pretty much the API that you expose to the outside world. Whole build scripts have been built against command line tools based solely on the fact that the switches that are there will not change. Switches might be added, but nothing ever gets dropped. Except at Microsoft I guess. There some genius decides that in some cases they would start silently ignoring the /console option on the command line. See this post on the exact behavior in all cases or read this one for non-Windows-2008 servers. Don't expect to understand the logic after that, but at least it is clear that they've spent a lot of time on this change. And after last week: so have I.

I know not to expect an apologetic phone call from Redmond on this, nor do I really expect them to change the behavior of the mstsc client. Instead I'll just warn my fellow programmers and be on the lookout for remote desktop sessions that don't behave like I expect them to.

Ok... time to breath in and breath out, count to ten and do my exercises. And get back to work of course. Now what was I working on?

Which keyboard is best on mobile phones?

2008-12-25T20:39:00.010+01:00

It seems that mobile phones are the main place where UI experiments are still being done. One of the types of UI experiment that recently drew my attention was in keyboard design.

Palm Treo

I just got a Palm Treo from work. That Treo has a built in QWERTY keyboard with horribly small keys. The keyboard is always available, together with the screen. And although that sounds convenient, it does mean that both the screen and the keyboard are small and the device is clunky.

GPhone

Compare this to the design of the first Google phone: the T-Mobile G1. When using this device as a phone the keyboard is stowed away under the screen. When you want to access one of its many PDA functions, you fold away the screen and the mini QWERTY keys become available. So while in phone mode the device is a lot smaller, but in PDA mode it is bigger.

iPhone

Lastly there's the iPhone solution: a virtual on-screen keyboard. There on-screen keyboards were pioneered by Windows PDAs about a decade ago, but leave it to Apple to breathe stylish new life into an existing concept. The screen on your iPhone is big and can be used for normal operation. Until you need a keyboard, then the screen is split into two: the top section remains a normal screen and the bottom section becomes a virtual keyboard. The virtual keys are as miniaturized as their physical counterpart. But they do miss the tactile feedback you get from the other devices. So while it looks endlessly better, inputting text on the iPhone is probably slower than on the others.

So what do you think? Which is the better way of inputting text on your mobile device? The Treo approach of an "always on" keyboard? The way the Google Phone hides the keyboard during normal operation? Or do you prefer the iPhone's virtual keyboard?

Optimistic locking vs. pessimistic locking

2008-10-25T09:12:00.002+01:00

If you have an education in computer science you typically learned about locking in databases. I was taught that the only safe way to update a database in a multi-user environment is to use a transaction:

acquire a lock on the record

read the current value from the record

calculate the new value

write the new value to the record

release the lock

Simple. Reliable. Safe.

And horribly slow...

Imagine thousands of users concurrently hammering your database. Most of these users will be working on different records. So for most of them, the lock is completely useless. But you're still paying the price for acquiring this lock on every database update. That is why this type of locking is called a pessimistic lock. You assume that multiple people will want to update that same record and prevent them from doing so.

I thought this was the only way to do locking, until I encountered something completely different about 10 years ago. At my company at the time we used an internally built software application to track stock levels of our products. And that application used the pessimistic lock pattern described above. The application worked fine for some time, but as it got more users it started to have more and more performance problems. I was brought in as part of a team to solve those problems. I immediately went to work on some of the heaviest operations and was able to bring startup performance back to acceptable levels in a few days. But the updates to the database were still going as slow as ever.

That's when the technical lead of my team came with an interesting suggestion. "Why don't we remove the database locks?" Of course the entire team was in shock. You couldn't remove the locks from the updates. That meant that the data could become corrupt. That is what we told the tech lead: you need locks to keep your data from becoming corrupt. "So why don't we ensure that the data won't become corrupt when we update it?"

Nobody understood what he meant. Keep in mind that all of us were brought up with the concept of pessimistic locking. So the lead asked us to show him the queries that were being fired against the database:

SELECT ID, StockCount FROM tblProducts WHERE ID=?

application code calculates the new stock for the product

UPDATE tblProducts SET StockCount=? WHERE ID=?

He thought about it for a while and then said "How about this?":

SELECT ID, StockCount FROM tblProducts WHERE ID=?

application code calculates the new stock for the product

UPDATE tblProducts SET StockCount=? WHERE ID=? AND StockCount=?

"So we pass in both the new value for StockCount and the value that we expect to be there. That way the update will only be performed when the StockCount has not been modified by anyone else."

I can still remember the shock going through my mind. Does this mean that I don't need to lock the record before reading it? What will we do when the StockCount did actually change between our SELECT and our UPDATE?

All these questions were answered in the next few hours. If some other user had performed a conflicting UPDATE while our code was busy figuring out the new stock vale, we'd have to reperform the entire sequence above. So the process turned more into a loop:

while the stock value has not been updated:

SELECT ID, StockCount FROM tblProducts WHERE ID=?

application code calculates the new stock for the product

UPDATE tblProducts SET StockCount=? WHERE ID=? AND StockCount=?

And sure the code got a bit more complex. And definitely in the worst case, this code will send a lot more queries to the database than the one with pessimistic locking. After all, it will fire multiple SELECT and UPDATE statements. Potentially an unlimited number of them, because of the loop.

But that is exactly why people call this the optimistic locking pattern. As I said earlier, most concurrent updates to a database are to different records. So most updates don't have a conflict. But with pessimistic locking all updates pay the price of acquiring a lock. So all updates pay get a penalty for a small fraction of them that do require the locking.

Optimistic locking is built on the assumption that most updates are without conflicts. When there is no conflict, there is no penalty. And only when there has been a conflicting update, do we go into the loop of retrying and pay the penalty. The penalty will be much higher than with pessimistic locking, but it will only be paid for the cases where it is needed.

So there you have it, the difference between optimistic and pessimistic locking and how I learned about them. Pick whichever one you think will work best for your project. Both of them are perfectly suitable for keeping your data safe. It all depends on the ratio of "updates with conflicts" vs. "updates without conflicts" for your application.

Using PageRank to determine the most powerful people on LinkedIn

2008-07-13T09:53:00.000+01:00

Last week I was reading the original paper by Larry Page and Sergey Brin on the PageRank algorithm. PageRank is an algorithm to determine the relative authority of pages that they came up with while at Stanford University. And it is of course also the algorithm they then just started using for a search engine at Stanford - a search engine they named Google.

If you just skip the math for a moment, the PageRank algorithm is surprisingly simple. The basic rules in determining the rank of a page are:
1) the links to a page determine its rank
2) a page spreads its rank over the links it contains

So if you have a page with three links on it, that page will contribute one third of its rank to each of those links. A page with a rank of 100 that has two links on it, will give 50 points to each of its targets.

One thing you may already have noticed is that the definition of the PageRank can't be determined in one go: you need to know the value of all incoming links to determine the rank of a page, which you need to know to determine the value of all outgoing links. According to the paper you can simply calculate all PageRanks in an iterative fashion until it converges after a quite limited number of steps. I'll gladly take their word for it, since that puts the algorithm back into the hands of us programmers. We can iterate over large collections of data like no other.

The PageRank algorithm is so simple that it not only works, it has turned the search engine world (and the internet in general) upside down. I'm quite sure that today's PageRank is determined in a much more complex fashion, but let's stick to this older and more simple model for now.

What else could we use PageRank for?

What would happen if you let PageRank loose on a completely different type of data? How about instead of taking all the hyperlinked documents on the web, we take a social network? So take the network of people at a site like LinkedIn, Facebook, Plaxo or Orkut and apply the same two basic rules:
1) a person's rank is determined by the incoming links
2) a person's rank is evenly divided over their outgoing links

So the more people link to you, the higher your rank. And if more people link to the people that link to you, your rank will get even higher.

If we let PageRank loose on a social network like this, what will happen? Who will get the highest PageRank? And what does the rank mean in such a scenario? Is your PageRank an indicator of the power of your social network? Will my boss indeed have a higher rank than me, as you would expect? And will his boss have an even higher rank?

There are of course differences between links on a social network site and the web in general. For example: in a social network like this the links between people are non-directed, when you are connected to someone that person is also connected to you. This will likely affect the results, but I am too bad at maths to foresee how.

Basically I have no idea what the outcome of such an experiment will be. It just seems to me that it would be incredibly interesting to see what the numbers are and then determine what they mean. Much like Page and Brin did with their original PageRank: first let the algorithm loose on the data, then see what you can do with the results. If the algorithm feels intuitively good, the results are bound to be interesting.

Does you feel the same? Can this be an interesting experiment? Or is this complete nonsense and is PageRank not applicable to this scenario? And do the social network sites have APIs that will allow you to do this? Or is your data access limited to the people in your own network (as they always promise in their privacy statement) and can only someone working for Facebook/LinkedIn/Plaxo create such an application?

Three ingredients to a better bug report

2008-03-22T09:45:00.003+01:00

At work I've recently been going through lots of bug reports. It is part of my job to determine the priority of each defect that gets reported and whether we should still fix it in the upcoming release. With multiple testers trying to break the product in all possible ways and a deadline approaching rapidly, analyzing the newfound defects seems to take more and more time.

What did you see?

People entering defects can actually do a lot to reduce the time it takes to analyze their entries. You'd be amazed at the number of defects that say something along the lines:

"when I do A, B happens"

There are of course cases when this is all there is to say about a defect. For example:

"when I click the Ok button, the browser shows an internal apache error"

Granted, it would be more useful if the report said a bit more about the error message. But it is at least clear that an internal error message should not be shown to the user.

What did you expect to see?

Unfortunately things are not always so clear:

"when try to I delete the last item from the list, I get a message saying at least one item is required"

When I get an error report like this, I'm not sure what to do with it. Most likely there is an internal rule in the program that this list may never be empty. And the program seems to enforce this rule by giving a message when you try to delete the last remaining item. So there is a "business rule" in the program and the developer wrote code to enforce that rule. Where is the defect?

In cases like these I ask the person who entered the defect why they think this behavior is wrong. Typically I get an answer like:

"if I can't delete the selected item, the delete button should be disabled"

This added information makes things a lot clearer. So the tester didn't disagree with the fact that there should always be at least one item in the list, they just didn't agree with the way it was handled.

Why do you think your expectation is better than the current behavior?

But the above leaves me with a problem. There is a clear rule in the program that the list must never be empty. The programmer implemented this one way, someone else thought it should have been implemented another way.

In cases like these I ask the tester (or whoever reported the defect) to explain why they think their expectation is better than the current behavior. In the example we've used so far, the reason could be something like:

"clicking the button when there is only one item in the list will always show an error message - the delete action will never be performed. Buttons that don't lead to an action being executed should be disabled."

This is a clear - albeit somewhat abstract - description of the reason why the person expected the behavior to be different.

Prior art

In this example I doubt whether anyone will disagree with the reasoning of the tester. But there are many cases where someone will disagree. Especially the developer that implemented the functionality will tend to defend the way it works.

That's why I normally prefer the defect to point to other places where similar functionality is available in the way the tester prefers it. So in the example defect:

"in screens B and C we have a similar list and there the delete button is disabled if there is only one item remaining in the list"

This type of argument works especially well when the functionality in screens B and C has already been in a released version of the product. The users of the product have experienced the functionality and they will expect the new screen to behave in the same way.

If no similar functionality is available in the application, I often look for other programs that have similar functionality. On Windows the notepad application is one of my favorite examples. Everybody has it and the functionality as not substantially changed for at least a decade. Of course the functionality your program has might not be in notepad. In those cases I often refer to programs like Microsoft Office, Outlook, Firefox or the Google home-page. Not because I think these are perfect programs, but because they're so ubiquitous that most users accept them as a reference point for the behavior they expose.

Summary

So a bug report should at least contain the following ingredients:

What did you see?

What did you expect to see?

Why do you think that 2 is better than 1?

Now if everyone starts filing their bug reports like that, I will have to spend a lot less time on analyzing them and can get back to fixing those defects sooner. Who knows... maybe we'll make that deadline after all.

Finding primes using a grid of browsers

2008-03-02T09:19:00.000+01:00

It's been a few months since I talked about my idea of combining the power of a lot of browsers into a so-called computing grid. Since writing that post I've been working on such a browser based grid. Progress has been slow, but steady. And I'm learning a lot in the process, so I'm not complaining at all.

My current prototype grid software has a server-side and a client-side part. Tasks and results get distributed over clients as needed. It's actually pretty cool to see one node start a task and another node finish it. But one thing that my grid lacks is a data storage service. Until I add such a service (in a scalable way) I am stuck with a grid that can only handle tasks like finding prime numbers. But even from such a classic task, there is still a lot to be learned.

For example one of the problems we'll need to solve for a browser based grid is the fact that the nodes in the grid are not dedicated to being part of the grid. Whereas people that download SETI@home at least consciously decide to contribute their computer cycles to the project, for a browser based grid this decision is less explicit. Of course this is part of the beauty of a using the web for this grid: it becomes very easy for people to join the grid. But it also means that we can't do certain things. Like eat up all available CPU cycles...

So how do we find prime numbers without eating up too much CPU time? As you probably recall from school: a prime number is a number that can only be divided by 1 and by itself. So the simplest possible algorithm to determine if a given number is a prime is something like:


    function IsPrimeNumber(number) {
        for (var i=2; i < number-1; i++) {
            if (number % i == 0) {
                return false;
            }
        }
        return true;
    }

As you can see this is quite a literal translation of the definition of primes we all learned. And although there are plenty of optimizations you can do to this algorithm, we'll stick to to this very simple version for now.

What happens if we execute this function on a node? It'll work fine for small numbers. But once the number goes beyond a certain threshold, the loop will start eating up quite some time. And since executing JavaScript in the browser is unrestricted but single-threaded, this means the script will eat up all CPU cycles for a noticeable amount of time. As I said before, that is something that I simply don't want to happen on this grid.

So we need to modify our algorithm to not eat up all CPU time. We could optimize it by for example skipping all even numbers above 2. But that would only double the maximum number we can check with this function before the user starts to notice. What we need is something that will make this function work without noticeable effect on the CPU for any number.

My approach to this has been to split the task into many sub-tasks. So instead of having one loop like in the version above, we split it into multiple loops that each run over a smaller range. Let's say we have the following helper function:


    function IsDividableBySomethingInRange(number, start, end);

This function will return true if number can be divided by one of the numbers in the range [start, end>. With this helper function, we can rewrite our original IsPrimeNumber function to use ranges.


    function IsPrimeNumber(number) {
        var CHUNK_SIZE = 1000;
        for (var i=0; i < number; i += CHUNK_SIZE) {
            if (IsDividableBySomethingInRange(number, Math.max(i, 2), Math.min(i + CHUNK_SIZE, number-1)));
                return false;
            }
 }
        return true;
    }

Now if we leave the algorithm like this, we are still executing the work in one go. To actually remove that problem we need to convert the IsDividableBySomethingInRange calls into sub-tasks, which can be executed on the grid separately. The IsPrimeNumber function/task then just has to wait until all its sub-tasks have completed.

The splitting of a task into sub-tasks that can be executed separately and independent of each other is a typical fork operation that I learned back in my college days. We would fork off some work to separate sub-threads, so it could be completed in parallel instead of executing each one in turn. Waiting for the sub-tasks is called a join operation. By creating a fork/join algorithm we're not just making task execution more deterministic, we're also improving parallelism by allowing the sub-tasks to run independently from each other.

So what we really wanted to create all along is a fork/join version of our IsPrimeNumber function/task.

Let's say that we have a fork operation that we can call to somehow fork off a function into a separate sub-task. And let's say that whenever such a sub-task is completed, it will call back into the original task with the result:


class IsPrimeNumber {
    var number;
    var result;
    function IsPrimeNumber(number) {
        this.number = number;
        this.result = true;
    }
    function forkSubTasks() {
        var CHUNK_SIZE = 1000;
        for (var i=0; i < number; i += CHUNK_SIZE) {
            fork IsDividableBySomethingInRange(number, Math.max(i, 2), Math.min(i + CHUNK_SIZE, number-1)));
 }
    }
    function joinSubTask(subtask) {
        if (subtask.result == true) {
            this.result = false;
        }
    }
}

This is a grid-enabled fork/join version of our IsPrimeNumber task. When we execute this task with a very large number on a grid of nodes, the task will be forked into sub-tasks and each of those sub-tasks can be executed on any of the nodes. When a sub-task is completed its result can be joined back into the IsPrimeNumber, which will assure that the combined result is correct.


IsPrimeNumber(2137)
        forkSubTasks
             |
             |--------->IsDividableBySomethingInRange(2137, 2, 1000)
             |                  |
             |------------------+-------------->IsDividableBySomethingInRange(2137, 1000, 2000)
             |                  |                     |
             |------------------+---------------------+---------------->IsDividableBySomethingInRange(2137, 2000, 2136)
                                |                     |                       |
        joinSubTask             |                     |                       |
             |                  |                     |                       |
             |<-----------------+                     |                       |
             |                                        |                       |
             |<---------------------------------------+                       |
             |                                                                |
             |<---------------------------------------------------------------+
             |
true<--------+

If you know a bit about Google's Map/Reduce algorithm (or the open-source Hadoop implementation of it) you will probably see the similarities between join and reduce.

There is still many things that we can improve here (not swamping the grid with all sub-tasks at once, keep track of the number of missing sub-tasks, etc). But essentially we now have a grid-enabled way of determining whether any given number is a prime.

Scrum: story points, ideal man days, real man weeks

2008-02-02T10:43:00.000+01:00

My team completed its seventh sprint of a project. Once again all stories were accepted by the product owner.

While one of the team member was giving the sprint demo, I started looking in more detail at some of the numbers. Because with seven sprints behind us, we've gathered quite some data on the progress from sprint to sprint. That's the velocity, for XP practitioners.

Looking at the data

So far we've had 131 story points of functionality accepted by the product owner, so that's an average of 18-19 per sprint. The distribution has not really been all that stable though. Here's a chart showing the number of accepted story points per sprint:

Although it is a bit difficult to see the trend in sprint 1 to 5, we seemed to be going slightly upward. This is in line with what you'd expect in any project: as the team gets more used to the project and to each other, performance increases a bit.

The jump from sprint 5 to sprint 6 however is very clearly visible. This jump should come as no surprise when I tell you that our team was expanded from 3 developers to 5 developers in sprint 6 and 7. And as you can clearly see, those additional developers were contributing to the team velocity right from the start.

But how much does each developer contribute? To see that we divide the number of accepted story points per sprint by the number of developers in that sprint:

Apparently we've been pretty consistently been implementing 5 story points per developer per sprint. There was a slight drop in sprint 6, which is also fairly typical when you add more developers to a project. But overall you can say that our velocity per developer has been pretty stable.

Given this stability it suddenly becomes a simple (but still interesting) exercise to try and project when the project will be completed. All you need in addition to the data from the previous sprints, is an indication of the total estimate of all stories on the product backlog. We've been keeping track of that number too, so plotting both the work completed vs. the total scope gives the following chart:

So it looks like we indeed will be finished with the project after one more sprint. That is of course, if the product owner doesn't all of a sudden change the scope. Or we find out that our initial estimates for the remaining stories were way off. After all: it's an agile project, so anything can happen.

Story points vs. ideal man days vs. real man weeks

Whenever I talk about this "number of story points per developer per sprint" to people on other projects, they inevitably ask the same question. What is a story point? The correct Scrum answer would be that it doesn't matter what unit it is. It's a story point and we do about five story points per developer per sprint.

But of course there is a different unit behind the story points. When our team estimates its stories, we ask ourselves the question: if I were locked into a room with no phone or other disturbances and a perfect development setup, after how many days would I have this story completed? So a story point is a so-called "ideal man day".

From the results so far we can see that apparently this is a pretty stable way to estimate the work required. And stability is most important, way more important than for example absolute correctness.

A classic project manager might take the estimate of the team (in ideal man days) and divide that by 5 to get to the ideal man weeks. Then divide by the number of people in the team to get to the number of weeks it should take the team to complete the work. And of course they'll add some time to the plan for "overhead", being the benevolent leaders that they are. This will give them a "realistic" deadline for the project. A deadline that somehow is never made, much to the surprise and outrage of the classic project manager.

I'm just a Scrum master on the project. So I don't set deadlines. And I don't get to be outraged when we don't make the deadline. All I can do is study the numbers and see what it tells me. And what it tells me for the current project is that the number are pretty stable. And that's the way I like it.

But there is a bit more you can do with the numbers. If you know that the developers in the team estimate in "ideal man days", you can also determine how many ideal man days fit into a real week. For that you need to know the length of a sprint.

Our team has settled on a sprint length of four weeks. That's the end-to-end time between the sprints. So four weeks after the end of sprint 3, we are at the end of sprint 4. In those four weeks, we have two "slack days". One of those is for the acceptance test and demo. The other is for the retro and planning of the next sprint.

So there is two days of overhead per sprint. But there is a lot more overhead during the sprint, so in calculations that span multiple sprints I tend to think of those two days as part of the sprint.

So a sprint is simply four weeks. And in a sprint a developer on average completes 5 story points, which is just another way of saying 5 ideal man days. So in a real week there is 1.25 ideal man days!

I just hope that our managers don't read this post. Because their initial reaction will be: "What? What are you doing the rest of the time? Is there any way we can improve this number? Can't you people just work harder?"

Like I said before: I don't believe in that logic. It's classic utilization-focused project management. It suggests that you should try to have perfect estimates and account for all variables so that you can come to a guaranteed delivery date. The problem with that is that it doesn't work! If there's anything that decades of software engineering management should have taught us, is that there are too many unknown factors to get any kind of certainty on the deadline. So until we get more control of those variables, I'd much rather have a stable velocity than a high utilization.

A WinSCP replacement for the Mac

2008-01-20T09:42:00.000+01:00

About half a year ago I wrote a piece on the inability to use my iMac as a web development machine. The reason was very simple: the lack of a utility on OSX with a feature set similar to WinSCP on Windows. It doesn't need to have all features of WinSCP, but at least being able to browse a remote SSH/SCP file system as if it is local and being able to edit remote files without being bothered too much by the latency and remote-ness are a must-have for me.

Apparently I'm not the only one looking for something to replace good old WinSCP on OSX, as this post is one of the most viewed and most commented on this site. In the comments visitors suggested many replacements: Cyberduck, Fugu, Smultron, Filezilla and Transmit. I tried all of them.

But none of these tools seem to do what I want, as -until last week- I still found myself at my XP laptop whenever I wanted to work on a site. Until last week? Yes, because about a week ago my trial license of Transmit ran out. And although it's no WinSCP replacement, Transmit is a decent SCP program. So I went to the vendor website to see what it would cost. On the website I ran into their "One-window web development" program, called Coda. After all the tries one more couldn't hurt, so I decided to give it a go.

After installing Coda you have to get used to the fact that it is not a WinSCP clone. So it doesn't have a dual pane Norton Commander like interface, but instead just has the list of remote files on the left hand side. And when you open one of those files -instead of getting the WinSCP editor popup- Coda opens the files in a large panel on the right hand side. Open a second file and it adds a tab on that panel, where WinSCP would open a second popup window.

Looking at it like that, it seems like Coda tries to mimic an IDE more than anything else. But then an IDE for remote web site development, which is exactly what I use it for. I'm now about half-way through my trial period for Coda and I must say I love it so far.

The file transfers are incredibly fast, way faster than with WinSCP it seems. They're also handled more gracefully. With WinSCP file transfers are handled in the main window, which is typically covered by my editing popup. So I have to alt-tab to the main window to see whether the transfer is complete. With Coda the file transfer is indicated by a throbber animation on the file listing on the left. When the transfer is complete, a brief notification animation is shown in the top right.

The editor is decent enough for my needs. It highlights most of the languages you typically encounter during web development and it follows most standard editing conventions. But it could do with some more keyboard shortcuts for faster editing and especially navigation. It also would be nice if it could integrate with external or online help files for supported languages. There's tons of great documentation out there for Java, JavaScript, HTML, CSS, etc. Unfortunately Coda limits its built in help system to the help that comes with the installation. So for anything beyond that, I still need to tab to my browser and search for the help there. This seriously breaks their "one-window web development" mantra and should be addressed if they want to stick to that claim.

That said: you might notice that the editor has lots of features that the internal WinSCP editor doesn't have, so I shouldn't be complaining. Syntax highlighting is great of course, as are other features like the overview of functions in the current file. The fact that I'm asking for features that you'd normally find in real IDE's and not in WinSCP are actually a big compliment to the Panic people: Coda really feels like a web development IDE!

The biggest downside of Coda I've found so far is the price tag. Like pretty much all software on the Mac it's commercial. Panic decided to set the price tag to $79. Which definitely doesn't make it an impulse buy for me. So I'll evaluate it a bit longer and see what happens when my trial runs out. I'll probably go back to my XP laptop and WinSCP. But if I then find myself missing Coda features, I'll give in and buy it. After all... that would mean it's better suited to my WinSCP. Who would have ever thought that would be possible?

Grid computing - using web browsers

2007-12-30T10:45:00.000+01:00

Grid computing is one of those areas that seems to have a magic appeal to software developers. There is something very attractive about taking some relatively simple computers and wielding their combined power to perform seemingly infinitely large computing tasks within reasonable times.

I've also always been attracted to grids. But as many developers, I too thought this type of power was not within reach for me. Only since Google started documenting the "cloud of commodity PCs" that power their vast computing power, does it suddenly seem quite feasible for even just "large" companies to have their own computing cloud.

But my problem remains the same. I don't work for Google, Yahoo or IBM and I'm not a large company myself. So I don't have access to a set of commodity PCs that I can combine into a grid. So for years I decided that I'd never get a chance to work on a grid, unless I'd start working for one of those big boys.

Recently I've been thinking about an alternate setup for a grid computer, more along the lines of the SETI@Home project and all its successors. Those programs all allow home PCs of users all over the world to take part in a giant computer network - a global grid in essence. So the people creating these programs get a lot of computing power, yet they don't have to manage the hardware. A powerful setup.

But such setups already exist. And they have one downside that keeps them from even more mainstream adoption: they require the user to install software to put their computer into the grid. And although the threshold isn't very high, it's still too high for many people. So a lot of potential computing power is not used, because the barrier of installing software is too high.

Now that got me thinking: is there an existing platform on modern PCs that we can just embed our own grid applications in? Preferably a platform that's been around for a few years, so all its quirks are known. And it would be nice if the platform comes with built-in internet connectivity.

Here's the idea that popped into my head: web browsers! They used to be nothing more than HTML viewers, but those days are long gone. Nowadays our browsers are hosting more and more complete applications, like GMail, PopFly and Yahoo Pipes. These applications prove that there is a lot of computing power in the web browser. Is it possible to use the web browsers that people have open on their PCs all the time and turn those into nodes in the grid?

It is a very simple concept: every browser that has a certain URL open is a node in the grid. For a computer to join the grid, they just surf to the URL. To leave the grid again, they navigate away from the URL. It doesn't get much easier than that, right? No software to install, just a page you have to visit. Put it in your favorites in the office, open it every morning when you boot your PC and that's one more node in the grid. From even my own limited reach, I know of at least 5 machines that I could "grid enable" in this way. Those are all PCs and Macs that are on for a large part of the day, just waiting for me or my family to use them. Or that's what they used to be... now I can't stop thinking about them as being nodes in my "web based grid".

If you're a software developer reading this, than your mind probably started wandering while reading the last few paragraphs. Is this possible? How would the nodes get their tasks? How would they report their results back? How would you manage the nodes in the grid? Where do you keep the data that is needed for/generated by the nodes? How do you handle XSS issues? Wouldn't the nodes quickly overload the server that manages them? The list of challenges is seemingly endless and definitely too much for me to deal with in one go.

All I know is that ever since this idea popped into my head, I can't stop thinking about it. And for every problem, I can see at least a few potential solutions. I have no idea whether they'll work or which one is best, but the only way to figure that out is to actually start building the platform.

Oh man... I really need to make this my 20% project. Or more likely... I really need a lot of people to make this their 20% project. Help?

The origin of the name Apache web server

2007-12-22T11:50:00.000+01:00

I read a lot. Not much literature and novels as unfortunately those seem to suffer under my more professional reading habits. I read lots of technical articles, white papers, blog posts and specifications. It's part of what I do to keep up to date with the things happening in the CS field. But in part I also read all kinds of stuff to gain a broader understanding of our profession.

Some of the longer things I read this year include "PPK on JavaScript", "The no asshole rule", but also the venerable "Art and science of Smalltalk". And some colleagues even caught me reading an OS9 AppleScript manual dated somewhere around 1999. They're still making fun of their discovery almost every day, but I don't mind... having read that manual has given me a better understanding of how the now much heralded Apple engineers thought about making an end-user programming language almost a decade ago.

Recently I read the bulk of Roy Thomas Fielding's thesis Architectural Styles and the Design of Network-based Software Architectures in which he introduces the principles of REST. As with any thesis it is a bit too abstract for my taste, but it did introduce me somewhat better to the background and theory behind REST.

Aside from that, I made one stunning discover when I read about Fielding's involvement in the creation of the Apache HTTP server:

At the time, the most popular HTTP server (httpd) was the public domain software developed by Rob McCool at the National Center for Supercomputing Applications, University of Illinois, Urbana-Champaign (NCSA). However, development had stalled after Rob left NCSA in mid-1994, and many webmasters had developed their own extensions and bug fixes that were in need of a common distribution. A group of us created a mailing list for the purpose of coordinating our changes as "patches" to the original source. In the process, we created the Apache HTTP Server Project

Please read that last part again, and again... and again. Until it hits you where it finally hit me. What hit me? Well... I finally understood that the name of the Apache web server might (originally) have had nothing to do with the Apache tribe. The server was created by taking an existing code base and then applying all sort of patches. So in a sense it was a patchy web server. A patchy... Apache...!

Brilliant! In all my years of knowing the Apache web server and the brand that was created around the Apache name, I never realized where it came from.

The Apache website itself has this to say about it:

The name 'Apache' was chosen from respect for the Native American Indian tribe of Apache, well-known for their superior skills in warfare strategy and their inexhaustible endurance. It also makes a cute pun on "a patchy web server" -- a server made from a series of patches -- but this was not its origin.

For the moment I'll take their word for it and accept that the name sounding like "a patchy web server" is pure coincidence. I bet it's also more convenient for them in selling the Apache brand: "we named our web server after its inexhaustible endurance" sounds a lot better than "we named our web server after the fact that it was created from a bunch of unrelated patches".

Why devx needs a better print function

2007-11-17T16:02:00.000+01:00

I like to read technology articles during my daily commute. And since the train is too crowded for a laptop and I don't have an ebook reader (yet), I still print articles that seem interesting to read during the train ride.

A lot of web sites still have a Print button. What happens when you click that button differs from site to site, but it roughly falls into these categories:

Show all pages at once
Many sites break articles into multiple pages. The print version of the article puts all of these pages together again, to allow them to be printed in one go.
Re-layout the site to print better
Tables seem to be notoriously difficult to print. That's why many sites revert to a table-less layout in their print version
Remove navigation elements
Global and local navigation elements are pretty useless on paper. So they're removed from the print layout.
Images - click to see full size version
Some graphics-intensive sites show images of reduced size in their normal view, showing the full version in a popup when you click some link. Since you can't click a link in the Print version, the full size images should always be shown there.

These are some things that I wish more site would do:

Replace animated adds by text adds
I don't mind showing adds next to good content. I do mind the ignorance of including animated adds in a print layout. I'm pretty sure no printer will deal with these in a useful way.
Use images that are more appropriate for B&W
Most people still use B&W printers. So it would be nice if sites allowed the option of replacing their colored images with version that are more suited to printing on a B&W printer.
A common example of this are mostly-black screenshots like from command prompts/shell windows. When printed these really eat through a toner at high speed. It would be nice if a site would allow me to replace those images with ones that are mostly white, making my toner last longer.

That's a pretty long list. And most of these things can actually be accomplished on a website without needing a special print version of the articles. Hiding navigation elements, showing non-animated adds and other layout tricks on a print version can easily be accomplished using CSS media types. And why do most sites still use tables for their layouts? Just remove those tables and you have one less difference between the screen and the print version. And I also think it would make sense to show all content on a single page.

So that actually leaves just one reason for having a Print button: showing full sized images inline. And that finally brings us to the title of the article: the print function of DevX.

DevX is a nice development site that sometimes has very interesting content. And one of the reasons their content is good is that they usually include quite a lot of screenshots and diagrams. This just makes their articles so much easier to follow. On screen the articles show the images at a reduced size. Which makes sense, because the images are often full screen screenshots which would otherwise leave hardly any room for text.

But if you've ever printed an article from www.devx.com you've probably noticed their print versions still only show the images with a reduced size. They're not replaced by the full-resolution version. They're not printed in a larger box. They're not even added at the end of the article, like appendices. The images in the print version are exactly the same as in the screen version: reduced to sometimes a tenth of the the original size.

So whenever I find an article in DevX that I want to read on the train, I start up Word and open the print version in there. Then I remove all tables, because they also don't print very well from Word. Then I go back to the browser and open each image, copy it to the clipboard, paste it in Word and then remove the useless downsized version.

And although I normally like the high volume of screenshots that DevX uses in their articles, this is actually a reason why I'd like them to use less screenshots and more text. Because this conversion to Word is not just a lot of mindless work; I sometimes forget to do it and print a DevX article as is. And by the time I realize what I've done, I'm already on the train. So I do my best and squint my eyes trying to read the text in there.

So there you have it: please DevX fix your @$@%&^# Print function.

Viewing and editing Scrum project management data with Google Mashup Editor

2007-10-06T08:20:00.000+01:00

Welcome to my first post on the Google Mashup Editor. In this article we'll create a tool for entering and storing data using Google's new mashup editor tool. Depending on available time, the evolution of Google Mashup Editor and the availability of alternative tools, I might improve on the basic data management application in later articles.

Scrum project management

The application we'll be creating is a Scrum project management tool. If you don't know Scrum yet, it's an agile project management framework. Please do yourself a huge favor and read this 90 page story about Scrum (pdf). It's a good and fun read and has already won over many organizations to at least give Scrum a try.

My reasons for wanting to create this type of application are many. One of them is that there seems to be no tool that satisfies my needs with the right price tag. XPlanner is good, but very basic. Mingle looks nice, but is too expensive and a real resource hog. ExtremePlanner also looks nice, but again: it seems a bit expensive for my taste. But one other reason is probably more important than the price issue: building this data model seems do-able and gives me a chance to get to know Google Mashup Editor a bit more.

Google Mashup Editor

Mashup tools seem to be a dime a dozen these days. These tools try to take programming to the masses, allowing everyone to create complex web applications based on existing data or logic.

Yahoo was the first big player in this field, with their Yahoo Pipes. They're aiming for a visual programming environment where the user manipulates blocks rather than writing code. Microsoft followed suit with Popfly, an even richer mashup creation environment combined with what seems to be the next generation of their MSN Spaces platform.

Google was the last entrant into this field (if I recall correctly) and the first glances at their entry into the field left me rather disappointed. No drag-and-drop programming, no cool default widgets, just a pretty basic text editor and some basic tags.

But if you look below the surface you can see that Google Mashup Editor (GME) is actually quite different from the other two. Where Yahoo and Microsoft just seem to focus on allowing you to read and combine data from various sources, Google also allows you to create new applications from scratch. In that respect GME is more of an application creation (and hosting) platform than a mashup editor.

Much of these additional possibilities seem so originate from Google's adoption of the Atom Publishing Protocol, exposed through the Google Data (GData) APIs. This API is what makes GME not only a mashup editor, but also a valid tool for creating completely standalone applications. These applications are then hosted on Google's servers, using Google's servers for data storage, using the GME to create and update the applications. Some people might not like to put so much in the hands of Google. But it will certainly lower the bar for creating scalable web 2.0 applications.

That's enough of the background talk. Let's get to work on the application.

Initial data model

We'll start by defining the basic entities and relations in our application. We'll probably expand on these later, but we can get pretty far with just the following.

A project is something on which a team works in sprints to create a product or a release of a product. This is all intentionally very vague, as our application doesn't need to know the details of the projects it manages.

A project has a product owner and a scrum master. Aside from that there are other team members, but we'll leave them out of the equation for now.

A sprint is a time period during which the team implements certain stories. A sprint has a start date and end date and a description of the general goal of the sprint.

A story is a piece of functionality that the team creates. It has a name, a description of how to demonstrate it and an estimate of the effort it will take to create the functionality. Stories can be either user-focused or technical in nature.

All stories combined are called the product backlog. Stories from the product backlog are planned into sprints. So each project has one product backlog and some of the stories in this product backlog are planned into each sprint.

This all translates into the following very simple data model:
Let's see how we can translate this data model into GME.

Creating the project list in GME

The first step is to create a new project in GME. This will show you a nice empty application with just a pair of gm:page tags.


<gm:page title="Scrum Project Manager" authenticate="true">

</gm:page>

Everything for our application will be inside the gm:page tags. If you want your application to have multiple pages, just add some more files to it. But for this application a single page will do.

Getting data into GME consists of two steps: defining the data itself and defining the GUI for it. The data itself takes the form of a gm:list tag:


<gm:list id="Projects" data="${app}/Projects" template="projectList" />

The gm:list tag defines a list of data that is used in the application. In many applications the data will be pulled from an external -RSS or Atom- feed. But we want to store the data inside the application, right in Google's mashup servers.

The data of our project list is stored under the ${app}. This is a location (a "feed" in GME terms) where the data of all users of the application is stored. If we don't want to share the data between users, we can store it under ${user}, which is data that is kept per user. Currently there is no way to have data shared between some users (but not all users of the application), although this feature will probably be added in the future.

To display the data in the list, the page needs a template. A template determines what fields to display and how to display them. It's easiest to use an HTML table, so we'll do that for now.


<gm:template id="projectList">
<table class="gm-table">
   <thead><tr>
      <td width="200">Name</td>
      <td width="100">Product owner</td>
      <td width="100">Scrum master</td>
      <td width="45"> </td>
   </tr></thead>
   <tr repeat="true">
      <td><gm:text ref="atom:title" hint="Project name"/></td>
      <td><gm:text ref="gmd:productOwner"/></td>
      <td><gm:text ref="gmd:scrumMaster"/></td>
      <td><gm:editButtons/></td>
   </tr>
   <tfoot><tr>
      <td colspan="4" align="right"><gm:create label="New project"/></td>
   </tr></tfoot>
</table>
</gm:template>

As you can see we're mixing standard HTML tags, with GME specific tags like gm:text, gm:editButtons and gm:create. Also notice the non-HTML repeat attribute on the second tr (a normal HTML table row). This tells GME to repeat that tr for every item in the ${app}/Projects feed.

If we now compile and test this application, we get an empty table with a "New project" button. Pressing the button adds an empty row to the table, with fields to fill in the values for a product.
Note that editing and creation functionality are for free with GME. Although they're not very flexible, they allow you to quickly get started.

Creating the list of stories in GME

Next is a list of stories for a project. Since stories are always part of a project, we store the data under the feed of a project.


<h2>Stories for selected project</h2>
<gm:list id="Stories" data="${Projects}/Stories" template="storyList" />

This is where GME really adds a lot of logic automatically. The location refers to a child of the ${Projects}/StoriesProjects list we defined earlier. Each project in the Projects list will have its own list of Stories.

This list also needs a template to display it, which is really similar to the one for the projects.


<gm:template id="storyList">
<table class="gm-table">`
   <thead><tr>
      <td width="200">Title</td>
      <td width="75">Type</td>
      <td width="25">Estimate</td>
      <td width="100">How to demo</td>
      <td width="45"></td>
   </tr></thead>
   <tr repeat="true">
      <td><gm:text ref="atom:title" hint="Story title"/></td>
      <td>
         <gm:select ref="gmd:storyType">
            <gm:option value="user" selected="true">User</gm:option>
            <gm:option value="tech">Tech</gm:option>
         </gm:select>
      </td>
      <td><gm:number ref="gmd:estimate"/></td>
      <td><gm:text ref="gmd:howToDemo"/></td>
      <td><gm:editButtons/></td>
   </tr>
   <tfoot><tr>
      <td colspan="5" align="right"><gm:create label="New story"></td>
   </tr></tfoot>
</table>
</gm:template>

Now the only tricky bit we still need to do for the list of stories, is that it needs to be refreshed when the user selects a different project. This is quite easy, by setting an event handler.


<h2>Unplanned stories for selected project</h2>
<gm:list id="ProjectStories" data="${Projects}/Stories" template="storyList">
<gm:handleEvent src="Projects"/>
</gm:list>

This tells the story list to refresh itself when an event happens in the Projects list we defined before. So select a different project will display the stories for that project.

So after adding the story list and adding some projects and stories, our application looks like this:
We can easily do the same for the list of sprints for the project. Since this is really similar to the list of stories, I won't show the code here. If you want to have a look at the code, look at the finished project on http://scrummer.googlemashups.com.

Last is the list of stories for the selected sprint. Note that stories can either be part of the project or part of the sprint. So for now we'll call the first type "unplanned stories". Later we'll want to share the stories between the project and the sprints.

Since the list of stories is -again- really similar to the list of unplanned stories, we won't show the code here. But when we now run our mashup it looks like this:
At the bottom you can see that I am entering a story. This is almost a usable application, at least for entering and browsing the data. To make it something you'd really want your entire team to use for your daily managing of Scrum projects, it would require more work.

That's it for now. If you want to have a look at the finished code or play with the application, go to http://scrummer.googlemashups.com.

Online burndown chart generator

2007-08-25T10:45:00.001+01:00

One of the aspects of Scrum is its focus on transparency - getting all information out in the open. And one of the areas that enables the transparency is the burndown chart. It's a public posting of the progress of the team throughout its current sprint.

On the horizontal axis you see the days of this sprint. The vertical axis describes the amount of work. At the top of the vertical axis is the number of "ideal man hours" we committed to for this sprint. The straight diagonal line is the "ideal burndown" that we're aiming for. The slightly less straight line is our actual burndown. As you can see this chart is from somewhere during the third week of our four-week sprint and we're slightly above target. But things don't look as desperate as a few days before, thanks to some colleagues getting back from Holidays (which it says in the small scribling that you probably can't read).

As a Scrum master I like to post this information as publicly as I can. So just having it on the wall of our team room isn't good enough, since there are many people that don't visit our team room. Ideally I'd like to have the burndown chart projected on a wall in the central hallway of our office, so everyone can see it first thing they come in in the morning. But as a nice step along the way to this, I chose to publish the chart (and the rest of our product backlog) on our project wiki.

In the first sprints I did this by taking a photograph of the burndown chart every morning, right after updating it. I'd then upload the photo to our wiki. The only problem is... uploading them every day turned out to be too much of a hassle. So the wiki actually only got updated once a week. And that's not good for transparency of course.

So this time around we went searching for a simple tool that would lower the threshold of updating the burndown chart on our wiki. We searched for an extension to MediaWiki that allows you to create a chart by just entering the numbers in your wiki text. That turned out to be quite a challenge. There are many charting and drawing extensions for MediaWiki, but they either didn't do what I wanted or we couldn't get them to work on our wiki.

In the end I just gave up and wrote a simple web page that -when fed with the right parameters- will return a PNG image of the burndown chart. You call the page like this:

burndown.jsp?days=1,2,3,6,7,8,9,10,13,14&work=200,170,165,150,125,95

And the page will return the following image:
So the days parameter indicates the day numbers shown on the bottom. I entered all of them for the entire sprint right away. The work parameter is the work remaining. I just entered the values that I know, which is why the green line stops halfway through.

The generated chart is really simple and not very pretty. But it is very easy to keep up to date and that's what counts most. I just add the remaining hours at the end of the URL every morning... and that's it.

Although I consider this generator a stop gap solution until I find something better, I imagine it might also be useful to other budding Scrum masters. For that reason I've put the page online for public use at http://apps.vanpuffelen.net/charts/burndown.jsp. Just click the link and you'll get some usage examples.

Let me know if this generator is useful to you in the comments section. Also let me know if there's something wrong with it and I'll do my best to fix it.

Update (January 1st, 2010): in my company we've created a custom version of this same tool and used that in many projects over the last few years. This public burndown generator has drawn over 60.000 charts in 2009 alone, so apparently we're not the only ones who use burndown charts. That's why I've now updated the tool with the best features that we've added over time at my company. Check the latest version on http://apps.vanpuffelen.net/charts/burndown.jsp for all the features and let me know what you think of them.

Scrum: utilization vs. velocity

2007-08-19T09:30:00.002+01:00

At work we're recently started using Scrum for running some projects. As expected we need to slowly learn the lessons. One of the things we're been having a lot of discussion on recently is the meaning of the focus factor. Let me begin by explaining what a focus factor is, at least in my company.

To determine how much work you can do in a sprint, you need to estimate the top stories. We estimate these stories in "ideal man days" using planning poker. This means that each developer answers the question: if we lock you into a room each day without any distractions, after how many days would you have this story finished?

After these estimates we determine people's availability for the project. After all, they might also be assigned to other projects, if only for part of their time. Even people that have no other projects, tend to have other activities. Like answering questions from customer support or consultants, department meetings, company wide meetings, job interviews with candidates or just playing a game of fusball, table tennis or bowling on the Wii. So basically nobody is available to a project 100% of the time. At most it's 80% - 90% and on overage it seems to be about 60% - 70%.

So the first stab at determining how much work someone can complete is:

available hours = contract hours * availability

But when you're working on the project, you're not going to always be contributing towards the goals that you've picked up. Within Scrum there is the daily Scrum meeting. It lasts no more than 15 minutes, but those are minutes that nobody in the team is working towards the goal. And after the meeting a few team members always stick around to discuss some problem further. Such time is very well spent, but it probably wasn't included in the original estimate. So it doesn't bring the "remaining hours" down very much. I see all this meeting, discussion, coaching and tutoring as necessary work. But work that doesn't bring the team much closer to the goal of the sprint. I used to call this overhead, but that sounded like we were generating waste. So in lieu of the agile world I switched to using the term focus factor. So now we have:

velocity = contract hours * availability * focus factor

So the speed at which we get things done (velocity) is the time we're working minus the time we loose to non-project work minus the time we loose on work that doesn't immediately get us closer to the goal. In the past I probably would have included a few more factors in there, but in an agile world this is already accurate enough to get a decent indication of how long it will take us to get something done.

If there's one thing I've learned from the agile movement and Scrum it's to focus on "when will it be done" instead of "how much time will it take". So to focus on velocity instead of utilization.

Utilization is the territory of classic project management. It's trying to make sure that every hour of every employee is fully accounted for. So if they're programming, they should have a time-writing slot for programming; if they're meeting, there's a slot for meeting; if they're reviewing designs, there's a slot for that and if they're drinking coffee or playing the Wii... you get the picture. Of course there's no project manager that wants all that level of detail. But in general they are focused on what you're spending your time on.

Agile thinkers see this really differently. They say: it doesn't really matter how much time you spend, what matters is when it is done. This sounds contradictory so let's see if a small example can make it clearer what I'm trying to say.

If I tell my boss that some feature he wants will be done at the end of next week, he is interested in only one thing: that it is done next week. If we get it done on time, he doesn't care whether I spent two hours per day on it or whether it was twelve hours per day. I care about it of course, because I don't want to work late every night. And there's also a limit to the amount of gaming I like to do during a day, so two hours per day will leave me bored quickly. But to my boss, all that matters is when I deliver, not how much effort it took.

This is why the focus for Scrum projects is on velocity and not on utilization. So in Scrum you want to know how many hours you still need to spend on a job, not how many you've already spent on it. A classic project manager might be really proud that you worked late all week and clocked in 50+ hours. An agile project manager will note that you reduced the "hours remaining" by 10 hours and nothing more. If you're looking for compliments on all your hard work, then Scrum might not be for you.

Learn more:

Scrum: story points, ideal man days, real man week

a simple tool to generate burndown charts online

Who estimates the stories in Scrum?

Will wireless work?

2007-08-11T18:19:00.000+01:00

Friends often call me a geek. They mean no offense, so I try to take none. As Chris Pirillo once put it: "geek used to be a four letter word, now it's a six figure one". Well... that last part isn't exactly true for me, but that's probably only so because I get paid in euros instead of dollars.

Part of what makes me a geek is the fact that I tend to be an early adopter of new technologies. I got my first always-on internet connection in 1996 paying somewhere around 40 euros for a speed that never seemed to top 1.5 kbps. Yes, that's kpbs for kilobits per second - so about 40 times slower than an analog modem. But it was always on... so I was one of those people that knew they had email a few seconds after the other party had sent it.

I bought an XDA in 2001, which was the first touch-screen phone/pda with an internet connection. It was what you'd call an iPhone these days, although it had to do with a lot less marketing. So for me: six years have brought us better marketing and a multi-touch screen. Still... I'l probably buy an iPhone when they're actually available here. Not because I need it. Just because I'm an early adopter.

As an early adopter you of course run the risk of buying things that will never catch on. Or becoming the involuntary beta tester of a device, which means the technology is not yet ready for prime time. One area where the latter happened to me is with wireless networking.
I bought my first wireless access point and card somewhere in 2001. It wasn't completely new back then, but it hadn't been adopted by the masses yet. So usability and interoperability left something to be desired. USB wasn't as ubiquitous as it is now, so for a desktop PC I had to use a PCI to PCMCIA (now called PC-card and almost extinct) adapter. But hey... it worked... at times. But about half of the time it didn't work and I had to roll out a UTP cable again. I've tried to get a reliable wireless network over the years, but the devices either didn't work reliably or just broke down within a few months of service.

Somewhere in 2004 I just gave up on it and restored the cables to their full and permanent glory. So when whole tribes, states and even countries started using wireless networking, my house is completely wired. It's been like that for years now; first with really long UTP cables running down the hallways. They might not be pretty, but at least they work most of the time.

Two years ago I started using ethernet-over-powerline adapters, which use he powerline network in my house serves as a network. These adapters turned out to be as reliable as using direct UTP cables. So in my study I connect the ADSL modem/router to the powerline through one of these adapters. And in other rooms I connect computers to the powerline through another adapter. And it just works.

The adapters are ridiculously expensive and not very rugged, but they do give me the true plug-and-play experience that I never got with wifi. And even though having blue adapters in many wall sockets is not very pretty, it's a lot better than all those colorful UTP cables lining the floor. So -although expensive- I was pretty happy with it. Most people may prefer wifi, I've been sticking with ethernet-over-powerline.

A few weeks ago I saw a new device from Devolo: a wireless extender. So you plug this adapter into a socket and not only can you plug in an UTP cable, but it also provides wireless networking. So once again I couldn't resist and ordered one. If the whole world is using wireless without problems, I can't stay behind - can I?

Both, either or any

2007-08-03T19:51:00.000+01:00

I recently was adding functionality to a part of our product that was full of lines like this:

lBool_vert_in = this.IsBoathBitsSet(iArrAllow[key], iArrDeny[key],
top.mTDSDefines_CheckInAction);

At first I was just annoyed by the obvious typo in there. I'm sure a boath is something (maybe a very expensive boat) but it has no place in our code. Making a typing error is not strange, it's actually quite normal. But not correcting it when you're copy/pasting it at least a few dozen times is a kind of laziness that I don't like very much.

Later I also needed to know what the function does. At first I thought the name was actually pretty self-describing. But I couldn't explain the behavior I was seeing in a part of the code that invoked this function.

this.IsBoathBitsSet = function(iBit1Value, iBit2Value, iDefineType)
{
return (
(
(iBit1Value & iDefineType)||
(iBit2Value & iDefineType)
) == iDefineType )
}

Now that strikes me as odd... the function name suggests that both bits need to be set. This would normally translate into an && and not into an || like in this code. So instead of just a simple typo, this function name is actually plain wrong. And it also immediately explained the behavior I was getting.

I was thinking of renaming the function. But what shuld I rename it to? At first I was thinking IsEitherBitSet, since it returns true if either bit it set. But that is also not entirely correct. Since it also returns true if both bits are set.

So maybe it should be called IsAnyBitSet. What do you think? How do you translate boolean operators into English?

Why is there no WinSCP for the Mac?

2007-07-29T08:42:00.000+01:00

Ever since Mac OSX came out, I understood that the Mac is the dream machine for software developers. It's a UNIX like system with a great GUI built on top of it. But how come then that even after I've owned an iMac for a few months, I find myself hardly doing any development work on it?

I try to do my personal development work on the iMac. And recently I even found it quite nice for developing a Java application that's been in my head for a few years now. But most of the development work I do at home is not Java work, it's web development work. And for that I still find myself behind my trusty old Windows XP laptop.

The reason for that is that I do most work through the remote editing feature of WinSCP. For those of you that don't know that program: it provides a Norton Commander style (dual pane) interface, with one pane being the local system and the other a remote system. Like a remote web server. And with a single key press you can open any remote file in a very simple local text editor, make a few changes and then save the file back. Their embedded editor might not be the most feature rich IDE, but it just works, it's there when I need it and apparently it's good enough for me to get the job done with.

But then why doesn't a WinSCP exists for the Mac? I know it's called WinSCP, so it's for Windows. But the Mac must have something similar, right? Well, similar yes... On the Mac there is Fugu, which at first sight seems similar. But it's completely not the same for my case.

Fugu doesn't come with a built-in editor and to me that makes a lot of a difference. They chose to integrate with existing editors instead. Which wouldn't be a problem, if they'd actually integrate with the standard OSX editor: TextEdit. But they don't. Instead they offer a whole list of more or lesser known editors, from BBEdit via TextMate to VI and emacs.

Unfortunately I don't have about 75% of the editors Fugu does integrate with. TextMate sounds great, but I haven't bought it yet as I'll need to invest some time to learn to appreciate it. And for the editors that I do have and that Fugu supports, their integration doesn't work. Now if it doesn't work for those editors, do I want to risk installing yet another editor on my system to find out if the integration works there? Well apparently I don't.

While typing this I already noticed that there are of course more options than just Fugu. There's Cyberduck, apparently Krusader also is nice and Disk Order sounds perfect if it would support SCP/SFTP. So I have some options ahead of me. But until I invest the time and find something that works at least as well as WinSCP, my development work on the Mac is limited to Java applications.

--- January 21, 2008 - Frank van Puffelen ---

I might have finally found my WinSCP replacement. Read more about it here.

Don't mistake the means for the goal

2007-07-24T20:28:00.000+01:00

Last weekend my wife was balancing her checkbook. Not that we have a problem making ends meet, but after an embarrassing situation at an ATM she really wanted to figure out why these things sometimes happen to her. I agreed to help her, as long as she would do the actual work herself and draw her own conclusions.

As it quickly turned out, she had no idea where her salary is going. So I suggested she'd categorize her spendings based on what she could find in the tele-banking application. She went to work on it. I knew that something was going wrong when I entered her room three hours later and found her still busy categorizing.

It turned out that she had found this online checkbook application, which could show all kinds of charts based on your spendings. All you had to do was upload a dump of the tele-banking information and categorize it using their "super friendly" web interface. Three hours later and she still wasn't done categorizing just a few months worth of spendings.

I asked my wife whether this was really worth the time. After all, categorizing spendings was not the goal. It was just supposed to be a means to quickly figure out where her money was going. She assured me that it was worth it; she was almost done and then she would know.

An hour and a half later she was indeed done and proudly called me in. "See. I spent this much on our Holidays. And that much on gas." Very interesting information I'm sure, but not what I was interested in. "I'm pretty sure I also paid part of those Holidays", I said. "Does this mean that I didn't pay my fair share?" She started clicking on the charts frantically. "It must be in there?" She couldn't find it and had to go back to the tele-banking application to look it up through a quick search.

Forty-five minutes later we had pretty much figured out where her money is going. And although the online checkbook gave some nice charts, we frequently had to go back to the "source" (the tele-banking application) for additional details. The online checkbook seemed like a nice, quick way of doing the categorizing. But when it turned out it wasn't very fast, she should have stopped entering data into it. Categorizing the spendings was just a means to figure out what type of things her money was being spent on, it wasn't the goal.

It turned out that I indeed didn't pay my fair share of the Holidays and that she was paying for some insurances that really should come from our shared account. With that and a solid resolution to determine where all the cash withdrawals are going she's pretty sure that those embarrassing ATM incidents should not occur anymore. Or at least not too frequently...

The Puf Principle

Block font

What I use my iPad for

Games

Surfing

Email

Not for reading

Not for video

Who estimates the stories in Scrum?

The progress

Analysis

How stable is the progress

Who provides the estimates?

Conclusion

Java Particle Systems part 2: adding some life

Introduction

Gravity

Grow then shrink

Mouse control

Summary

The code

See also

Seven rules to keep your twitter followers

1. Yet another link retweeter

2. Your opinion matters to me

3. Challenge me

4. Don't forget I'm eavesdropping

5. Give me context

6. Tell me something I can use

7. I don't need more spam^H^H^H^H marketing

8. I read your blog already

And yours?

Safari 4 on Windows crashes when you provide incorrect credentials

Particle systems in JavaScript

Pod racers

parts of a particle system

Parts of a particle

Particle behavior

lifetime management

visualize the particle system

demo

next steps

the finished script

mstsc /console doesn't work anymore

Which keyboard is best on mobile phones?

Palm Treo

GPhone

iPhone

Optimistic locking vs. pessimistic locking

Using PageRank to determine the most powerful people on LinkedIn

Three ingredients to a better bug report

What did you see?

What did you expect to see?

Why do you think your expectation is better than the current behavior?

Prior art

Summary

Finding primes using a grid of browsers

Scrum: story points, ideal man days, real man weeks

Looking at the data

Story points vs. ideal man days vs. real man weeks

A WinSCP replacement for the Mac

Grid computing - using web browsers

The origin of the name Apache web server

Why devx needs a better print function

Viewing and editing Scrum project management data with Google Mashup Editor

Scrum project management

Google Mashup Editor

Initial data model

Creating the project list in GME

Creating the list of stories in GME

Online burndown chart generator

Scrum: utilization vs. velocity

Will wireless work?

Both, either or any

Why is there no WinSCP for the Mac?

Don't mistake the means for the goal