Duncan Pierce duncanpierce.org

For 2 decades, I've worked as a software developer, as well as training and coaching developers in the disciplines of software development.

Through my work and through conferences, workshops and other events, I come into contact with hundreds of software developers every year, and I get a good overview of what's going on in our profession.

And a "profession" it most certainly is. While computer programming is relatively easy to pick up, writing software commercially on any appreciable scale takes decades to master.

There is just so much to know, and so many practical skills that take thousands of hours of practice to really get to grips with. As a discipline, software development is every bit as complex and challenging as physics, medicine or teaching, and requires a lifelong commitment to learning, practicing and improving.

Beyond the obvious and many technical skills developers need to keep up to date with, software development also requires a high standard of general education in maths, written and verbal communication skills, and strong-enough general knowledge from which to quickly build a computable understanding of a wide range of problems that software is used to solve.

Year on year, I see the standard of software developers joining the profession slipping. They tend to be not quite so good with the technical skills, not quite so good at communicating, not quite so proficient with maths and logic, and not quite as well-informed and worldy-wise. Companies that hire young software developers are finding it harder and harder every year to recruit people of the right calibre. I work with some clients on recruitment, and I've been finding that - on average - perhaps 1 in 100 software developers that might apply for a job are really up to snuff. You have to kiss a lot of frogs to find your Prince Coding.

The impact of this on the businesses who rely on software developers, which, these days, is pretty much all of the FTSE500, can be profoundly damaging. While the media cries over the lack of good talent available to the computer games industry and "digital creative media", they completely overlook the fact that the bulk of our economy has been "digital" for decades - with computers playing an increasingly central role in the operations of all organisations that have to do things on a large scale.

If Acme Supermarket Plc has to wait to start a software project until they've found a team good enough to deliver, then Acme have to wait to roll out changes to the way their business operates. I've seen first-hand household British brands brought to their knees by delays in new IT. These days, your ability to create and adapt software systems is a very limiting factor on the ability of your business to adapt and stay competitive.

The importance of software development to the UK goes far beyond business, though. Our entire scientific and engineering base is very heavily reliant on software that has to be written specially. Increasingly, we find the lack of software development skills in science is holding UK R&D back. Our future prosperity and quality of life depends on our ability to discover and to innovate.

For all these reasons, I see programming in schools as an imperative. We desperately need new talent. Not just games companies like Eidos, or special effects companies like The Mill. We all need the decline in Britain's software development capability to be reversed.

My theory is this: hidden among the millions of children in school right now are maybe 1,000 who would make amazing software developers. And they, and we, may never know it. Not only could they be robbed of a potentially very rewarding and personally fulfilling career, but we would all be robbed of the fruits of their labour.

1,000 really great software developers entering the profession every year could, over a decade or so, tip the balance.

Firstly because the best software developers tend to be an order of magnitude more productive and creative than the average, so 1,000 great developers could achieve the same as 10,000 so-so developers.

Secondly because the more great developers come into the profession, making it easier for employers to find good people, the more the so-so developers will be forced to raise their game to stay in work.

We need to find out who these 1,000 kids are. We need to give as many children as possible the chance to find out for themselves whether programming is for them. For the vast majority, it won't be. But for that 0.1% who may find they really enjoy it, and have a real talent for it, we would be shooting our economy in the foot by not giving them the chance to discover programming - and ultimately software development - for themselves.

And once they've been identified, we must move mountains to fuel their passion and remove all obstacles to their development as, well, developers. This will require focused, targeted collaborations between practitioners like me, and the software development community of which I'm a part, as well as schools, teachers, parents, employers, the media, and everyone else who has a stake in the outcome.

I don't much care for kids being made to learn "computer science" (which is not the same thing, by the way). That sort of thing can come later, for those children who - fuelled by burning curiosity and desire to do more, go further and build better software - seek it out. It should be there waiting for them.

First, we need to give as many children as possible the chance to try programming. Not for its own sake, but because you can make cool stuff with code. Programming as "making cool stuff" should be the dominant model until that 0.1% reveal themselves, and then we can start talking theory and maths and discipline.

For my own part, I'll be focusing this year on getting kids coding, by finding support for teachers who want to learn more through a teacher-practitioner coaching programme, and also by devising "cool projects" for kids and for schools with themes ranging from pop music to alien hunting.

I'll also be working with the developer community to start fleshing out a framework for identifying and nurturing that little nugget of raw talent that may emerge.

It's the question that comes up everytime I introduce someone to Test-driven Development: "But what about performance?"

The thing about TDD is that adage "be careful what you wish for" applies. The solution we end up with is constrained by tests. There may be a million and one ways of achieving a goal, and some will perform better than others. The trick with TDD is to ask the right questions.

What I like about TDD - and similar precise approaches to defining requirements - is that it forces us to be explicit and unambiguous about what we want from our software.

So, my stock in trade reply to the question "But what about performance?" is "yes, what about performance?"

Software performance has different dimensions, and if it's important then we need to define exactly what performance we require in specific scenarios. A great way to do this is using non-functional tests.

There's the dimension of time, for example. How long should it take for the code to run?

Imagine a search algorithm that looks for a customer name in a sorted list. We could just loop through the list, and if there are only 1,000 customers and the occasional search, that might be fine. But if there are 10,000,000 customers and users are frequently searching, then a simple loop probably isn't going to cut the mustard.

We can constrain our search algorithm with a basic timing test, like the one below, that makes it explicit that our worst case search - the customer we're looking for isn't in the list - should take a maximum of 1 millisecond to complete.



Execution time is only one dimension, of course. What if we need to constrain the memory footprint of our code while it's running? In Java, we can use the JVM to get information about memory usage, and we can create a multithreaded test to monitor how much more memory is being eaten up as our code executes. Let's imagine we need to constrain the memory footprint when sorting our list of 10 million customers by name, forcing us to use an in-place sorting algorithm that uses up a maximum of another 10KB of memory:



And here, with massive caveats for my less-than-amazing knowledge of the Java Runtime (I make no warranties, the value of shares can go down as well as up, etc etc):



Leaving aside the fact that my brute-force method for calculating memory footprint is a bit hokey (and on running the tests several times, quite variable, it seems), the basic idea is hopefully useful. No doubt some fine fellow will point out a much better way.

You may be able to envisage now how we could use tests to explicitly constrain other non-functional runtime qualities of our code. But we can also often find ways to constrain code at design time, too.

We might have a requirement that our methods should be short and simple. Static code analysis tools like XDepend and Checkstyle can give us hooks into the structure of our code and enable us to create tests that, when code fails to live up to our quality standards, alert us to that fact early enough to do something meaningful about it.

Using executable tests, we can steer our software between acceptable limits of performance, scalability, portability, maintainability, and a whole heap of other -ilities we might care about.

But what about the more, how shall we put this, etheric -ilities, like usability, accessibility and so on? These things tend to be pretty ill-defined and qualitative. Can we make them explicit and testable, just like execution time or memory footprint?

I believe that we can, and to without reason, because I've done it and seen it done. We could, say, define a test that fails if a carefully selected group of target users (e.g., legal secretaries with more than 2 years Windows and web browsing experience), when presented with our application for the first time, fail to get their heads around it fast enough to complete certain tasks we set them within a specified time, without any help or documentation.

With a bit of imagination and lateral thinking, it's possible to meaningfully test many more software qualities than we usually do. And my experience of non-functional TDD is that we tend to get what we have tests for, and we tend not to get what don't have tests for. So agreeing executable non-functional tests tends to lead to better non-functional software quality, if it's done well.

As I warned before, though, be very careful what you wish for.

I get asked this question a lot, but it never occurred to me to write down my usual answer.

How do we evaulate shrink-wrapped software against our needs?

Well, that's easy. You still need to do the usual business requirements analysis. Identify who will be using this system, and what their goals will be for using it. In the good old days, we called these "Use Cases". Yep, even if you're buying and not building the software, you still need use cases.

The next step is to flesh out the design of your use cases, as we might normally do, by describing how the user interacts with the software to achieve their goal.

When we're describing software we haven't built yet, this is design. When we're describing how we'll use software that already exists, this is a process of validation. Can the user achieve their goal using the software we're evaluating?

Even with the most feature-rich packages, we tend to find we don't get an exact match. It's not always possible to achieve every user goal using the software. So as we validate the software against our use cases, we may identify gaps. There are almost always gaps.

The next question we need to answer is can we fill those gaps? Let's say we're evaluating Microsoft PowerPoint for our training business. It doesn't do everything we need out of the box. Let's pretend we have a use case where the trainer needs to populate a slide with an organisation chart showing the reporting structure of the group attending the course. She has a spreadsheet with those names listed in alphabetical order and with information about who reports to whom. using PowerPoint's built-in scripting language, Visual Basic for Applications (VBA), it is indeed possible to take that information and automatically generate an Org Chart.

So that gap could be plugged, with some work. Write a reminder about it down on a blank index card. This is now a potential "User Story" for some programming work that would need to be done if we went the PowerPoint route.

Of course, people identify gaps in software all the time, and it's possible that someone somewhere has already found a solution to plugging some of your gaps with handy tools and utilities. Google is your friend here: search for solutions before you think about reinventing the wheel. If you find one, and there's money involved, write down roughly how much on the index card.

Finally, don't forget the non-functional requirements. A package may offer the right features, but it may not be able to handle a high-enough volume of users, or it may not be secure enough for your purposes, or it may take a long time for users to learn. Evaluate thye software against these criteria, too. Be as explicit as you can. Handwavy requirements like "it must be scalable" aren't very helpful for validating software. What do you mean by "scalable" - a certain number of users at any one time, or a certain number of transactions per second, or the ability to run it on more servers?

All too often, businesses buy a solution and then validate that it does what they need - often by actually trying to roll it out. Whether buying or building, the key is to have clear, testable requirements and to validate the software against them. Don't be seduced by their sales patter and let them lead you like a donkey to the slaughter to their feature list. What their software does is far less important than what we can do with their software.

It's been announced today that Bletchley Park has a new CEO, Iain Standen, a recently retired Colonel from the British army.

Before he disappears into his lovingly tended garden and a life of well-deserved leisure, I just wanted to pay tribute to Simon Greenish, who has been the key factor in the turnaround of Bletchley Park's fortunes since he took over in 2006.


Simon shows Colossus to kids' TV genius Johnny Ball, while the late Tony Sale does his famous velociraptor impression

Five years ago, Bletchley Park's finances were in bad shape. There was a very real danger of it folding due to lack of funds. I know first-hand how hard Simon has worked - harder than most people could even begin to imagine - and he's received very little recognition outside of the staff and volunteers there.


Two maths legends. And Alan Turing.

For every pound raised by enthusiastic supporters like myself, Simon and his amazing team have raised £50, working madly long days and suffering untold stresses and strains to secure funding from a range of public and private bodies. It's this work - often done in secret, and often not reported - that has earned Bletchley Park it's future, and for that I'm hugely grateful. I have no doubt that, were it not for Simon Greenish, Bletchley Park would now be in the hands of property developers.

As it is, Bletchley Park's future looks very positive indeed. Like the best boy scouts, Simon has left it in a much better condition than he found it, and - while there'll be many more challenges to come for Iain and the team - things are definitely on the up.

So, I raise a glass to you, Dr Simon Greenish, saviour of Bletchley Park. Top bloke, that Greenish chap!

I'm sure I can't be the only software developer who thinks the way we hire for our teams is completely f**ked up in a lot of cases.

When I compare it to how, say, bands hire musicians it seems we care a heck of a lot less about who we work with. I know some of you will complain that hiring is often taken out of our hands, and therefore how can we to be to blame? Well, if we let such important decisions be taken out of the hands of the people who can tell the difference, then maybe it's our fault all the same.

One of the most technically demanding jobs in music is playing drums for prog rockers Dream Theater. After their previous drummer and co-founder, Mike Portnoy, quit the band, the search began to find a replacement. These were very big shoes to fill. There may only be a few dozen drummers in the world who play those parts well enough, and fewer still with the right temperament to fit in with a long-established band.

How does a band like Dream Theater go about finding that person? Do they pick up the phone and call Drum People Inc. and ask for a prog rock and metal drummer to start rehearsals on Monday? I can just see it now - the agent persuading traditional jazz drummers to add "prog", "metal" and "double bass" to their CVs.

And did Dream Theater leave the hiring process to their Director of Human Resources - a tone deaf, musically illiterate Justin Bieber fan who has not even the most basic appreciation of what it is Dream Theater actually do?

No, that would be silly.

How does a band like Dream Theater go about finding the right drummer, then? Well, it starts with drumming, oddly enough. That is, they listen to a lot of audition tapes sent in by drummers from all around the world. They must have received hundreds of them, so that's a few days of dedicated listening. Luckily, you only have to hear a drummer for a few minutes to know whether they'd be up to the standard of someone like Mike Portnoy. But it's still a lot of listening.

But this is a critically important decision for the band. In a 20+ year career - mostly with the same brilliant drummer - what's a few days listening to audition tapes?

The ones who stand out go into the "maybe" pile. And, of course, most drummers of any note have things like web sites and blogs and Twitter accounts and YouTube channels these days, and they and their bands are discussed in various message boards and on other kinds of social media. That is to say, they have a searchable public profile, and catalogue of music we can check out, and a reputation.

They can even watch drummers rehearsing for their audition.



By checking out the portfolio of work the drummer's done and researching them online, Dream Theater were able to whittle down hundreds of auditionees down to the three drummers they thought would be worth auditioning in person.



You can tell a lot about someone from their work, but what you can't tell is how they'd work with you. That's really what an audition should be about - not "can you play this tune?" but "what does it sound like when you play this tune with us?" and "can we write a new tune together?"

Auditions aren't about technical ability - that should already have been established before you even think of inviting that person in and taking up everyone's valuable time. Auditions are about the dynamic: how does this band perform with this person integrated into it? What would Dream Theater with Peter Wildoer, celebrated Swedish death metaller and jazz fusioner, sound like? Possibly a bit heavier - a bit more technical death metal and a little less Asia or Yes? That might be why he didn't get the gig. There's no doubt he's up to the standard of Portnoy. Wildoer's one of the best. He just wasn't the fit they were looking for.

Most notably, when I watched the band's video of the auditions, Wildoer was not spitting feathers because they'd rejected him. To him, being one of one only three drummers in the world even invited to audition was a great experience. God knows I'd be flattered if they even listened to my audution tape for more than 30 seconds!

He was at their level musically and technically, and they weren't wasting his time. The audition video's been viewed 1,000,000 times. That's good exposure and good experience. To get to play and jam with musicians of that calibre, and with that level of passion and professionalism, and have a good time to boot, is not really a waste of anyone's time.

If a great team flattered me by inviting me to "audition" for them, and my technical ability was not under question because we'd already established I was technically good enough to work with them based on my portfolio and reputation, and they said "Hey, Jason, come and spend a day with us and let's play some of the old tunes together and maybe jam a few new ones", and they paid my expenses and put me at my ease - then if I didn't get the job, I'd still feel like I got something out of it.

Dan North (@tastapod on That Twitter) rightly points out that TDD is neither necessary or sufficient for good software design.

I make no bones about it. TDD is not the only way to achieve high quality, reliable, maintainable code.

He tweets "Some of the worst code in the worst codebases I've ever seen was 'strictly TDD'."

Flip it over, for balance, and I can also honestly claim that some of the most horrendous code I've ever seen was not test-driven. Based on admittedly small-scale studies, TDD'd code tends to be simpler. But good design is about a lot more than that.

Which is why my TDD course focuses on these aspects of the discipline - the simplicity, the refactoring (LOTS of refactoring!), readability, minimising duplication, localising dependencies ("Tell, Don't Ask"). These basics should be core to any TDD text or course. If more programmers paid attention to just those basics - simplicity, readability, duplication and localising dependencies (darn it, if they just put more effort into readability!), the software world would probably be a better place.

But I also make it clear that those basics will only get you so far, which is why I have 3 other courses that all - from one angle or another - focus on software design. All of my courses are really design courses.

In the refactoring course, we learn about common code smells and how to safely eliminate them. In the OO Design course, we learn about SOLID and other dependency management principles, as well as how to more objectively identify dependency issues, and then refactor the code to address them.

In the Agile design course, teams use a bit of collaborative up-front to establish a shared understanding of the basic architecture and co-ordinate their efforts to flesh out and implement different features of the same software. When they implement it, they test-drive it, and continuously integrate, and refactor and etc etc.

It all dovetails nicely together to hopefully present a more rounded picture of software design in an Extreme programming sort of approach. But altogether they are still not sufficient.

There's a knack to good software design, and a whole heap of other stuff you need to know. I've been a student of software design for decades, and I'm just getting started.

So Dan's quite right. You need more than TDD, and there are other ways of achieving good design.

Still with the Binary Search problem from the previous blog post. Let's go back to where the problems started:



I think my mistake was to proceed from here in a linear fashion. It is, after all, a divide-and-conquer algorithm. Binary Search works a bit like splitting a deck of cards and then looking at the card there in the middle. if that's not our card, we ask whether our card is higher or lower. If it's higher, we discard the middle card and the bottom of the pack. then we split the top half in the middle, rinse and repeat until the card in the middle is the one we're looking for. (Or we run out of cards.)

So it feels intuitively to me that I should be "splitting the pack" with each new test case, and that, for here on in, we should be finding the number we're searching for.



The simplest implementation I could think of was:



This was encouraging. It's starting to look a bit like the iterative Binary Search algorithm I was going for. We now have this "middle" variable, and we do the comparison there.

Next, I thought, what if our item is not in the middle of the list, but in the middle of the top half of the list?



Which brought me to introduce a movable "bottom" to the array and introduce the loop (since I now had two iterations):



And what about if our item is in the middle of the bottom half of the list?



This test doesn't terminate if you run it with the existing model code, by the way. Just thought I should mention that.

I can pass this test easily by checking if the key is lower than the middle number, and by introducing a movable "top" for the next iteration so we can discard the top half.



Now, I could be wrong - it is late on a Saturday night, and I've been staring at a spreadsheet most of the day - but that's Binary Search, isn't it?

It still feels a little hokey. Especially the leap to iterating. I suspect there are smaller steps in between some of these steps, and will be looking for those next. But the gist of it feels right. The "knack" to test-driving a divide-and-conquer algorithm might be to divide-and-conquer with your tests, rather than follow a naive linear path.

The next big question is: how would I know, if I didn't know I was shooting for a non-linear algorithm, to follow this kind of roadmap?

I think the answer lies in doing some thinking about performamce. If we're looking at potentially larges lists, then a time complexity of O(N) sounds undesirable when N is a big, big number. Basically, we use our experience and a bit of design ken. No surprise there, as that's the key to cracking any problem in software design. There's no algorithm for algorithm design. I might have interspersed these tests with the kind of complexity tests I demonstrated in the previous blog post.

Progress?

ADDENDUM:

For those of you who favour recursion over iteration, it's a 2 minute job to refactor to:



Good job we wrote those tests, eh?

So I was noodling about test-driving an implementation of Binary Search (to see how a divide-and-conquer algorithm could be TDD'd), and something interesting came up.

My usual starting point is the simplest failing test I can think of - that is, the one that would be easiest to get passing. In this case, I started with the case where the item I'm searching for isn't in the list:



Triangulation begins by very simply returning the value it expects: -1



So far, so easy.

Then I looked for the next nearest failing test. I figured an array of only one element, where the element is the item we're looking for:



And pass it thus:



The next nearest failing test I could think of was an array of 2 numbers. The first number couldn't be the key value we're searching for, because our code would already pass that test, so I make it the second item in the array:



(With a little bit of clean-up to localise the knowledge of how to interact with our BinarySearch object, of course. I'm a good Boy Scout.)

Now here's where I hit a snag. The simplest way to pass this test I instinctively thought of was to use a loop. This follows Uncle Bob's "Transformation Priority Premise", sort of. I went from a hard-coded value to looking at a variable and an IF, to using a loop. So, to my TDD instincts, this feels right:



It's simple. It passes the tests.

And that's just it. It will now pass all of my tests. Provided the array is sorted and contains unique integers (my pre-condition), any array and any key value will be either correctly found, or not found. No more failing tests.

But I ain't done yet. This obviously isn't Binary Search. If I created an array of, say, 1,000,000 integers, it might have to do 1,000,000 comparisons in the worst case. In terms of algorithmic complexity, it's O(N) complex. I want it be O(log2(N)).

But where TDD's concerned, I'm done. No more failing tests. Surely?

I've come up against problems like this before, and what I will ususally do when we realise that an implementation is going to be too inefficient is to start writing tests about the performance or efficiency we require.

As folk on That Twitter were pointing out (over and over again - curse you Twitter for having such a short memory!), we can actually make assertions about algorithmic complexity by finding a way to expose the number of comparisons our search does.

For example, how many comparisons should it need to do to discover that some key value isn't included in an arbitrary array of 10 integers? It should be no more than 4.

A simple brutish way of exposing this information could be to add a feature to BinarySearch that tells us how many comparisons it did for the last search:



And we can test it like this:



We could, of course, do something more sophisticated using mock objects that wouldn't require this slightly unsightly addition to the BinarySearch interface. But with integer arrays, we're a bit stuffed, frankly. So ugly and brutish it is. (I actually used a test spy injected into the constructor that collected the comparison count when I first did it, but then thought "oh, sod it - it's just integers!"). Mocks present another problem. If the things we're comparing are mock objects, there's some considerable hill-climbing to do to set up a sorted collection of unique comparable mock objects. Fiddly, and I'm not sure it's worth the effort.

I can now write tests that force me to optimise my search implemention until it's properly O(log2(N)). But these new tests don't give me any real clues as to what that final algorithm could look like, just that it has to be X efficient. So, while it's a step towards test-driven algorithm design, it's not test-driven algorithm design. The tests don't tell me what code I need to write.

Now, this is an artificial exercise. I know I wanted to end up with Binary Search. If I know the answer I want to get, technically it's not "design" and not really TDD. But that wasn't my goal. I wanted to try and figure out how I could have test-driven Binary Search, had I not known that what I was going to end up with that specific algorithm.

Is TDD a useful tool for algorithm design? To be honest, I've come away from this little exercise with my doubts that it is, in this case.

Some algorithms wear their hearts on their sleeves, when the internal design of the algorithm is a close match for it's externally-visible behaviour. e.g., an algorithm to convert integers into roman numerals (or Farenheit into Celcius, etc).

Binary Search, on the other hand, is a secretive little bugger. You can't tell it's Binary Search just by what it does, and therefore it's hard to drive its design from the outside.

Some argue - rightly, I think - that the O(log2(N) requirement is a feature of Binary Search. I'm inclined to agree, but it doesn't point towards that algorithm, any more than "I want to pay income tax less than X%" points towards moving to Switzerland. The requirement and the implementation are more orthogonal.

Now I took a different tack doing it again this morning, and made a little progress towards understanding how, had I been tasked with implementing an efficient search algorithm I might have let a different sequence of tests lead me towards Binary search, or some other divide-and-conquer algorithm. So I haven't lost all hope yet that TDD isn't going to help in designing new algorithms. And certainly adding tests that fail if our algorithm isn't performing as we need it to can help constrain us, steering us away from inefficient algorithms early in the process.

It's been a fascinating process, and as I noodle with other kinds of well-known algorithms I'm fully expecting now to come up against other challenges. And while it can be frustrating, I take that as a sign that I'm learning something.

My hope is that having 250 smart and passionate programmers working on algorithms and the cross-over with craftsmanship and Clean Code will throw up all sorts of interesting problems, and potentially new solutions. To quote Simon Pegg in Star Trek: "It's exciting!"

Finally, as an aside, here's another approach I've used many times to constrain test-driven designs with non-functional requirements, which is very simple and equally brutish, but allows us to think in terms of things a user will experience (like "what's the maximum time this search can take?"):





I've also used similar techniques to write tests that fail if a certain memory footprint is exceeded, and I'm sure it could be applied to other kinds of performance requirements. IF YOU HAVE THEM - I should add. (Don't go making them up and optimising willy-nilly!)

As 2012 approaches, my mind lingers on matters of faith. More particularly, our limitless capacity for believing the impossible, even in the face of overwhelming evidence that what we believe isn't true.

A classic case study in hypercredulity is that of self-proclaimed UFO contactee Billy Meier. Billy was a Swiss farmer who, in the 1970's, came forward with tall tales about visits from beautiful - and suspiciously Swedish-looking - aliens who took him into their spacecraft and whisked him away for tours of the universe (and, eventually, the distant past).

Meier claimed the visitors came from the Pleiades (astronomers may know it as Messier object M45), and he supported his stories with a series of very spurious - and really rather comical - still photographs and home movies.

Putting aside the fact that the Pleiades star cluster is dated by astronomers as being between 75 and 150 milions years old - mere newborns compared to our Sun - and that it's very unlikely such an advanced technological civilisation could have emerged in so short a time, Meiers physical evidence has been roundly debunked with little effort.

Photos of Pleiadian visitors have been positively identified as stills from Meier's favourite TV show, Rowan & Martin's Laugh-in (and, yes, I too can believe that Goldie Hawn was sent from heaven, but certainly not from the Pleiades.) Photos of dinosaurs allegedly taken in Earth's prehistoric past (as well as dinosaurs photographed on other prehistoric planets - yes, his claims really were that naive) have also been positively identified as illustrations from a children's book.

His flying saucer stills and movies not only look laughably fake, but have been very accurately reproduced by amateurs with no model-making or special effects experience using materials that were readily available to Meier. One especially keen debunker even took the trouble to perform his recreations using only one arm, as Meier must have done.

Meier's own ex-wife has denounced him as a fraud. And the list continues, piling up a mountain of evidence that the Billy Meier UFO contact case is a hoax - and not even a very good one.

And yet, millions still believe every word of it. You can point them directly to the evidence of a hoax, and they'll believe his story even more. The more they see that contradicts his claims, it seems, the more fervently they believe them.

Faith is like one of those knots that, the more you struggle to get free, the tighter the knot becomes. Thousands of years of organised religion have conditioned many people to see faith - unquestioning belief without evidence (or even in the face of contradictory evidence) - as a virtue. When people can view reality through a lense that tells them that when the facts contradict the belief, they should believe it even harder, we get ourselves into all sorts of pickles.

It's as though we have some special compartment inside our brains that's insulated from things like reality and logic, where we can ring-fence off certain cherished beliefs so that the real world can't get at them. One wonders whether chimpanzees have a similar ability.

And, like it or not, I have to remind myself that:

a. I have one of these compartments, and should therefore be more mindful about which of my beliefs enjoy this special protection, and...

b. There are things people believe about software development that have obviously somehow ended up in their reality-proof mental "panic room", that are impervious to evidence or reason

For example, the whole Mythical Man-Month thing. We have a pretty overwhelming body of evidence now that adding more developers to a team won't produce the software any faster. And yet managers still have this knee-jerk reaction when the schedule's slipping to hire more devs. I've had this discussion with managers many times, presenting the evidence and reasoning about the causes logically and dispassionately. But usually to no avail. Most managers believe that a bigger team will get more done, and the more that belief is undermined, the harder they cling to it. So much so that, after adding a few more developers makes the situation worse, their solution is to hire even more. And then more again, until we've got a team of 100 developers tripping over each other's feet and struggling to produce the useful output of a team of 6.

Earlier this month, I posted a link to the proceedings of a DoD workshop on "software engineering" from the late 1960's, in which many of the issues we face today were discussed, and solutions proposed that bear an uncanny resemblence to what experts recommend today. In the intervening 43 years, little has changed except the scale of the problems. If small and frequent releases, close customer collaboration or test automation was considered advisable then, it's 100-1,000 times as advisable now. And yet, as an industry, we still resist, clinging on to the same old misconceptions that were tripping teams up since the dawn of modern computing, and in the face of a large body of evidence and experience that's grown in the meantime.

To me, it's surely not that hard. Just as I can look at Billy Meier's photos and see them as obvious and derisary fakes, I can see so very clearly the handful of genuinely simple things software teams could literally just decide to get right that would boost their chances of success enormously.

But I have to accept that there are people who look at what I look at, and just don't see it. They can't. It's like trying to make someone to see the tiger in a Magic Eye picture. There's a part of their brain that won't allow it, and my well-meaning attempts to penetrate their Fortress of Woo and ground projects in reality will more often than not just make them believe what they believe even more.



(And, no, the world very probably isn't going to end next December. In case you were wondering.)

I promised myself I wouldn't work until the New Year, but just wanted to put this idea out there.

June 14th is a date to put in the diary for software craftsmen. The fourth Software Craftsmanship conference (SC2012) will be at it's spiritual home, Bletchley Park, just a week or so before the 100th anniversary of Alan Turing's birth.

As well as Turing, education's very much on my mind. So I've been kicking around an idea that could address an important educational gap in UK software development - the yawning divide between theory and practice.

I work closely with many employers, large and small, who are looking for great software developers. Many - probably most - UK developers are self-taught, including me. What employers tell me is that the self-taught programmers tend to hit the ground running, compared to programmers with computer science degrees (a surprising number of whom struggle to program at all, having had so little hands-on practice during their studies.)

But I also note from my own experience that many self-taught programmers have big gaps in their understanding of things like logic, discrete maths, data structures and algorithms, languages and compilers and so on. I include myself in that. I've cherry-picked elements of computer science as and when I needed them, so while I know a fair amount of theory, it's by no means comprehensive.

We end up with CS graduates who've covered the theory, but often don't have a practical grasp of the applications of the theory (and, therefore, don't really understand it). And we have a majority of self-taught programmers who, when asked to sort a list of strings into alphabetical order, might by default stick them in a database table and use SQL to do something that I believe any programmer should know how to do. (I've seen that several times, believe it or not.)

I've watched teams run scared of anything remotely theoretical that comes up on their projects, and I've watched teams of CS graduates hack out some pretty shitty code because nobody taught them how to, say, write good unit tests.

Looking to the future, our software army needs to be better equipped. Of that I have no doubt. There's little competitive edge in spewing out apps that any programmer could build, nor in spewing out clever, but unreliable and unmaintainable, apps. These two worlds need to be brought closer together to produce more well-rounded software developers.

With this in mind, I want to make a proposal. With a Turing theme in mind, I'd like to focus the software craftsmanship conference - and the community that supports it - on marrying computer science and craftsmanship. I've been noodling with code katas based on simple CS ideas - e.g., test-drive an implementation of binary search - and I think this could be a way to go in bridging the gap.

For the practicing craftsman, we could leverage their practical skills to introduce some computer science in the very practical, concrete way that self-taught programmers tend to respond to.

For computer scientists, we could leverage their theoretical understanding to introduce elements of craftsmanship, and give them a practical, hands-on way to learn the theory by applying it, and get the necessary practice at programming many CS graduates seem to miss out on.

I'm proposing that the SC conference community put our heads together and write a book - or a series of books - under the banner of "Computer Science For The Software Craftsman". It would cover the basics that most CS graduates would learn, but every concept would be presented in the form of a code kata or other practical exercise that allows programmers to learn the theory by tackling applications in code, whilst ensuring a high-quality implementation using the techniques we know and love.

It could be a valuable learning resource for self-taught programmers like me, and a valuable practice resource for computer science students (and teachers).

We could complement it with online resources like screencasts and wotnot, so people can watch and learn and share - perhaps forming themselves into peer groups and working through the material learning from each other.

I believe this would be a non-profit resource owned and published by the community, and in the spirit of the conference, any money raised after costs would go directly to Bletchley Park.

I think a good place to start might be data structures and algorithms. We could organise into groups, each taking a specific topic (e.g., search algorithms) and work in pairs to design and test katas and write explanatory notes. As a community, we'd vote to select the katas we'd like to see demonstrated at the conference, based on screencasts done by each pair, and ensuring coverage of a sufficient selection for a book on the subject. (Yes, we'd use a divide-and-conquer algorithm to write our book on data structures and algorithms!)

I'm actively exploring ways to increase space at SC2012, so hopefully there could be 250+ attending - making 125+ pairs. If more than half of us comes up with something, we'll easily have enough useful material to make a rather grand book and a dandy accompanying web site.

Of course, if you think it's a crazy idea, you can voice your concerns to me on That Twitter - @jasongorman

But if folk are generally in support, we'll make a start on this in January, when tickets will also go on sale for the biggest, baddest SC conference yet.

And with that said, it's back to Xmas telly and mince pies for me. See you in 2012.

On a completely different note, but very much on my mind, are the unprecendented austerity measures being taken by the UK government to try and cut our deficit.

While Prime Minister David Cameron fights for the interests of the City in Europe, and turns a blind eye to the tens of billions in tax avoided/evaded every year by Britain's wealthiest people and corporations, the poorest and most vulnerable in society are experiencing cuts to benefits and services that have seen families made homeless, students driven to drug dealing and prostitution, and some people literally to suicide.

We have over a million young people unemployed, and are facing the very real prospect of a lost generation. Many of these kids may never get jobs. Meanwhile, cuts to higher and further education are really beginning to bite. For the weakest and poorest in society, they are experiencing a "perfect storm" of drops in income, severe cuts to services, rising prices on essentials like food and heating, and education - the key to escaping unemployment and poverty - growing further and further beyond their reach.

How could anyone, in all conscience, enact policies like these that hit the worst off so hard and yet barely inconvenience the wealthiest? Well, it's pretty simple.

Our government is largely made up of people who come from the top 1% of the wealth and income scale. They've had the best educations money can buy. They've had advantages from their wealthy and connected parents that the other 99% can only dream of. How many children can expect a summer job working in an investment bank or a lobbying firm courtesy of Daddy? The disparity in opportunity for children of the richest and the poorest has steadily increased over the last 30 years back to near-Dickensian proportions. The notion that Britain is run by rich kids for rich kids has not be truer for 100 years.

People like Cameron, Clegg and Osbourne can very easily do what they do and then go home and get a good night's sleep because they don't actually know any poor people. In fact, they probably don't know very many averagely well-off people. They don't have to face the wrath of friends or family impacted by the cuts, because they don't have any friends or family impacted by the cuts.

How can we expect proper, fair representation from such unrepresentative representatives? Our government ministers have no empathy towards the bottom 10% of society because they have no direct experience to draw from.

So I'm going to make suggestion. It seems clearer to me now that we desperately need a government that is much more representative of the society it governs. We need a government that is capable of really empathising with the poorest and most vulnerable in society. We need a government made up of representatives who have walked a mile in the shoes of the bottom 10% of society.

They can still be bankers and sons of bankers and Old Etonians and ex-Bullingdon Club members, sure. Just as long as they can demonstrate significant first-hand experience of what life is like for the bottom 10%.

I'm going to propose a Citizenship Card. We can all apply for one. The scheme would be run by the same folk who police charities. Every charity could register for the scheme, and hands-on voluntary work undertaken for those charities would earn you points. The more direct the exposure to society's poorest and most needy, the more points the work would earn you. An afternoon a week in the local charity shop might earn you 10 points, or driving the old folks to the shops might earn you 25, while a day a week cleaning rooms in a homeless shelter might earn you 100.

Some front-line public sector jobs (the dangerous, badly-paid ones, especially - like soldier, fireman, policeman, nurse, inner-city school teacher etc) would also earn you points. As would being an unpaid carer for a family member or loved one.

Basically, all those important, often thankless and usually unpaid jobs that the current government summarises as our "Big Society" would earn you points.

To stand as an elected official - e.g., an MP or a local councillor - you would have to "cash in" some of your Citizenship points. It could be 5,000 for a local council post, 20,000 for an MP. If you are elected, those points are then removed from your card. To stand again for re-election, you might have to roll up your sleeves and volunteer to recharge your card.

The idea is that, to hold public office, you must regularly be directly exposed to the problems of the poorest and most vulnerable members of society. They must become real people to you - with names and faces and personalities you will get to know. You must work alongside people doing the hardest - and often the worst-paid - jobs in society, and they will become real to you, too.

Pardon my French, but if, while in office, you fuck them over, it won't be too long before you'll be seeing them again and you can explain yourself face-to-face. If you don't, then your career in government would be shortlived.

As far as I can see, it's only the prospect of having to look these people in the eye and come to know them as human beings that could address this growing gap between rich and poor, and help mend a broken society where we seem to have one set of rules for the rich and another for the rest.

Distracted today by interesting discussions on That Twitter. One in particular about the folly of giving decision-making power to one group of people ("leaders") and tasking another group with understanding the implications of those decisions ("workers", if you like).

Why do we have dedicated leaders on teams - people who have to seek expert advice so they can make "informed decisions"? Why aren't those decisions being made by the experts themselves?

I don't have a problem with leadership, I should stress. At some point we all need to take the lead so the team can move forward in a specific direction.

My concern is that we appoint someone as leader. Leader as job title is what I have a problem with.

I also don't dispute that a single point of contact between the team and the business is a often a good idea. But let's not confuse the role of "team representative" with "leader".

In politics, and management, the distinction is often blurred. We elect "representatives" who then say they are "in power" and start telling us what to do, rather than asking us what we think should be done. Many of society's ills can be traced back to this confusion between people who speak for us and people who decide for us.

A team can find ways to make decisions that don't require a dedicated leader. I've done this many times in the past. When a decision needed to be taken, the team put it to the vote. Should we use Ibatis or Hibernate? Show of hands. Hibernate it is, then?

At this juncture in the debate, there are a couple of objections that tend to come up:

1. What if your team tends to make uninformed decisions?

Well, that's probably because you hired the wrong people. If you don't trust the majority opinion of your team, then you don't trust your team.

2. Who is accountable for the ultimate outcome of decisions, then?

The team. We stand or fall as a coherent unit.

There may be someone on the team the customer gets on best with. Y'know, one of those "people persons" people tend to bang on about. Dandy. It's like having an expert in talking to the customer. If you have someone on the team who excels at something the rest of you don't, it makes perfect sense to let them lead in that area. Empower them to act as the "face of the team", by all means. But be very clear as a team where that power ends.

Similarly, if someone on the team is acknowledged as the expert in, say, code quality andf is good at spotting code smells and SOLID and all that good, healthy stuff, then empower them to lead in that area.

Generally speaking, empower people who excel in key areas to lead in those areas, and take your lead when you're not the best person to be making those decisions.

When the team can't agree, have a simple constitution that kicks in - a basic democratic decision-making process, with checks and balances for changing course when the team discovers they made a wrong call as early as possible.

If we're unwilling to do this, and, as dedicated leaders, unwilling to relinquish decision-making power to experts and to the team as a whole, we are essentially saying "I know better than this team". And maybe you do, but the fact that they're your team suggests perhaps you didn't. Either you hired an inadequate team, or you joined an inadequate team, or you just stood back and let someone else build the team for you, and didn't insist on getting the right people. None of these makes you look good, frankly.

One very interesting Twitter response suggested that leaders "carry the can". To quote: "Do-ers don't go down if the org goes down". This is patently not true. If the org goes down, we all go down. In fact, in many businesses, it's the workers who go to the wall first. Middole management, when ordered to cut costs, are not in the business of firing themselves.

Why can't the whole team carry the whole can? Teams succeed or fail as a whole. They should share the risks, and share the rewards equally. We are grown-ups, after all.

Very positive news today from Bletchley Park. Google are very generously donating £550,000 to help towards meeting the match funding needed to secure a Heritage Lottery Fund grant that will be used for the first phase of redeveloping the site.

There's still much work to be done, but this donation is going to make a huge difference, and I know the Bletchley Park Trust are absolutely over the moon about it, as are supporters like me.

It would be remiss of me not to mention that this beautiful partnership all started thanks to one very dedicated Googler and Bletchley Park supporter, Simon Meacham. Much kudos to for Google UK's head of external relations, Peter Barron, who's championed Bletchley Park for many months now.

Those Googlers are nice people, and no mistake.

If you'd like to help us renovate Hut 6 at Bletchley Park, please consider donating a few quid





.

While we're on the subject of shiny new things that are actually really old things, the next time someone tells you they're working at the "cutting edge" with things like OO programming, functional programming, MVC, TDD and all that lovely modern stuff, you might want to quote a few dates at them and see just how "cutting edge" they're feeling after that.

First object oriented language - Simula 67 (as in 1967)

First functional programming language - LISP (1958)

First use of Model-View-Controller - (unknown - probably Xerox Parc, early 1970's, term coined in 1979 in SmallTalk book)

First use of Test-driven Development - (unknown, but documented on NASA Project Mercury 1959-1963)


Personally, I just can't get the hang of all this new-fangled waterfall data-driven procedural programming. I'm going to stick with these old techniques, thanks.

This blog post by Peter Krantz neatly cocks a snoop at all those who refer to Agile methods as "fashions" or "fads".

Peter finds quotes relating to many of the key problems XP's tries to tackle - like the need for iterative and evolutionary design, the need for early user feedback, the need to stop treating design and "production" as artificially separate activities, the need for small and highly-prioritised releases, the need to automate testing and the need to recognise the universal truth that some developers are orders of magnitude more productive than others, and that who you have on your team is therefore of vital importance.

If you hunt around, you'll even find evidence of teams applying what looks suspiciously like TDD in the 1960's (e.g., on NASA's Project Mercury).

It may have become fashionable after 2001 to do some of these things, and there may indeed be teams doing them without understanding why they're doing them. But the pioneers of software development have been espousing these things since before many of us were born.

This is a potentially touchy subject, and I may well be ill-advised to even go there, so please bear with me as I try to navigate through it.

I'm one of many men working in computing who'd be very happy to see more women working alongside us.

Nobody seems quite sure what we can do to encourage more young women into the field, and there's much evidence to suggest that the damage is done - certainly in terms of girls seeing programming as "boy's stuff" - by a very early age. Which means that, whatever we do, the target of our efforts needs to be girls under the age of 7 so we can maybe change their minds before Polly Pocket changes their minds for us.

One thing I'm pretty convinced won't help is positive discrimination for women in computing higher education and the work place. And I know many women who have achieved great success in software who feel the same way - indeed, many feel somewhat insulted, as such talk belittles their very real (and very un-positively discriminated in favour of) achievements. Indeed, all of us who've learned our SOLID design principles know that the "L" is female.

The statistics of it makes it unworkable. It's also highly unethical and very unfair both to the men who get discriminated against - recruitment and promotion is a zero-sum game, remember - and the women who succeed because they're good at it.

The current ratio of men to women in software development, for example, is roughly 10:1. That ratio is baked in right from the start. It's not that women find it 10x harder to get into computing. It's simply that 10x less even want to.

if I'm hiring a team of developers, I may have to interview ten people for every person I would actually hire. (And I'm being generous here.) So a team of 4 might require me to see 40 candidates, regardless of gender. It can take weeks - months sometimes - to build even a small team.

What happens if I resolve that 2 of those team members must be female? Finding 20 females candidates will be as hard as finding 200 males candidates, because there are 10x less. My options are to either spend a lot more time looking - and the clock's ticking, folks, because this is business - or to lower the bar so that it's 10x as easy for a female candidate to get the job as her male counterpart.

And here's where it matters, for me. Not only is this grossly unfair on male candidates, and on the business that will now suffer the consequences of hiring substandard developers after massively lowering the bar, it's also very unfair on the good female software developers out there.

I've seen with my own two eyes that women and men are roughly as good at writing code. That is to say, the proportion of good programmers to not-so-great programmers is gender-independent. Let's continue with our model of 10% good and 90% not-so-good. If we lower the bar for women by a factor of 10 to recruit the team without taking significantly longer, we end up interview an average of 1 female candidate for 1 female role. In other words, we hire the first woman we see. That's a bit too much of a lottery for my liking.

What that means is that, as a woman, you'll be surrounded by other women who come from the not-so-good pile. Who does this help?

The same goes for promotion. If you make it actively easier for women to progress at work, you'll end up with a lot of very disgruntled men and an upper tier of tech management that comes from the not-so-good pile. Now who does that help?

It seems to me, from basic statistics and personal experience, that positive discrimination would do nobody any favours - least of all women. Widen the goalposts and what you end up with is a bad reputation for women in computing, and a whole heap of pain for men who've done nothing wrong, but who now have to work 10x as hard to make the same progress in their careers.

Now, I've taken a pretty extreme ratio for the purposes of illustration. But whatever ratio you apply, if you require a greater proportion of women being hired and promoted than actually enter computing, you'll get to some degree the same effect.

The right model is to encourage more women into computing in the first place. And the time to address that is very early in their educational and personal development.



ADDENDUM: female friend has just pointed out that if she was from the not-so-good pile, positive discrimination would be very mch in her favour. Worth thinking about.

Carrying on from my last blog post about backwards compatibility with previous versions of a component, it's a bit like the Liskov Substitution Principle (part of S.O.L.I.D.) - only applying to new versions of classes.

I propose a new design principle to make it official.

Version Substitution Principle

A version of any component can be substituted with any later versions of that component.

What this means in practice is that new versions of a component (e.g., a DLL) must fulfil the contracts of the previous version such that existing client code is unaffected by an upgrade.

As I suggested in the last blog post, we can check this by running the tests that speak to the component's public interfaces of the previous version against our new version. Provided, of course, that we have such tests and that they provide sufficient assurance that contracts are being fulfilled.

If you're working on libraries and frameworks, or on components within a larger software system, this is worthwile, I tend to find.

Here's a quick tip, inspired by a tweet from @hotgazpacho, who was venting his frustration at .NET developers who didn't seem to have grasped the issue of component version compatibility.

I think it was a comment on a vendor's web site about "maintaining compatibility" of a new version of one of their libraries with old client code by freezing the component's version number that set him off.

He's quite right, of course. Can I let you in on a little secret? Those assembly version numbers? They don't mean anything, really. They're like license plates on cars. You can take the license plate off your old car and put it on your new car, but that doesn't mean they'll perform the same.

If you really want to know if the latest version of your component is compatible with existing client code, here's how we used to do it...

You write a suite of tests against the public interfaces of your component. These tests will no doubt evolve as your component evolves, but it is entirely possible to version your tests, too. So each release of your component gets a corresponding test suite. If you want to know if version N+1 of the component wuill be compatible with verion N client code, run the version N tests against the N+1 release.

That way, you can check whether N+1 fulfils the contracts of N. And that's what we mean by "backwards compatibility".

I spend a lot of time talking to people about Test-driven Development, and I've learned over the years to translate what folk tell me about their TDD experiences based on my own experiences of then doing TDD with them (or watching them do it.)

On a whim, and with a large slice of cynical irony, I've constructed my own Test-driven Development Maturity Model (TDDMM), and mapped that on to stuff people tell me about TDD.

You can use this guide to accurately assess someone's TDD experience duing similar conversations. Or you can print it off and wipe your arse with it. Both options are about as useful.

DISCLAIMER
Before I begin, I should make myself absolutely clear: I'm not berating people who don't do TDD. (Though I'll be interested to know what you are doing instead that would produce similarly clean, reliable code that can be quickly and cheaply re-tested throughout development.) This is a maturity model aimed at people who claim to have done TDD. Before you ask, it's a sort of joke. Well, maybe.

Level 0 - Doesn't Know What TDD Actually Is

You'd be surprised how many folk say they've done TDD, but then go on to demonstrate that they probably haven't even read the Wikipedia entry on it. They'll say things like "We do TDD, but we're not purists about it." "Purist", in software development, is secret code for "someone who actually does it at all". They'll often go on to say that, in their "pragmatic" approach (more secret code), they write unit tests after they've written the code. Test-Driven Development. The clue's in the name, folks.

Level 1 - Knows What It Is, But Has Obviously Never Done It

These folks tend to be academics (either literally, or of a type). They're well-read enough to know the basics of TDD, but further probing often reveals that they lack any practical experience. The giveaway tends to be a lack of technical detail in the recounting of their experiences, coupled with a vague and handwavy flavour to it all. They will often speak with a deceptive air of authority, but we must remember that we risk taking sex tips from monks if we heed their advice. One must be careful, though, because people who have a lot of TDD experience can also come across this way. If you're not sure, then 5 minutes with them, you and a laptop will remove any doubt.

Level 2 - Tried TDD. Made My Fingers Bleed.

Getting started with TDD is a bit like learning to play the tuba. It takes quite a while to get pleasing results out of it, and most people don't have that kind of patience. They can often turn into the worst kind of TDD haters, taking every opportunity to poo-poo it in public. But no amount of public poo-poo can disguise the fact that their dislike ultimately stems from their inability to get anything approaching a musical sound out of the end of it during the few days or weeks they tried to learn it. Sad fact is, most people who try to learn TDD give up within 3 months, while they're still climbing its steep learning curve. But rather than admit they never made it to the summit, they prefer to tell people not to bother because "the view from up there is rubbish".

Level 3 - Doing TDD. Well, kind of.

In the pyramid of TDDMM, Level 3-ers are genuinely way ahead of the curve. 9/10 who claim to have done TDD are Levels 0-2. The kinds of noises I hear from these hardy folk tend to fall into the category of "but it's sooooo hard". Often, they'll talk about the overhead of all those bloody automated tests. They take ages to run. They're brittle and hard to change. They're slowing us down. Etc etc.

Danger, Will Robinson! This stage in the development of your TDD ability is make-or-break. I hear too many stories about teams ditching TDD, and ditching their test suites, to make the going "easier". And it will feel like that for a wee while - like throwing the engine out of the plane when it's losing altitude. Chances are, their TDD is a little unbalanced. It may be 99% RED LIGHT-GREEN LIGHT and only 1% REFACTOR. Moving to Level 4 requires an epiphany. Namely that test code is production code, too. And all the programming and design disciplines that apply to production code need to be applied to test code. Oh, and it turns out that the whole "refactoring" thing is really rather more important than we thought. In fact, it's the secret sauce of sustainable test-driven development.

Level 4 - Doing TDD. And Living The Dream. Well, sort of.

Demonstrably, people who've been doing TDD for years and who've hacked through the dense undergrowth of brittle tests, incomprehensible tests, slow tests, highly-coupled untestable designs, over-reliance on mocks and etc etc, tend to find that they actually go faster when they do TDD, and they can sustain that pace on the same code base for years.

Here's the thing. Even to us old lags, who've been doing it since before anyone thought to call it "Test-driven Development", TDD never gets what you might call "easy". There's a battle-scarred world-weariness about the Level 4 TDD-ers. They might not speak of it as enthusiastically or with the fresh-faced idealism of an early Level 3-er, but underneath it all there's a resigned air of "well, of course we do TDD", and a recognisable rolling of the eyes when they hear people talk about "pragmatic TDD", or about how "TDD doesn't work" or about how "TDD slows you down".

Level B - People Who Speak To The Dead

On an entirely perpendicular maturity scale are those very experienced practitioners who go around making bizarre and unsubstantiated claims about programmers who "don't need to do test-driven development", as if they've somehow transcended the need for it.


Now, I'll be the first to admit that you can achieve the same kinds of results without writing your tests first. But, y'know, it's the darndest thing; in order to achieve those results we tend to end up doing things that look a heck of a lot like test-driven development. Take, for example, Formal Methods. Whether it's expressed as tests, or in some more general form, any techniques I've seen that produce TDD-like results always start with a testable specification and always involve a process of verifying that the code we're writing satisfies that specification, either by traditional testing, or through other kinds of testing - like guided inspections, symbolic execution, model checking, proofs of correctness etc. And if our goal is to get the feedback from this testing a soon as possible, then we need to be able to perform those tests as the code's being written. It is true that some programmers are sooooooo clever, they can see that their code will work just by reading it, and this may reduce the need for things like unit tests in that first instance.

But I've yet to meet a programmer who was genuinely soooooooooooooooooooooooooooooooo clever, they could re-read, and therefore re-test all of their code several times a day to make sure they haven't inadvertantly broken something. Being clever, as testing techniques go, doesn't scale well.

But let's assume that such a rare beast exists - a unicorn among programmers. We'd better hope that this unicorn is immortal. And omnipresent. Otherwise, what happens when someone less brilliant and clever tries to change their code? You can be the safest driver in the world, but you'll still need to wear a seatbelt and have insurance. Because unless every driver is as safe as you, you're going to need it.

The trail of automated tests I leave behind me is more there for future developers who may work on that code than for me, though it makes my life so much easier, too. They estimate that software costs 7 times as much to maintain as it does to write in the first place. if leaving behind a suite of good automated tests reduced that to 6 or less, than time writing and maintaining those tests is time well-spent.

And all the hard evidence we have available to us suggests that's the case. On the other hand, there's no good evidence - not one shred of it - to support the existence of these mythical unicorn programmers who don't need to do TDD (or anything resembling TDD).

Programmers who claim to be so good they don't need to do TDD have a special place reserved in my vision of Hell that is the Test-driven Development Maturity Model.

(LAUGH NOW)

A cornucopia of computing history, code-breaking and Top Secret if-I-told-you-I'd-have-to-kill-you insights into the life of a mathematician in the intelligence community can be had at the 2011 ACCU Bletchley Park lectures on Saturday Nov 12th.

They've got top-flight speakers including Ada Lovelace Award winner Frances Allen (yes, a woman! Whatever will they think of next?), inventor of CPL - the precursor to C - Dr David Hartley, and code-breaking and Bletchley Park expert and ex-Chairman of Logica Brian Oakley.

There'll also be updates from the Bletchley Park Trust and The National Museum of Computing and a tour, as well as sandwiches (hurrah!) and a bar (that sells booze - double hurrah!)

All that and more for a fraction of the price of a day at QCon. And it gets better. Every penny of the profits goes to help the Bletchley Park Trust and TNMOC, so you can learn, network, do your bit for our computing and code-breaking heritage and then drink yourself silly afterwards.

And it's being organised by Bletchley Park's own Kelsey Griffin and Claire Urwin, who do a mean conference.

Visit the web site to find out more and book your place.