Duncan Pierce duncanpierce.org

I had some thoughts about classes that wouldn’t fit into a talk I’m building about functional programming in Ruby, so I recorded them as a video.

Topics:

• Using hashes instead of classes.

• Classes as a documentation tool—specifically, as a way of making functions easy to find.

• Preferring module inclusion to subclassing (which is akin to preferring adjectives to nouns as a way of organizing the documentation of verbs). (Vaguely similar to duck-typing in Haskell.)

• Object dot notation as a more readable way of writing function composition. (Similar to the motivation for the -> macro in Clojure or type-directed name resolution in Haskell.)

I’m working on a talk called “Functional Programming for Ruby Programmers”. While doing so, I’ve somewhat changed my opinion about mutable state. Here’s my argument, for your commentary. Being new, it’s probably at best half-baked.

1. Back in the early days of AIDS (the disease), I remember a blood-supply advocate saying “You’ve slept with everyone that everyone you’ve slept with ever slept with.”

   The nice thing about immutable state is that it stays virginal and knowable. It is what it was when it was created. With mutable state, it’s more like “Your code might be infected by any code that ever touched your data before (or after!) you first hooked up with it.”

   However, I don’t personally find that a huge problem in programming, debugging, or understanding code. There are other problems I’d rather see fixed first.

2. It’s often said that “code with immutable state is easier to reason about”. I realized a long time ago that there’s some sleight-of-common-language going on there, like the way people who wanted code to be less branchy got rhetorical leverage by renaming branchiness “complexity“.

   In the claim, “reason about” is (I believe) being taken to be synonymous with “prove theorems about”. Pace a whole long trend in artificial intelligence, I don’t believe theorem proving is the basis for, nor a good analogy to, reasoning. Thus I took the “reason about” statement to be a sort of solipsistic wankery from people with roots in the theorem-proving community, not an argument that should sway me, Pragmatic Man!, who left the world of proofs-of-design-correctness around 1984.

   What I’d forgotten is that optimization is theorem proving. One can only optimize if there’s a proof that the optimized code computes the same result as the original. If you consider the desire to implement, say, lazy sequences efficiently, you see how the guarantees that immutability gives are important. Ditto for automatically spreading work to multiple cores.

3. So:

   My previous attitude was pretty much “Yeah, I have a preference for avoiding mutable state. Being immutable isn’t a huge win, but as long as I have people providing me with data structures like zippers, it’s not really any harder than fooling with mutable state. Still, I don’t see why I need an immutable language. If I don’t want to mutate my hashmaps, I just won’t mutate my hashmaps.”

   My new attitude is: “Oh! I see why I need an immutable language.”

I still don’t make a huge deal about immutability. Its benefit is greater than its cost, sure, but it’s not the Thing That Will Save Us.

Here are three Ruby functions. Each solves this problem: “You are given a starting and ending date and an increment in days. Produce all incremental dates that don’t include the starting date but may include the ending date. More formally: produce a list of all the dates such that for some n >= 1, date = starting_date + (increment * n) && date < = ending_date.

Solution 1:

Solution 2:

Solution 3 depends on a lazily function that produces an unbounded list from a starting element and a next-element function. Here’s a use of lazily:

As a lazy sequence, integers both (1) doesn’t do the work of calculating the ith element unless a client of integers asks for it, and also (2) doesn’t waste effort calculationg any intermediate values more than once.

Solution 3:

The third solution seems “intuitively” better to me, but I’m having difficulty explaining why.

The first solution fails on three aesthetic grounds:

• It lacks decomposability. There’s no piece that can be ripped out and used in isolation. (For example, the body of the loop both creates a new element and updates an intermediate value.)

• It lacks flow. It’s pleasing when you can view a computation as flowing a data structure through a series of functions, each of which changes its “shape” to convert a lump of coal into a diamond.

• It has wasted motion: it puts an element at the front of the array, then throws it away. (Note: you can eliminate that by having current start out as exclusive+increment but that code duplicates the later +=increment. Arguably, that duplicated increment-date action is wasted (programmer) motion, in the sense that the same action is done twice. (Or: don’t repeat yourself / Eliminate duplication.))

The second solution has flow of values through functions, but it wastes a lot of motion. A bunch of dates are created, only to be thrown away in the next step of the computation. Also, in some way I cannot clearly express, it seems wrong to stick the inclusive_end between the exclusive_start and the increment, given that the latter two are what was originally presented to the user and the inclusive_end is a user choice. (Therefore shouldn’t the exclusive_start and increment be more visually bound together than this solution does?)

The third solution …

• … is decomposable: the sequence of dates is distinct from the decision about which subset to use. (You could, for example, pass in the whole lazy sequence instead of a exclusive_start/increment pair, something that couldn’t be done with the other solutions.)

• … eliminates wasted work, in that only the dates that are required are generated. (Well, it does store away a first date — excluded_start — that is then dropped. But it doesn’t create an excess new date.)

• … has the same feel of a data structure flowing through functions that #2 has.

So: #3 seems best to me, but the advantages over the other two seem unconvincing (especially given that programmers of my generation are likely to see closure-evaluation-requiring-heap-allocation-because-of-who-knows-what-bound-variables as scarily expensive).

Have you better arguments? Can you refute my arguments?

I’m trying to show the virtues of a lazy functional style. Perhaps this is a bad example? [It’s a real one, though, where I really do prefer the third solution.]

Introduction

This is a draft. Worth continuing the series? Let me know.

Critter4Us is a webapp used to schedule teaching animals at the University of Illinois Vet School. Its original version was a Ruby/Sinatra application with a Cappuccino front end. Cappuccino lets you write desktop-like webapps using a framework modeled after Apple’s Cocoa. I chose it for two reasons: it made it easy to test front-end code headlessly (which was harder back then than it is now), and it let me reuse my RubyCocoa experience.

Earlier this year, I decided it was time for another bout of experimentation. I decided to switch from Cappuccino to jQuery, Coffeescript, Haml because I thought they were technologies I should know and because they’d force me to develop a new TDD workflow. I’d never gotten comfortable with the testing—or for that matter, any of the design—of “traditional” front-end webapp code and the parts of the backend from the controller layer up.

I now think I’ve reached a plateau at which I’m temporarily comfortable, so this is a good time to report. Other people might find the approach and the tooling useful. And other people might explain places where my inexperience has led me astray.

The example

Suppose an instructor wants to reserve the horses Genesis and Guicho each Tuesday morning from October 8 through the 29th. For historical reasons, that’s done by making a reservation for the 8th and then making three copies. Here’s what that page looks like today:

(As you can see, I haven’t gotten to making the UI attractive yet.)

After the stop date is chosen, a click of the “Duplicate” button causes three successive Ajax calls to be sent to the server. (That’s a bit gratuitous - I just like the idea of the UI showing each reservation “coming to life” in turn.)

Ideally, Genesis and Guicho will be reserved for each Tuesday. However, it’s possible that one of them has already been reserved for the 22nd. In that case, the reservation is still made, but the animal is dropped from it. The user has to edit that reservation to pick a replacement animal.

Testing the Haml

I usually test in an outside-in, mock-heavy style derived from Growing Object-Oriented Software (GOOS). “Derived”, naturally, means I depart from GOOS in my very first step: I avoid automating end-to-end tests. For my small-scale purposes, I don’t find GUI tests (Watir, Selenium, etc.) worth the price. To be clear: I think about GOOS-style tests, and I execute them—I just don’t automate them.

Having thought of an end-to-end test for a particular user task, the next step is to sketch out a page that supports it. I sketch it out in Haml to be the beginnings of the page above.

My page testing is driven by this observation: the user will use the same page for many different reservations. It has a constant part that’s independent of the reservation, and a part that depends on the particular reservation. I only write automated tests for the variable part. I test the constant part by looking at it, by looking at the generated HTML, and by trying the finished story by hand.

There are three parts of this particular page that vary:

• There’s a snippet of text that reminds the user what reservation she’s working with.

• There has to be a starting date used to set up the calendar.

• There has to be some URI that’s used to PUT a new repetition into the system. That URI will look something like /reservations/441/repetitions.

Here’s the setup for tests to check those three claims:

1. Whenever I create a model object, I usually give it a random constructor that builds it with innocuous values unless I give it explicit ones. I’ll explain why I’m giving it only one value below.

2. I hate the spooky action at a distance of setting instance variables in a controller in order to have some effect way off in another file written in a different language. I’d much rather have a nice explicit function-call interface and a builder or transformation approach. Most of the world thinks differently, so I concede. However, I do try to come closer to a function-call interface by passing in locals instead of setting instance variables.

3. Passing in a whole Reservation model object makes me queasy for reasons Bob Martin has explained. I’m intruding the model into the view rather than passing in just what the view needs. This is a consequence of some amount of laziness after bowing to the peer pressure to use partials. I do make some effort to separate presentation-relevant data from model-relevant objects, as you’ll see later.

4. In order not to repeat myself, I make one Ruby file the authority over all URIs and links. Therefore, any URI gets passed into the HAML. In pages like these, which exist to aid some specific user task and have a single URI that they use when the appropriate data has been supplied by the user, I call that single URI the “fulfillment” link.

That given, here’s the test in which I claim that a reservation summary appears on the page:

I use shoulda because I like nested contexts and strings to name tests, but I don’t need the rest of Rspec. I use Assert{2.0} because I find Hamcrest-style matchers too wordy and Assert{2.0} gives me usefully detailed failure messages. (It prints every subexpression of a false expression. Wrong is a more recent implementation of this idea.)

The actual reservation summary is created with a partial. By checking for ‘Dr. Dawn’, this test claims the partial has been “called”. (It’s thus a clumsier and less explicit example of a mockish test.) I’m testing only that here because I want the freedom to what the summary looks like without having to change this test.

The next test claims that a correct Javascript date exists in the HTML (presumably in some Javascript code):

Notice that the check is, again, minimalist. The test isn’t about how the date is used, just that it exists. Peeking ahead, it’s actually used like this:

There’s a gap in the test coverage, in that no automated test checks that this initialization code is correct. Pretty low risk, given that it’s straight-line code that cannot avoid execution whenever the page is loaded. So long as I try this page out once, and remember to try the page again if I change the initialization code, I should be safe.

The final test is this:

assert_text_has_attributes is part of a thin wrapper on top of the HTML parsing part of Capybara (not the integration testing part). I used to use assert_xhtml, a sweet package, but it’s dead in Ruby 1.9 and I haven’t found another one like it. Let me know if you have one.

The Haml code that passes these tests looks like this:

The <head> is, annoyingly, generated by another action at a distance, with a layout.haml file that gets wrapped around the above code. That file uses the third of the “parameters”:

What now?

The static page and the acceptance test I have in mind call for some Javascript. That needs to be tested into existence.

“The anarchy of evasion” is a term that (I think) I read a few years ago. I occasionally search for it, but I’ve never been able to find anything about it again. As I recall, the idea is a reaction to the fact that an anarchist movement that forms an independent new society or transforms an existing society is… hard. So an alternative is to live within an existing society but try to make as many as possible of your activities and interactions run according to the principles you wished the whole society ran with. You behave as you wish, but in a way and context that increases the chance you won’t be noticed and therefore stopped.

The idea appeals to me, perhaps because I’ve always liked the idea of complex and interesting things happening unnoticed in the cracks. Probably a result of reading The Littles as a kid and books like You’re All Alone and Neverwhere later.

Parallels with things like open source, intentional communities, squatting, and freaks and geeks in US high schools are left as an exercise for the reader.

If anyone has links to the idea, let me know.

Motivation

Consider a Ruby backend that communicates with its frontend via JSON. It sends (and perhaps receives) strings like this:

Let’s suppose it also communicates with a relational database. A simple translation of query results into Ruby looks like this:

(I’m using the Sequel gem to talk to Postgres.)

On the face of it, it seems odd for our code to receive dumb hashes and arrays, laboriously turn them into model objects with rich behavior, fling some messages at them to transform their state, and then convert the resulting object graph back into dumb hashes and arrays. There are strong historical reasons for that choice—see Fowler’s Patterns of Enterprise Application Architecture—but I’m starting to wonder if it’s as clear a default choice as it used to be. Perhaps a functional approach could work well:

• Functional programs focus on the flow of data through code, rather than on objects with changing state. The former seems more of a match for a typical webapp.

• It’s common in functional languages to lean toward a few core datatypes—like hashes and arrays—that are operated on by a wealth of functions. We could skip the conversion step into objects. Rather than having to deal with the leaky abstraction of an object-relational mapping layer, we’d embrace the nature of our data.

Seems plausible, I’ve been thinking. However, I’ve never been wildly good at understanding the problems of an approach just by thinking about it. It’s more efficient for me to learn by doing. So I’ve decided to strangle an application whose communication with its database is, um, labored.

I’m going to concentrate on two things:

• Structuring the code. More than a year of work on Midje has left me still unhappy about the organization of its code, despite my using Kevin Lawrence’s guideline: if you have trouble finding a piece of code, move it to where you first looked. I have some hope that Ruby’s structuring tools (classes, modules, include, etc.) will be useful.

• Dependencies. As you’ll see, I’ll be writing code with a lot of temporal coupling. Is that and other kinds of coupling dooming me to a deeply intertwingled mess that I can’t change safely or quickly?

This blog post is about where I stand so far, after adding just one new feature.

A path through the app

Critter4us is an app that’s used to reserve teaching animals at the University of Illinois vet school. Reserving animals is like reserving meeting rooms, but with some different business rules. For example, Boombird the horse doesn’t care if students practice bandaging on him every day. However, it would be inhumane to practice giving him injections every day, so that can be done at most twice a week.

The story I’ve been working on is one that makes a copy of an existing reservation but with a new “timeslice” (something like “January 1st through 3d, in the mornings”). Ideally, the same animals will be assigned to the copy, but that’s not always possible. If someone else has already reserved the animal for an overlapping timeslice, a new animal has to be found. Or if a procedure (like giving injections) would be within an animal’s “blackout period”, a new animal has to be found for it. For historical reasons, the reservation is made without the unusable animals and the user is alerted to edit it to add new ones.

Here’s the heart of the code for the feature:

The rest of this post will explain how that works and how it’s in a functional style. When I mention classes, I’ll link to the source.

FullReservation

The object model for the old program starts with a reservation, which contains information like “Who made the reservation?” and “For when?”. It also contains zero or more groups, each of which contains zero or more uses. A use links an animal to a procedure to be performed on it. I defined the class structure first, then mapped it onto a database schema, deliberately deferring any worries about efficiency:

The object-to-relational mapping library (Sequel) let me work with the data in a way that hid (”unflattened”) the table structure:

The existence of three tables is made somewhat implicit.

I’m replacing that old Reservation object with a new FullReservation. In FullReservation, I chose instead to make them explicit:

That notation is awkward to type and it doesn’t lend itself to the Symbol#to_proc hack, so I follow Javascript by allowing dot notation as a pun for key lookup:

Namespacing

I’m supposedly doing this in a functional style, and the very first thing I’ve done is make a class? What’s up with that?

I have two reasons. First, I think having the order of function application flow left to right fits the (Western-language-speaker) perception that time flows from left to right and from top to bottom. That makes this:

… easier to read than this:

The second reason is namespacing. You’ll shortly see that everything is built on top of an immutable, lazy FunctionalHash object. A FullReservation just collects those methods that wouldn’t make sense for anything but a FunctionalHash being treated as a reservation. It’s about avoiding name collisions more than about modeling the world.

Inheritance gives me nested namespaces, something I dearly wish I had in Clojure. For example, there are a variety of functions that apply to FunctionalHashes that represent database tables, but are irrelevant to other ones. That code is contained in FullReservation’s superclass, DBHash.

Actually: not quite. The text of the code is found in three different modules that are included into DBHash. I expect to do a lot of mixing-and-matching to create namespaces for particular DBHash classes and even particular objects (via extend).

Extracting a FunctionalTimeslice from a FullReservation

In the reservations table, there are three columns devoted to “when is the reservation for?” They are :first_date, :last_date, and :time_bits. (The first two are Date objects; the last represents the set {morning, afternoon, evening}. In a proper object-oriented design, you’d expect a FullReservation to contain a Timeslice that in turn contains those three values and some timeslice-specific methods as well. I chose to handle such sub-objects differently. Instead of asking a reservation for its timeslice, you make a timeslice from a reservation using setlike operations.

FunctionalHash has an only method that produces a smaller FunctionalHash containing only the named key-value pairs. So this is a timeslice:

I give the timeslice access to a timeslice-specific namespace by wrapping it in a class:

Adding a new Timeslice to Full Reservation

It’s often said that code without mutable state is easier to reason about. I don’t personally find that as big a deal as other people do, but I’ve gotten used to immutability from my Clojure programming. So FunctionalHash disallows messages like this:

The equivalent of assigning a value to a key is done by merging it and creating a new FunctionalHash. The equivalent of deleting a key is done by making a copy of FunctionalHash without the given key. At the moment, this implementation is grossly space-inefficient. Eventually, I’ll port it over to Simon Harris’s Hamster, which implements structure sharing and other optimization techniques. I’m even thinking I might port his code to C.

Here’s the way to change a reservation’s date:

A few notes:

• change_within is a way of “merging” into a nested hash. It’s the equivalent of this:

• Remember that, here, timeslice is a three-key hash, not an object.

• I’m also removing the id to remind myself that the reservation produced here no longer corresponds to one in the database.

Working with disallowed animals

Any time a reservation is made, it disallows some animals (because they’re now in use) and may disallow some animal/procedure pairs (because of rules about how frequently a procedure can be performed). That information is calculated once and stored in Postgres tables named excluded_because_in_use and excluded_because_of_blackout_period.

The code to look up which animals are in use during a timeslice is factored into three pieces. Inlined, it would look like this:

(Fall turns an array of hashes into an array of FunctionalHashes.)

Because that code doesn’t refer to a reservation at all, it seems reasonable to put it in the FunctionalTimeslice namespace.

The FullReservation can use the list of animal ids to prune out its uses:

That’s all easy enough, but the contract with the user is that she’ll see a list of names of animals that couldn’t be included in the reservation. Getting that list is easy enough, given that we have ready access to the rejected uses. Here’s the code, with changes to the previous version highlighted:

That’s fine, but what do we do with the value named by ___animals_already_in_use___? I’d hate to return it along with the new version of the reservation because its caller would have to look like this:

I’d rather avoid names for intermediate steps in the creation of the reservation copy. I want the various versions to flow anonymously through a chain of functions so that I need only name the original and the final copy. (original and copy would be better names than reservation and new_reservation, it occurs to me, but I’m not going to go back now and change all these gists.)

That suggests slamming the animal list into the next version of the reservation, like this:

That kind of creeps me out, and it exacerbates temporal coupling. Nevertheless, it lets the caller look nice, which might mean something. (If mathematicians can go on about elegance, why can’t I?) I hope my tests will loudly tell me when coupling causes a change to function X to break function Y.

as_saved

The last step of creating the copied reservation is oddly named:

as_saved? This is a stylistic affectation that I’m not sure is a good idea. What I’m trying to imply is that the main thing this function does is create a new FullReservation with a bit of extra data, namely data.id, merged in. (The id needs to be sent off to the front end.) The fact that the id is created by changing persistent state somewhere is just an implementation detail. It could just as well be that every possible reservation always already exists somewhere as a big immutable pool, so as_saved just does a lookup to find the matching id.

(Which, it again occurs to me too late, perhaps makes as_saved a name that, strictly, reveals too much about the implementation.)

(Interestingly, I understand that the human immune system works roughly like the silly implementation above: you’re born with some 10 billion different antibodies and the response to infection (mostly) involves finding the useful ones, not creating new ones that match the foreign agent.)

only and the nature of classes

After the as-it-appears-in-the-database FullReservation is created, the pieces that the frontend code care about are extracted and returned to the controller code, which turns them into JSON:

But something creepy is going on here. What’s the type of the result of only?

How can anyone possibly believe that the two-element hash, containing nothing about the reservation in question (but only about the difference between it and its original) is a FullReservation?

For a time, I considered changing only to produce a FunctionalHash, rather than (as it does) an object of the same class as the receiver of the method. Then I smacked myself and reminded myself that I’m using classes to identify namespaces, not natural kinds. Saying that the result of only “is a” FullReservation would be absurd. But it’s less absurd to say that (1) we started with a hash, (2) the functions in the namespace FullReservation applied to it, (3) we derived a second hash from the first, so (4) it’s probably a good guess that the same namespace will be also useful for the second hash.

That is, it’s all about conservation of work. If I stripped every result of only down to a bare FunctionalHash, I’d sometimes have to add a namespace back. By not stripping it, sure, I may get irrelevant functions in the easily-accessible namespace, but I can just ignore them.

Laziness

I chose the name FullReservation not just because Reservation was already taken. It’s because a FullReservation contains all the values that can possibly be relevant to a reservation. But some HTTP requests only care about the reservation’s id. Some only care about some of the data (like the timeslice). Only a few care about the uses and the groups.

Laziness of the sort implemented in Clojure and Haskell seems a nice match for this. When a FunctionalHash key is assigned a block/lambda, it doesn’t treat that as a value. Rather, the FunctionalHash runs that block to calculate the value when the key is dereferenced. After that, the value is cached (and is immutable, just like any other value).

So consider these two steps from our controller:

We create a new reservation by saving a modified reservation to disk. That gives us a new row in the reservations table, some new rows in the groups table, and some new rows in the uses table. But nothing of the groups or uses is used from then on, so it would be a waste to populate the new_reservation with them. How is that avoided? By creating a FullReservation like this:

The uses and groups and even the row in the :reservations table are only loaded when they’re needed, so it costs little to use a FullReservation for everything. With this structure, I’m trying to gain more control than an object-to-relational mapping library gives me, while still freeing myself from the micromanagement of loading. Time will tell if that works.

(Note: I stash the original id that led to the FullReservation in :starting_id. Part of the motivation was to allow a completely fresh FullReservation to return its id without going to the database at all, and another part was to retain the original id even after later changes made it no longer an index into the reservation contents. This dual purpose makes the code confused, I think.)

(Note: Postgres supports the SQL RETURNING extension. So it’d be relatively easy to fully populate a saved FullReservation. I’ve used RETURNING several times, but always later discarded it for one reason or another.)

The grand conclusion

I’ve always hated end-of-talk or end-of-post summations. So I don’t really have one here, except that this approach feels promising, I want to continue trying it, I’d like to hear your comments (sorry about the antiquated blog software), and I’d especially like to hear what happens if you try out this approach.

While waiting for my product owner to get back to me, I was going through open browser tabs. I read this from Swizec Teller:

The problem is one of turning a list of values, say, [A, B, C, D] into a list of pairs with itself, like so [[A,B], [A,C], [A,D], [B, C], [B,D], [C,D]].

He had problems coming up with a good solution. “I can do that!” I said, launched a Clojure REPL, and started typing the whole function out. I quickly got bogged down.

This, I think, is a problem with a common functional approach. Even with this kind of problem—the sort of list manipulation that functional programming lives for—there’s a temptation to build a single function bottom up. “I need to create the tails of the sequence,” I thought, “because I know roughly how I’ll use them.”

For me (and let’s not get into all that again), it usually works better to go top down, mainly because it lets me think of discrete, meaningful functions, give them names, and write facts about how they relate to one another. So that’s what I did.

First, what am I trying to do? Not create all the pairs, but only ones in which an element is combined with another element further down the list. Like this:

As I often do when doing list-manipulation problems, I lay things out to visually emphasize the “shape” of the solution. That helps me see more clearly what has to be done to create that solution. There’s one set of pairs, each headed by the first element of the list, then another set, each headed by the second element. That is, I can say the above fact is true provided two other facts are true:

It’s easy to see how I get the heads—just map down the list (maybe I have to worry about the last element, maybe not—I’ll worry about it if it comes up). What are those heads combined with? Just the successive tails of the original list: [2 3], [3]. I’ll assume there’s a function that does that for me. That gives me this entire fact-about-the-world-of-this-program:

Given all that, downward-pairs is easy enough to write:

It’s important to me that I have reached a stable point here. When building up a complicated function from snippets in a REPL, I more often create the wrong snippets than when I define what those snippets need to do via a function that uses them. (I’m not saying that I never backtrack: I might find a subfunction too hard to write, or I may have carved up the world in a way that leads to gross inefficiency, etc. I’m saying that I seem to end up with correct answers faster this way.)

Now I just have to solve two simpler problems. Tails I’ve done before, and here’s a solution I like:

For those who don’t know Clojure well, this produces this sequence of calls to drop:

This does roughly the same work as an iteration would do because of how laziness is implemented.

That leaves me headed-pairs, which is a pretty straightforward map:

This seems like a reasonable solution, doesn’t strike me as being terribly inefficient (given laziness), has a readability that I like, doubtless took me less time than developing it bottom-up would have, and comes with tests (including an end-to-end test that I won’t bother showing).

The whole solution is here.

UPDATE: Yes, I could have used do or even gotten explicit with the sequence-m monad, but that wouldn’t have addressed the original poster’s point, which I took to be how to think about functional problems.

After around 11 months of not working on it, I needed to make a change to Critter4us, an app I wrote for the University of Illinois vet school. The change was simple. When I tried to push it to Heroku, though, I discovered that my Ruby gems were too out of date. So, I ended up upgrading from Ruby 1.8 to 1.9, to Sinatra 1.3 from a Sinatra less than 1.0, to a modern version of Rack, etc. etc. In essence, I replaced all the turtles upon which my code-world was resting. There were some backwards-compatibility problems.

One incompatibility was that I was sending an incorrectly formatted URI to fetch some JSON data. The old version of Rack accepted it, but the new one rejected it. The easy fix was to split what had been a single `timeslice` parameter up into multiple parameters. [Update: I later did something more sensible, but it doesn’t affect the point of this post.] “Crap!”, I thought. “I’m going to have to convert who knows how much test data.” But I was pleased when I looked at the first test and saw this:

The key point here is that neither the format of the URI parameters nor the resulting timeslice object is given in its real form. Instead, they’re represented by strings that basically name their type. (In my Clojure testing framework, Midje, I refer to these things as “metaconstants“.)

The only responsibility this code has toward timeslices is to pass them to another object. That object, the `internalizer`, has the responsibility for understanding formats. The test (and code) change is trivial:

The test is even (and appropriately) less specific than before. It says only that the GET parameters (a hash) will contain some key/value pairs of use to the internalizer. It’s up to the internalizer to know which those are and do the right thing.

The point here is that the size of the test change is in keeping with the size of the code change. It is unaffected by the nature of the change to the data—which is as it should be.

This application is the one where I finally made the important decision to use mocks heavily in the Freeman and Pryce “London” style and—most importantly—to not fall into the trap of thinking “Mocks are stupid!” when I ran into problems. Instead, I said “I’m stupid!” and, working on that assumption, figured out what I was doing wrong.

I made that decision halfway through writing the app. One of the happy results of the mocking that followed was that a vast amount of test setup code devoted to constructing complete data structures went away. No more “fixtures” or “object mothers” or “test factories.”

In some way, my original post makes it seem as if normal programmers are like the drug dealer D’Angelo Barksdale eating in a fancy restaurant in “The Wire“—everyone stared, the waiter rubbed D’Angelo’s ignorance of fine-restaurant customs in his face, and D’Angelo’s introspective attitude doesn’t hide a desperate desire to leave. Well, yes (though nothing in a programmer’s life is as bad as anything in a “The Wire” character’s life), but there’s more to it.

I think Clojure already suffers from being a language for hardcore programmers to solve hardcore problems. For example, that hardcore attitude makes things like nice stack traces seem unimportant. More: fishing through stack traces can almost become a perverse right of passage. Do you have what it takes to program in Clojure, punk?

The history of programming, as I’ve lived it, has been a march where expressiveness led the way and performance followed. I came of age when the big debate was whether us new-fangled C programmers and our limited compilers could ever replace the skilled assembly-language programmer. C won the debate. When Java came out, the debate was whether a language with byte codes and garbage collection could be fast enough to be a justifiable choice over C/C++. Java won the debate. Ruby is still being held up as too slow in comparison with Java, but even the pokey MRI runtime is fast enough for a whole lot of apps. Perhaps the emerging consensus is that Twitter-esque success will require migrating from Ruby to JVM infrastructure—and that’s exactly the kind of problem you want to have.

Against that background, it’s notable that Clojure 1.3 is taking a step backward in expressiveness in the service of performance. (By this, I mostly mean that performance—calling through vars, arithmetic operations—is the default and people who are worried about rebinding and bignums have to act specially. So my super-cool introduction-to-Clojure “Let’s take the sum of the first 3000 elements of an infinite sequence of factorials” example will become less impressive because arithmetic overflow is now in the programmer’s face.) That’s worrisome. There’s not much new in 1.3 to tempt the hitherto-reluctant Java or Ruby programmer. And unless I care about performance or am a utility library maintainer, I don’t see very much in 1.3 to tempt me away from 1.2.

Now, 1.3 is not a huge hit to expressiveness, and Clojure already has pretty much all the expressive language constructs that have stood the test of time (except, I suppose, Haskell-style pattern matching). So what else is left for someone of Rich Hickey’s talents to work on but performance? Documentation?

That’s a good point, but I worry about how performance will drive the constructs available to Clojure users. For example, multimethods feel a bit orphaned because of the drive to take advantage of how good the JVM is at type dispatch. There’s a whole parallel structure of genericity that’s getting the attention. And conj is a historical example: it’s hard to explain to someone how these two things both make sense:

As with arithmetic overflow, it strikes those not performance-minded (those not hardcore) as odd to have lots of generic operations on sequences except for the case of putting something on the end. “So I have to keep track of what’s a vector? In a language that creates lazy sequences whenever it gets an excuse to?”

Catering to performance increases the cognitive load a language puts on its users. Again, the hardcore programmer will take it in stride. But Lisp and functional languages are different enough that I think Clojure needs more focused attention on the not-hardcore programmers.

Now, the quick retort is that Clojure isn’t my language. It’s a gift from Rich Hickie to the world. Who am I to demand more from him? That’s a good retort. But my counter-retort is that I demand nothing. I only suggest, based on my experience with the adoption of ideas and languages. It’s hard making something. More information (from users like me) is probably better than less.

The problem with talking about the tone of the Clojure community is that almost no statement can ever be proven. Rich Hickey and I went around and around about this on Twitter. So after citing this statement of his:

If people think that spending fifty percent of their time writing tests maximizes their results—okay for them. I’m sure that’s not true for me—I’d rather spend that time thinking about my problem.

… I could say that I read a snarky “[those losers]” implicitly attached to “okay for them.” I could say that “fifty percent of your time writing tests” and “rather spend that time thinking about my problem” are complete mischaracterizations of TDD as it’s actually done. And I can say that those mischaracterizations and that snark will be aped by programmers who want to be Rich Hickey.

I could say all those things, but the statement as written isn’t false. It can even be given a generous reading. But—if I’m right—programmers like me won’t stick around to hear defensible statements defended. They’ll just look for friendlier places where they aren’t expected to deal with decade-old arguments as if they were new and insightful.

Please note that I’m not saying Rich Hickie’s design approach is wrong. I suspect for the kinds of problems he solves, it may even be true that his approach is better across all designers than TDD is.

TL;DR: Attitudes common in the Clojure community are making that community increasingly unappealing to people like me. They may also be leading to a language less appealing to people like me than a Lisp should be.

I think people like me are an important constituency.

The dodgy attitudes come from the Clojure core, especially Rich Hickey himself, I’m sad to say. I wish the Clojure core would be more supportive of programmers not like them, and of those with different goals. Otherwise, I’m afraid Clojure won’t achieve a popularity that justifies the kind of commitment I’ve been making to it. What should I do?

I’ve caused a bit of kerfuffle on Twitter about Clojure and testing. I’ve been somewhat aggressive because I’m trying to suss out how much effort to put into Clojure over, say, the next year.

[Note to the reader: scatter “I think” and “I could be wrong” statements freely throughout the following.]

Here’s the deal.

What’s at issue is a style of working (roughly XPish). After some years of work by many, it’s entirely possible in the Ruby world to work in that style. After doing the whole Agile spokesmodel thing for a decade, that’s all I want: a place where there are enough people with my style that we can (1) find each other to work together, and (2) work together without having to re-justify our style to every damn person we meet. In Ruby, XP style is mainstream enough that we can let the product of our work speak for us. (Hence, my recently co-founded contract software company.)

I think it was at the 2nd RubyConf that Nathaniel Talbott introduced Test::Unit, a Ruby version of jUnit. I can’t be positive, but I’m pretty sure Matz (the author of Ruby) didn’t care all that much about testing or, especially, TDD. Wasn’t his style. Perhaps still isn’t. However: he did not, to my recollection, express any reaction other than his usual “People are doing things with my language! I am happy!” And while Matz gives keynotes about design and such, they are pretty obviously about how he feels about design and about how he approaches it. They are invitations, not instructions.

One of the strengths of Ruby in the early days was this air of invitation. (I don’t really follow that community closely any more.)

Rich Hickey, for all his undoubted brilliance as a language implementor and his fruitful introspection into his own design process, presents a less welcoming demeanor than Matz’s. From what I’ve seen, he is more prone, in his public speaking and writing, to instruction than to invitation.

Which is OK with me, all things being equal. But there’s a problem. Brilliant solo language designers set a tone for the community that gathers around the language, especially in the early days. We’ve seen that with Python and Ruby. (Rails too, which acts as a contrast to the “Ruby Classic” tone.) It’s happening again in Clojure, where I’d say the picture of the ideal developer is someone who, with grim determination, wrestles a hard problem to the ground and beats it into submission with a multi-issue bound copy of The Journal of Functional Programming. I’m all for people who do that! So I don’t have to! But I personally prefer something more akin to the style of Ruby’s _why, who’d sneak up on a problem and tickle it until it was laughing too hard to be a problem any more.

I didn’t always prefer the Ruby style. Just after I started my first post-graduate job, Tony Hoare’s Turing Award lecture came out, containing the famous quote:

I conclude that there are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult.

I positively thrilled to that. That was the man I wanted to be.

As time went on, though, I ran into difficulties. What if you can’t solve the problem with that first method, whether because you’re not smart enough, or the problem domain is inherently messy, or the problem keeps changing on you? What then? Do you just give up? Or do you try to find a third method?

Those difficulties, plus some timely encounters with some strange thinkers, notably Feyerabend and his Against Method, put me on my current career path. Which causes me to say this:

For someone doing core infrastructural design, such as the huge task of tricking the JVM into running a Clojure program speedily, an emphasis on purity and simplicity—one bordering on asceticism—seems to me essential. God speed Rich Hickey for that!

But when a community adopts purity, simplicity, and asceticism as core tenets—ones approaching matters of morality—you can end up with a community that makes things unpleasant for not just the impure and hedonistic, but also for just plain normal people. And they will go elsewhere.

Much as I love Lisp and admire Clojure, if it can’t grab the enthusiasm of, say, the median pretty-good Ruby or Java programmer who has the TDD habit, it’s not economically sensible for a cofounder of a new company to put as much effort into it as I have been putting.

If Clojure turns into a tool for hardcore people to solve hardcore problems, without making concessions for people with softer cores—if it remains a niche language, in other words—I should probably spend more time with Ruby. And I do think, right now, that Clojure is more likely to be a niche language than I thought last year.

So in the last few weeks, I’ve been sending out little probes, mainly to see if anyone with some star power can provisionally buy the story above. If so, I can hope to see an intentional broadening in a direction designed to shackle me and those like me more tightly into the Clojure community—and so help it succeed.

So far, meh.

If I’m sensible, a continuing “meh” response will prompt me to reduce my effort around Clojure—to resist the temptation to make another tutorial, to cut back on new Midje features (but not on fixes and the 1.3 port, I hasten to reassure its users!), and so on. If Clojure then wins big, I can jump back on the bandwagon. I will have lost some early mover advantage, but that doesn’t look to be huge anyway. (Not like it was with Agile, or would have been with Ruby if I hadn’t drifted away before Rails came out.)

I haven’t decided what to do. Reaction to this post might help. Probably the best thing would be for me to be convinced I’m nuts. Hey, it’s happened before!

A postscript on the validity of what I claim

A postscript on worries about the effect of performance on expressiveness

I’ve been asked for things to read after my “Functional Programming for Object-Oriented Programmers” course. Here are some, concentrating on the ones that are more fun to read and less in love with difficulty for difficulty’s sake.

If you were more interested in the Clojure/Lisp part of the course:

• The Little Schemer and other books by the same authors. These teach some deep ideas even though they start out very simply. You’ll either find the way they’re written charming or annoying.

• Land of Lisp is a strange book on Lisp.

• If you were totally fascinated by multimethods, you could try The Art of the Metaobject Protocol.

If you were more interested in the functional programming part of the course, the suggestions are harder. So much of the writing on functional programming is overly math-fetishistic.

• Learn You a Haskell for Great Good is another quirky book (lots of odd drawings that have little to do with the text), but it covers many topics using the purest functional language. There’s both a paper version and a free online version. The online version has better drawings (color!).

• I haven’t read Purely Functional Data Structures, though I think I have it somewhere. @SamirTalwar tweeted to me that “I’m enjoying Purely Functional Data Structures. It’s quite mathsy, but with a practical bent.” The book is derived from the author’s PhD thesis.

The approach I took in the course was inspired by the famous Structure and Interpretation of Computer Programs (free online version). Many people find the book more difficult than most of the ones above. Part of the reason is that the problems are targeted at MIT engineering students, so you get a lot of problems that involve resistors and such.