Tidbits from the Pluralsight trenches

Taking executable specs to the next level: Executable Documentation

with 4 comments

Here at Pluralsight, we’ve been using Machine.Specifications (mspec) for a little over a year now to implement our unit tests. Given how quickly formal documentation goes out of sync with code, many people smarter than me have advocated that unit tests can be used as documentation for a class. And this might work, if

a) The question you have is very specific (e.g. you can do a FindUsages on a method/property and quickly see the tests that exercise it), and

b) A test exists that exercises the method/property you’re trying to understand.

This morning I had a look at the specs for one of our aggregate roots that resulted from its TDD to see if it was adequately documenting its public API, lifecycle semantics, etc. What I saw was a jumbled mess of specs that weren’t designed at all to focus on what a user of that class cares about. Don’t get me wrong – the specs we had in place were all very helpful, but they were focused on testing little bits of functionality, much of which was internal to the package. And even if I could filter out the internal stuff and just leave the specs covering the public APIs, there was no order to any of it. I couldn’t look at my specs and figure out how the class should be used, from its creation through its various states, until it’s archival. Its lifecycle wasn’t clear.

Plus, the language of the spec wasn’t focused on being readable for an end user. When I look at a typical specification that I wrote during TDD, I saw something like this:

When making an invoice

  • Should have a date of Now
  • Should have the customers name
  • Should have no items

This is perfectly fine as a unit test, but isn’t quite what I want for documentation. Here’s language that I think would be more helpful as documentation. Note how I use the actual names from the public API:

When you invoke MakeInvoice on an InvoiceFactory the new Invoice

  • Has todays date
  • Has the customers name
  • Does not have any items

Heck, I can write specs like that. So I figured I’d give it a try this morning. The exercise was very enlightening.

Leaving all of my existing unit testing specs in place, I added a new folder, InvoiceDocs, and I wrote additional specs to act as “executable documentation” for the Invoice class. Here’s what I did in InvoiceDocs:

  • I added specs covering the entire public API of the Invoice class, omitting anything that wasn’t useful to a client of the class
  • I *ordered* the specs so they showed the lifecycle of the Invoice class, from creation to archival, documenting the exact API names for a newcomer
  • I used language that would be approachable for a newcomer

And in the process, I learned some things. I now know how to create executable documentation for my public API as I develop the class using TDD. Next time I will write specs for my public API in a separate *Docs folder using these conventions as I TDD. As I wrote my documentation specs, I found some things that were awkward to document. This led me to refactor my domain model to make it easier to document, which naturally made it easier to use. I found the executable documentation driving me to refactor toward deeper insight. Very cool – as if we don’t have enough ways to “drive” design, now we have “documentation driven design” ;-)

When all was said and done, I ended up with executable documentation that will fail the build if it’s not correct. Kind of. There’s still one flaw that we should think about fixing together. In MSpec the “When” statement is driven by a class name. Embedding API names in the class name isn’t something that will lend itself well to refactoring, and this will get out of date unless I am vigilant. So the solution is far from perfect. It’d be very cool if we could have the option of putting the When statement into an expression of some sort so that the names of APIs would be part of an expression that can be refactored. One step at a time, I guess.

It took me a few iterations to figure out how to get my specs to show up in a nice, ordered fashion in the test runner. After a few initial attempts at adding numbers to the class/file names, I realized I could simply use the Subject attribute of each specification to order my tests appropriately. The namespace was already stating the subject, so this didn’t seem to hurt anything, but I’d like to start a conversation around better ways of ordering tests/specs in runners so that we can start accomplishing this more naturally. Here’s what one of these “ordered” specs look like:

    public class When_you_invoke_MakeInvoice_on_an_InvoiceFactory_the_new_Invoice
        // ...
About these ads

Written by Keith Sparkjoy

June 26, 2011 at 8:54 am

Posted in Modeling

4 Responses

Subscribe to comments with RSS.

  1. Keith,

    I have found myself doing things similar to this and when I do, I have learned to recognize that I need to bump the level of abstraction on my tests up one.

    If you are documenting state management of the class for a future developer, okay, I get it.

    If, however, you are intending to document system behavior, I would offer than writing these specs a thin layer up from the class is a more long-lived approach.

    Your test class:

    Might read:

    I know that the first expression feels solid and comfortable for a developer, but might the point be to make this report useful for both a dev and a business user?

    Also, your mention of tying class names to classes is well made. It is a pretty big deal, actually, for refactoring and code cleanliness.

    In short, what’s wrong with that one level higher approach, really?

    David Starr

    June 26, 2011 at 10:16 am

  2. David,

    Yes, this is documenting a non-trivial class that has a few states that it needs to be driven through (it’s not actually an invoice, I just used that as an example). So ordering the specs was key to getting docs that make sense.

    I hear you about moving up one level of abstraction. That way the specs are helpful from both a business standpoint and a developer standpoint. A dev can always look at the spec to see the names of methods/properties he needs to use for that scenario, and that way I lose the problem of having the docs get out of sync with the code during refactoring. I really like that middle ground. Thanks for pulling me back down to earth :-)

    I do still think that separating out (and ordering where it makes sense) the specs for the public API makes a lot of sense.



    June 26, 2011 at 10:46 am

  3. Great thoughts. This is definitely an important insight. Take a look at relishapp (here is the rspec site on it for an example of really executable documentation.
    Also, when I’m working, I try to always run my description-view (in rspec, it is rspec -fd), which just displays my descriptions, contexts and example names. I can look to make sure that they represent the appropriate level of abstraction.


    June 26, 2011 at 10:47 am

  4. David,

    After going back into my code, I realized that in the domain, the class names and API names reflect the ubiquitous language of the domain, so there’s really no harm in using these terms explicitly, since they *do* reflect the business concepts. And if they don’t, there’s a problem somewhere.

    So I’ll keep in mind the idea of moving up a level of abstraction, but when spec’ing the domain, I’d argue that you shouldn’t find yourself needing to do this too much.

    For example, When_creating_an_Invoice (note the capital I on Invoice) works well in both the business sense *and* in the developer sense, because there isn’t any difference between the two.



    June 26, 2011 at 11:15 am

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Get every new post delivered to your Inbox.

Join 213 other followers

%d bloggers like this: