We have just completed our annual report exercise. Since our law library is part of a larger institution, our report is only a small portion of an aggregate. Internally, the report is often a roll up of so-called benchmark reporting: where are you on the metrics compared to where you were last year. The most important part of this data collection and analysis is behind the scenes, and is what steers a law library through the next year. But it’s important to bring the stories to the surface to explain the library’s value.
Benchmark Against What?
There’s a lot I dislike about the annual summary approach. Not that funders or decision-makers don’t need regular updates. That makes complete sense and it’s vital for a law library to explain what it is doing, not least what it is doing with the money it’s given.
What does strike me is that the approaches tend to be what’s easy, than what’s useful. In particular, the term benchmark is unhelpful. At the end of the year, we are scratching a notch on a tree. Next year, we’ll compare to see if our new notch is higher or lower than the last one. And the difference means … what, exactly? If the format allows only for reporting out numbers Year X against numbers Year Y, it becomes reporting for the sake of reporting, without any value.
And I think that can do a disservice, because people start to make comparisons solely based on apples and apples. There was a great example of numbers as nonsense from an analysis of two public library branches in the UK. Libraries A and B were going to close based on comparisons with Library C:
- Library A is open 9 hours a week, 288 “active” borrowers, and has 3,800 annual visitors in an area of 6,786;
- Library B is open 24 hours a week, 1,300 active borrowers, and has 31,000 annual visitors in an area of 15,139;
- Library C is open 34 hours a week, 3,270 active borrowers, and has 67,800 annual visitors in an area of 37,078
The reality is that there are cost issues. Assuming that all staff are working at the same level, then there must be other reasons for the differences. Personally, I think low hours are a death knell for a library, and are likely to cause potential users to use another library. If I had my druthers, I’d adopt an hourglass scheduling system, bulking up staff in the morning and evenings, because I think that reflects when the non-courthouse bar works, and why lawyers like remote access to resources. But adding hours is money, just as better placement of facilities is a cost, let alone expanding a collection.
It’s not fair to assume the municipal government didn’t consider more than these numbers, they did. But there’s a value for money approach that is based on just this sort of data comparison. Benchmarking feeds this purpose, and, I think, is part and parcel of the how do we show law library value issue. I know that, when I consider the public libraries I use – I’m unusual, I’m sure, in that I have cards at 3 public libraries – it’s the details that matter:
- I’m more likely to visit libraries that are open longer hours or are open in the evenings when I can get to them;
- I’m more likely to use a library that has convenient parking, even if it’s further away
Collection hasn’t been a deciding factor and I don’t use many services other than the book, ebook, and database collections.
It’s important that we understand what makes someone use a law library and what inhibits them. To that extent, benchmark data gives us operational information. The numbers can help us to understand some of our interactions, but, reported out raw, don’t necessarily say anything on their own. Usage could be based on where we are located, when we’re open, what we have available.
But it usually can’t be boiled down simply to numbers. Sometimes it’s that one lawyer has a particular client whose case requires research that she can’t do with her own research tools. There’s probably no way to make the law library an ongoing service for that person; and they obviously know its there when they need it, so marketing is unlikely to bump that needle.
Benchmark numbers are, for me, like combination locks. The question is what you’re trying to unlock by changing a tumbler. I’m going to return to them in a second but I want to touch on one other thing.
Timing is Everything
As I’ve noted before, once the operational review or strategic consultation or what have you is under way, law libraries are more subject to the winds of benchmarks than they might otherwise be. You can do all the marketing you want but it’s critical to align activities and costs and start telling your stories as early as possible.
I’ve always assumed one of the things that drove the Inner Temple‘s plan to recreate an educational space using part of the law library is that CLE makes more money than law libraries. When someone looks at law library space and measures footfall – visitors, foot traffic – against value, it’s a losing proposition. In particular, if law libraries rely on anything that is supported by providing space, I think that becomes more and more of a challenge outside academia, where the libraries play an archival role.
Go Beyond Benchmarks
What’s easy is to measure things that are discrete: books off shelves, database searches, reference questions, foot traffic. There’s value in that data. I don’t want to overstate the value they can deliver in highlighting trends – sometimes changing numbers don’t mean what they seem. But none of these things really measure that value for money which funders care about.
I tend towards reporting out less data and looking more for the stories that might show value, because the numbers don’t by themselves. This means, at year beginning, thinking about which combination tumblers are going to get a spin. And which aren’t, since each part of the combination lock requires money.
I’ll use reference statistics as an example because they’re so universal. Like many law libraries, we count three levels of service interaction, what I’ll term easy, medium, and hard. They may reflect effort, but they may merely represent time effort rather than expertise effort.
At one time, our library did what is the least valuable data gathering, which is to measure for 4 weeks and call it a year. I’m not a fan of this type of cherry picking and extrapolating data out like that suggests to me not only that there’s a good chance it’s wrong, but that other opportunities are missed to understand library operations. When staff are reluctant to count everything – and one argument not to is that they will miss some interactions !? – it can be useful to explain to them that the library’s strategic position relies on their ability to measure what they do.
Our reference numbers are slightly down this year. When we shifted from the 1-month-is-a-year approach, we had a huge drop in reference questions answered. A few years ago, we had another, smaller drop. Then it rose again and has since plateaued.
The drop a few years ago occurred when we had an open position. The reality is that reference happens based on the number of people available to do it. So what’s our story?
Here it is: our reference staff served the equivalent of half of the lawyers and paralegals our organization regulates. Not only that, we’re serving particular segments, including articling (apprentice) lawyers who are at a critical point in their career and who are heavy users of our services to support their competence.
I think that’s a clearer picture of what we’re doing than that our reference numbers have gone down 1% this year when they’d gone up 1% last year. That doesn’t explain anything. Our staff is working full out, every day. A 1% shift in either direction isn’t measuring that they put in the same amount of effort as last year.
Now the question becomes, what next? Is it important for us know whether those 20,000 interactions were with 10 people or 10,000? Is it important for us to somehow have more interactions? or interact with a higher percentage of regulated licensees? Is there unmet need, calls that are dropped before they’re answered? Should we be dedicating (embedding) library services for a practice group or discipline counsel?
This is the behind the scenes analysis that will decide if we’re going to do anything about the number of reference interactions. We can’t lower the price of our product to generate interest. We can’t put our product in more stores. And, at the end of the day, we don’t have any hard numbers showing that our work is causing lawyers to be less incompetent. If we decided that we needed to increase our reference interactions, then we need to reallocate resources. If we decide that we have the resources allocated but we want more detailed data, we need to change how we record interactions.
January is a good time for that: shift what people are doing, eliminate projects to free up time or retask people not currently doing reference to do it. Go through your annual report, your data points gathered from last year and your trends, and see where the stories are about the value you provide. The data aren’t the story, they only provide you the foundation. Don’t just report out and move on; make decisions to change, or not. If your story isn’t where it should be in January, now is the time to start making changes so that you can be in a position to tell the story next year.