We have just completed our annual report exercise. Since our law library is part of a larger institution, our report is only a small portion of an aggregate. Internally, the report is often a roll up of so-called benchmark reporting: where are you on the metrics compared to where you were last year. The most important part of this data collection and analysis is behind the scenes, and is what steers a law library through the next year. But it’s important to bring the stories to the surface to explain the library’s value.
Benchmarks are Questionable
There’s a lot I dislike about the annual summary approach. Not that funders or decision-makers don’t need regular updates. That makes complete sense and it’s vital for a law library to explain what it is doing, not least what it is doing with the money it’s given.
What does strike me is that the approaches tend to be what’s easy, than what’s useful. In particular, the term benchmark is unhelpful. At the end of the year, we are scratching a notch on a tree. Next year, we’ll compare to see if our new notch is higher or lower than the last one. And the difference means … what, exactly? If the format allows only for reporting out numbers Year X against numbers Year Y, it becomes reporting for the sake of reporting, without any value.
There was a great example of numbers as nonsense from an analysis of two public library branches in the UK. Libraries A and B were going to close based on comparisons with Library C:
- Library A is open 9 hours a week, 288 “active” borrowers, and has 3,800 annual visitors in an area of 6,786;
- Library B is open 24 hours a week, 1,300 active borrowers, and has 31,000 annual visitors in an area of 15,139;
- Library C is open 34 hours a week, 3,270 active borrowers, and has 67,800 annual visitors in an area of 37,078
The reality is that there are cost issues. Assuming that all staff are working at the same level, then there must be reasons for the differences. Personally, I think low hours are a death knell for a library, and are likely to cause potential users to use another library. If I had my druthers, I’d adopt an hourglass scheduling system, bulking up staff in the morning and evenings, because I think that reflects when the non-courthouse bar works, and why lawyers like remote access to resources. But adding hours is money, just as better placement of facilities is a cost, let alone expanding a collection.
It’s not fair to assume the municipal government didn’t consider more than these numbers, they did. But there’s a value for money approach that is based on just this data. Benchmarking feeds this purpose, and, I think, is part and parcel of the how do we show law library value issue. I know that, when I consider the public libraries I use – I’m unusual, I’m sure, in that I have cards at 3 public libraries – it’s the details that matter:
- I’m more likely to visit libraries that are open longer hours or are open in the evenings when I can get to them;
- I’m more likely to use a library that has convenient parking, even if it’s further away
It’s important that we understand what makes someone use a law library and what inhibits them. It could be where we are located, when we’re open, what we have available. But it usually can’t be boiled down simply to numbers. Benchmark numbers are, for me, like combination locks. The question is what you’re trying to unlock by changing a dial. I’m going to return to them in a second but I want to touch on something strategic.
Timing is Everything
As I’ve noted before, once the operational review or strategic consultation or what have you is under way, law libraries are more subject to the winds of benchmarks than they might otherwise be. You can do all the marketing you want but it’s critical to align activities and costs and start telling your stories as early as possible.
I’ve always assumed one of the things that drove the Inner Temple‘s plan to recreate an educational space using part of the law library is that CLE makes more money than law libraries. When someone looks at law library space and measures footfall – visitors, foot traffic – against value, it’s a losing proposition. In particular, if law libraries rely on anything that is supported by providing space, I think that becomes more and more of a challenge outside academia, where the libraries play an archival role.
Go Beyond Benchmarks
What’s easy is to measure things that are discrete: books off shelves, database searches, reference questions, foot traffic. There’s value in that data. I don’t want to overstate the value they can deliver in highlighting trends – sometimes changing numbers don’t mean what they seem. But none of these things really measure that value for money which funders care about.
I tend towards reporting out less data and looking more for the stories that might show value, because the numbers don’t by themselves. This means, at year beginning, thinking about which combination tumblers are going to get a spin. And which aren’t, since each part of the combination lock requires money.
I’ll use reference statistics as an example because they’re so universal. Like many law libraries, we count three levels of service interaction, what I’ll term easy, medium, and hard. They may reflect effort, but they may merely represent time effort rather than expertise effort.
At one time, our library did what is the least valuable data gathering, which is to measure for 4 weeks and call it a year. I’m not a fan of this type of cherry picking and extrapolating data out like that suggests to me not only that there’s a good chance it’s wrong, but that other opportunities are missed to understand library operations. When staff are reluctant to count everything – and one argument not to is that they will miss some interactions !? – it can be useful to explain to them that the library’s strategic position relies on their ability to measure what they do.
Our reference numbers are slightly down this year. When we shifted from the 1-month-is-a-year approach, we had a huge drop in reference questions answered. A few years ago, we had another, smaller drop. Then it rose again and has since plateaued.
The drop a few years ago occurred when we had an open position. The reality is that reference happens based on the number of people available to do it. The story about our reference numbers isn’t that they have gone down 1% this year when they’d gone up 1% last year. It’s this: that the reference staff served the equivalent of half of our organization’s membership. They’re working 100% of the time, and if we want those numbers to continue to go up, we need to reallocate resources.
Here’s the threshold question: do we need those numbers to rise? Is it strategic for the library to increase the number of reference questions answered? Is there currently unmet need? does higher reference interactions mean happier customers or more competent legal professionals?
If the answer is yes, then it’s worth it putting in additional resources. Reallocate staff from other, less strategic activities. If you have no other staff, then it’s a matter of rejigging the time the solo law librarian can dedicate to reference. I’m assuming most of us aren’t going to be able add new people, although if the demand is there, and your law library is aligned with its organization, it’s a pitch worth making.
If the answer is, no, that’s a decision too. There is still a need to discuss, as part of any annual report or internal reporting, what positive and negative fluctuations mean. Given a stable audience and a stable staff and stable funding and collection, fluctuations may not be something you can impact. It could have been a cold winter, a hot summer, the G20 summit protests, and so on.
In our case, we’ve decided to convert a technical position into a reference one, and I expect our reference numbers will go up accordingly. It’s not magic. But that is a reflection based on what that position had previously been doing, and that those resources would be better spent answering questions.
The new year is a good time to go through your annual report, your data points gathered from last year and your trends, and see where the stories are about the value you provide. The stories aren’t the data, they only give you a skeleton. Don’t just report out and move on; make decisions to change, or not. If your story isn’t where it should be in January, now is the time to start making changes so that you can be in a position to tell the story next year.