Evidence summaries and a culture of criticism

As always, Thursday afternoon's EBL class sparked off a number of thoughts and questions, exactly as intended. I cannot ask for a better outcome than to do this. Focus of the class was the journal club discussion/critical evaluation of this article:

McKnight S. & Berrington, M. (2008, March). Improving Customer Satisfaction: Changes as a Result of Customer Value Discovery. Evidence Based Library and Information Practice, 3(1):33-52.

The process we've been using for this core part of the class is to rotate as facilitators, responsible for assigning the use of a particular critical evaluation tool (these being the same ones provided to the EBLIP journal's Evidence Summary team) then leading analysis of an article that was previously chosen. In this case, it was chosen because Joanne Marshall will be talking about Servqual and Libqual next week. The author of this work, however, chose to use a model for user satisfaction imported from marketing. I also frankly wanted to see how an original research article published in the Evidence-based Library and Information Practice journal might fare.

We kept returning to the culture of LIS, and danced as we've done before around the idea and attitude toward criticism of the literature, sensitive to the difficulties many in LIS have to negativity. Some feel that in order to encourage more robust literature, potential contributors should be approached more gently, more positively. An extension of this is the aversion to cultures in some disciplines where criticism involves spiked fingernails or even poisoned sledgehammers, no-holds-barred nastiness in which one (apparently) must engage in order to be taken seriously as a researcher.

It seems undeniably to be the case though, that in LIS generally (how do you like my 'weaselly' words, spoken as a true grad student!), we are averse to, unused to, and even hypersensitive to criticism. Several times, when I've discussed my EBL interests or focus with well-respected LIS professionals (including those with national and even international reputations), they've responded with what I could only term a sort of culturally-entrenched insecurity. "When you talk about EBL," one said, "I feel that I've been doing it wrong, haven't done enough." These administrators do not refrain from conducting research, but if they feel this way - how do others feel?

As an aside, I regard EBL (or some of its tools, though they aren't affiliated with EBL alone) particularly critical evaluation of the literature, to be a cultural tool. One student said yesterday that she will not read research the same way again, after close and organized readings, and creating an 'evidence summary' paper. Setting aside for a moment the question of whether we do read the literature - isn't this changed view of it something we need to be seeking as a profession? We must (I say, climbing once more onto a wellworn soapbox) engage with our own literature, as a community.

How do critical evaluation tools help us to deal with this sensitivity? Shall we decide that if librarians are touchy, tant pis, and move on anyway, kniving sharply with tools meant for different cultures more accustomed to such stern looks - and potentially, deterring cautious attempts by those who are alienated by such an approach?

To think about, here: is the culture from which EBP arises more patriarchal, positivist, despite recent trends toward the integration and evaluation of qualitative research? How might more feminist models find space in our adaptation of the EBP model? For consideration:

Rogerian argument: http://www1.esc.edu/personalfac/hshapiro/professional_communications/advice_outline.htm

The more traditional & patriarchal model: http://www-rohan.sdsu.edu/~digger/305/toulmin_model.spare

Are there studies of the effectiveness of the critical evaluation tools?

In class last Thursday, we used the HCPRDU mixed method tool to evaluate...... and found it utterly cumbersome, with lots of N/A responses to a case study conducted in university libraries. This UK library setting was the setting for a series of workshops, followed by changes in practices, collections, and services. Patrons were surveyed using paired values for both positive and negative aspects of the libraries based on a model intended for use in the marketing sector.

We discuss what questions would be suited for the evaluation of a bibliometric study, a case study. I think about a question set that one might apply to the results of a survey conducted on Medlib-L, or to a 'how I done it' article if, as may frequently be the case, there is a shortage of material.