Evaluating and Monitoring HOLLIS Plus Resources: a proposal to the Digital Acquisitions Committee


The Committee on Electronic Reference Services (COERS)


The Committee on Instruction in Library Use (CILU)


January 1999



COERS and CILU were asked by the Digital Acquisitions Committee to create a process for keeping track of and evaluating HOLLIS Plus resources after they have been acquired. Up until now there has been no formal process for evaluating these resources before contracts are renewed. In addition, OIS has been responsible for responding to user comments for all HOLLIS Plus resources. We hope that the process we have devised will make it possible to monitor HOLLIS Plus resources more efficiently and to distribute the work of responding to resource users more evenly.

When we started looking at this process, we were only looking at what should happen after a resource had been selected for inclusion in HOLLIS Plus. We soon realized that it made more sense to look at the complete process from prospective evaluation of a resource for inclusion, on to monitoring it after selection, and finally to the reevaluation stage. This proposal encompasses all three of these stages.

Goals of the Program

We think that the structure we have laid out will make it possible to:

Prospective Evaluation

It is the opinion of CILU and COERS that pre-purchase evaluations should be a regular aspect of electronic purchase decisions. Resources will be evaluated using the criteria in the appendix of this report.

When looking at a possible addition to HOLLIS Plus, the Coordinator for Digital Acquisitions or the Digital Acquisitions Committee will ask COERS and CILU to convene a group to evaluate it. These committees will be responsible for recruiting people from within and/or outside of these committees to perform the evaluation. In addition to recruiting public services staff, librarians with collection development responsibilities should also be a part of this process.

By recruiting from within and outside of COERS and CILU we will be drawing from a broader base of people to work as evaluators. If we only used members from these two committees, the same few people would be conducting all of the evaluations. By drawing from beyond COERS and CILU we will be able to use the skills and knowledge of all of our colleagues.

The prospective evaluation for single sponsored resources would be somewhat different from the evaluation process of multi-sponsored resources. While COERS and CILU would convene a group to evaluate a multi-sponsored resource, the sponsor of a single sponsored resource would usually conduct their own evaluation.

There may be times when it is appropriate for a group other than COERS and CILU to oversee the process that we have developed. For instance, it may be more appropriate for the Science Libraries Council to oversee the process for science databases. In cases such as this, the oversight group should seek representation from other groups in the library community such as COERS, CILU, and RRIC.

The findings of the evaluators will be distributed to the Digital Acquisitions Committee, HOLLIS Plus Working Group, the Harvard library community, and the vendor, as appropriate.


After a resource has been selected for HOLLIS Plus a steward or stewards would be assigned to monitor it. When we began designing this process we looked at stewardship and reevaluation as parts of the same position. After discussing this we have come to see the stewardship as a somewhat separate duty from the reevaluation. It makes sense to separate these duties for several reasons. First, stewards will perform different functions from the evaluators. The steward will be more of a source of information for the library community, which is an ongoing function. The reevaluation will be a short-term project. In addition, by separating these duties, the position of stewardship will be more manageable for the stewards.

Selection of Stewards

For resources sponsored cooperatively by several libraries, the group overseeing the evaluation of the database would recruit stewards. Libraries sponsoring resources by themselves may designate their own stewards. In either case, the oversight group or the library appointing a steward is responsible for notifying the Coordinator of Digital Acquisitions so that there will be a centralized list of stewards.

Stewards should be recruited in as many ways as possible. In addition to recruiting stewards from COERS, CILU, and the HOLLIS Plus Working Group, we could solicit volunteers via listservs such as hulinfo. Because of the large number of HOLLIS Plus resources we will need to recruit many stewards. It is important that stewards know the resource they are monitoring and use it frequently.

For widely used resources such as Lexis/Nexis and Proquest Research Library there will be several stewards from different parts of the library system. Our rationale for this is that these resources will probably require more work in terms of responding to user comments, etc. In addition, different areas of the university may have different issues or concerns with the resource. Having stewards from these different areas will ensure that these concerns are heard.

When recruiting stewards it will be important to point out the educational and career development benefits of working on this. Hopefully, this will provide an incentive for people to volunteer as stewards.

Stewards of Aggregated Resources (such as Ovid)

For stewards of the aggregated resources such as OVID, there will have to be coordination to share information, concerns, and user comments regarding the interface (as opposed to resource content.) The best way to accomplish this is for these stewards to get together to discuss these issues. For instance, there could be OVID or Webspirs stewards groups.

Steward’s Responsibilities

Time Period for Stewardship Term

The length of stewardship should be tied to the length of the resource contract. The steward would usually finish a term by assisting with the reevaluation.

Keeping a Centralized List of Stewards (Database of Databases)

The Digital Acquisitions Coordinator is responsible for the production and maintenance of the Database of Databases, a record of resources, contract information, licensing requirements, distribution of funding, and sponsors. The names of all stewards should also be kept in this database.

Resources that Should Be Monitored

We should try out this process with resources that the Digital Acquisitions Committee sees as having top priority. Our understanding is that the top priorities are the high use and aggregated services:

Here are some other resources to consider after this program is put in place:

Eventually, all HOLLIS Plus resources that we pay for should be included in this process.


This is a separate function from the ongoing work of a steward. A steward should be one of the reevaluators or, at the very least, a source of information for them. For single sponsored resources there may only be one person conducting the reevaluation. For other resources there will be a group of reevaluators.

Timing of the Reevaluation

Resources should be reevaluated before a contract is renewed. In some cases it will be unnecessary to conduct a full scale evaluation before renewing a contract. If no significant issues have been raised with the resource, it will usually be renewed without going through the reevaluation process. Before renewing the contract the Coordinator of Digital Acquisitions should consult the steward to see if a reevaluation is warranted. It is expected that the reevaluation process will begin six months prior to the expiration date of the contract.

Selection of Reevaluators

For the multi-funded databases, the Coordinator of Digital Acquisitions will notify the appropriate oversight group who will then be responsible for recruiting reevaluators. For single funded databases, the sponsor will be notified that it is time to reevaluate. At this point the sponsor will appoint a reevaluator but they should feel free to ask COERS and CILU for help with this.

Reevaluator's Responsibilities

It will be the responsibility of the reevaluators to make recommendations for the retention or changing of a resource to the sponsoring libraries. In order to make a recommendation, reevaluators will:

Aggregated Resources

As is the case with stewards of resources from the database aggregators (Ovid, etc.) there will have to be coordination among reevaluators to share information, concerns and user comments regarding the interface itself (as opposed to looking at resource content.)

Relationship to Funding Responsibility

There may be times when there is disagreement between funding sources and reevaluators regarding the retention of a resource. In these cases, reevaluators are responsible for presenting the funders with an evaluation of the content, operation, and value to researchers of the resource.

Relationship of Stewards and Reevaluators to the HOLLIS Plus Working Group

The HOLLIS Plus Working Group assists OIS staff in the production and maintenance of the HOLLIS Plus web site. (Looking at whether or not it is technically feasible to add the resource, getting an abstract written, etc.) The stewards and reevaluators are more concerned with the resources after they have been mounted on HOLLIS Plus.

Appendix: Public Services Criteria for Evaluating Networked Resources



Who is the intended audience of this resource? Does this match the needs of our users?


Describe the content of this resource.




Subject scope: does it match the description?


Are there any comparable resources available?


Is this resource available in different platforms? Which platform is preferable?


Is there an easily accessible list of titles included (when applicable)?


Is the resource updated regularly?


Quality control: are there frequent misspellings, inaccurate citations, etc ?



Is there an on-line index and/or thesaurus? Do they work well?


Are there different searching levels?


Can the entire resource be searched at once? Is this an issue?


How intuitive and easy-to-use is this interface?


Is Boolean searching supported?




Is proximity searching supported?


Are truncation and wildcard available?



Can records be easily marked?



Can records be easily e-mailed and/or downloaded?


Is printing easy?


For resources with full-text, any special printing issues to know about?


Evaluate response time.


Evaluate the on-line help



Will instruction for the public be a significant issue with this resource?


Would you recommend that we prepare documentation to support public use of this resource?


Is system down-time kept to a minimum?


Can you summarize community response to this resource (when applicable)?



Any comments on tech support?


Any other comments?





Summary comments: would you recommend adding this resource to HOLLIS Plus