Reference Services Evaluation

  1. Introduction:
    1. Live chat services have become more familiar in libraries in the past few years. They provide opportunities for people to get reference help when they cannot make it to the library. Unlike email they can happen in real time and allow for a more casual interaction, though still professional. This will evaluate the effectiveness and usage of live chat reference service. It will also be focused more on the user side of things.
  2. Planned Participants
    1. What population will be studied?
      1. Those that use the live chat service
    2. How are study participants selected?
      1. When they start a new chat session, they will be asked if they want to participate in a study (so that we can read the transcripts).
    3. Will the study results be representative of the entire population?
      1. It is not feasible for the study results to be representative of the entire population because to participate in chat services, one must have access to the Internet as well as a computer. This leaves out people who have neither. It is possible that some people may use the chat services while using library computers, but for the most part, people using chat services will be doing so away from the library. However, it should be representative of the population that uses the chat services.
    4. What incentives can be offered to encourage people to participate in the study?
      1. Those who participate will be put in a raffle for a couple of gift cards.
  3. Data Collection Methods
    1. Strengths and weaknesses of each method relative to the goals and objectives
      1. Surveys
        1. Surveys work best when you want to know people’s attitudes and opinions on things; in this case, it is their attitudes towards the chat services. Surveys will be the primary method along with observations (analysis of chat logs) because surveys are helpful for the user perspective (Liu, 82).
      2. Observation
        1. This will be important in the evaluation. Observation can help gather statistics and is important in studying behavior (in this case, the interaction online). It can be used to analyze how often they get chat requests, how/ long conversations typically last, how formal or informal a session is, etc. Its weakness in the evaluation of chat services is that one cannot see the body language of either party, so interpretation may not be complete.
      3. Focus groups
        1. The biggest weakness of this in regards to live chat service is that it will probably be very difficult to gather people who use this service since it is all online and they are not physically in the library. The strength of focus groups is that is shows interactions between people involved. In this case, though, it would not be much help because it is not necessary to see how each person interacts with the others. It would not do much harm, though.
      4. Interviews
        1. Interviews can help go more in depth than a survey and better interpret the results of a survey. This could allow the researcher to determine what a user liked and disliked about their experience with the chat service. It could be a problem to get people to come to a library. Interviews over the phone could work, but the researcher would not be able to see body language.
    2. What data collection instruments are available in the research literature that can be used or adapted for the study?
      1. Chat reference software records chat reference statistics (Luo, 80).
    3. How can the instrument be pilot tested?
      1. The researcher will pilot test this on one day (vs a longer period of time) to get an idea of how long it will take to analyze results and see how many people take the survey and if they would be willing to do an interview. Using the results from this one day can help tweak and guide the evaluation.
  4. Environment
    1. What location(s) will be used to collect the data?
      1. The library system that this data will be collected for is an urban library setting that has several branches. However, the chat service takes place online so the analysis would not have to be done at the library (although it is good to have a presence there).
    2. How will study participants at this location be informed about the goals of the evaluation project?
      1. When the participants enter into chat service, they will be asked if they would be willing to take a survey and it will give a short summary of the goals. The librarians involved will be told in person.
  5. Personnel
    1. How will personnel be trained to collect the data?
      1. The library staff who answer the questions through the chat service need to be trained to ask the patron at the end of each session if they are willing to take a quick survey or be part of an interview.
    2. What elements need to be included in the training program for data collection?
      1. How to analyze the data from the chat transcripts (for greater accuracy); the library staff involved need to be trained to ask the patron at end of session if they are willing to take a quick survey
  6. Equipment and Supplies
    1. What specialized equipment and supplies are needed to collect, analyze, and disseminate project evaluation findings?
      1. Chat reference software
    2. How will funding be obtained?
      1. First, the library will be approached to see if they would be able to fund it with their budget. If not, the researcher can look for grants or approach donors who may be willing to fund it.


Luo, Lili. “Chat Reference Evaluation: A Framework of Perspectives and Measures.” Reference Services Review 36.1 (2008): 71-85. Library, Information Science & Technology Abstracts. Web. 24 Oct. 2015.