Product Evaluations: Completing a Needs Assessment
Jinfo Blog

By Robin Neidorf

Item

Over the course of doing business, every organisation encounters the need to conduct product evaluations. Sometimes evaluations are no more complex than taking advantage of a free trial period. For larger purchases, however, as well as products that require the input of many different stakeholders, it's important to approach product evaluation with the same rigour and attention to detail as any other research project.

A needs assessment is a critical step in creating a product evaluation plan that truly incorporates the unique needs of an organisation. This article, adapted from the recently published FreePint FUMSI report, Product Evaluation Report and Tool Kit, offers three approaches to building needs assessment tools:

• Surveys
  
• Focus Groups
  
• Workplace Observations

Surveys

Surveys are effective for gathering input from stakeholders, particularly those based in many different locations. Surveys can be conducted electronically through Web-based tools or the old fashioned way, on paper. If you haven't written surveys in the past, you may be surprised at how much planning and thought must go into crafting one to ensure you get good information.

This sample survey is designed to elicit user needs regarding a specific type of information. The survey could easily be implemented through a Web-based tool, making data collection and analysis much easier for the project manager.

Financial Information Needs Assessment The research centre is in the process of evaluating two database products to license for internal use. In order to help us make the right decision, we ask you to take 5 minutes to complete this brief survey about your needs and preferences for such a product. If you have any questions about this process or how the data will be used, please contact the research centre at extension 3353.

How frequently do you need or request financial information?

• Multiple times every day
  
• Once a day
  
• Three or four times a week
  
• Fewer than three times a week

When you access or request financial information, how critical is the data to your ability to perform your job?

• Essential - I can't work without it
  
• Important but not absolutely essential * Nice to have but not a real problem if missing

Are you familiar with the following database products?

The following list of features is among the evaluation criteria for these products. Please rate the following features based on how important they are to your work with financial information:

What other features not named above are important to you in a financial database product or service?

Quick Tips for Crafting Surveys

• Be Brief: Your survey should be as short as possible and ideally take fewer than 10 minutes to complete.
  
• Question Arrangement: Be sure to create logical flow to the order of your questions - don't jump from one topic to another without raising a few 'signposts' to guide your respondents.
  
• Simplify your language: 'What is the frequency of your usage of the information centre over the last 30 days?' is better understood as, 'About how many times have you sent a request to the information centre in the last 30 days?' Relax your grammatical standards if the questions sound too formal.
  
• Assure a common understanding: Write questions that everyone will understand in the same way.
  
• Avoid leading questions: Leading questions demand a specific response. Here's an example of a leading question:

Which of our premium databases is your preferred resource for news items?

As written, the leading question assumes that the premium database collection is the preferred resource - rather than the open Web, for example.

• Avoid 'double-barrelled' questions: Double-barrelled questions are difficult to answer and impossible to interpret. Here's an example of a double-barrelled question:

Is the Info Update newsletter interesting and useful?

The newsletter may be very interesting but not very useful.

• Limit open-ended questions: The more open-ended (or 'fill in the blank') questions you include in your survey, the less likely respondents are to provide thoughtful and meaningful responses, and the less likely you are to get a high response rate. Open-ended questions also require much more work in analysis and interpretation.

Types of Questions and How to Use Them:

Focus Groups

Like surveys, focus groups offer an opportunity to gather input directly from stakeholders regarding their wants and needs. One advantage of focus groups is that they create a forum for give-and- take. The group process can be creative, and a good focus group can surface needs that an evaluator did not even consider.

On the other hand, focus groups require more coordination of schedules, and reviewing results can be time-consuming since so much of the data is unstructured. And although more and more researchers are conducting online, asynchronous focus groups, it is still much more common for focus groups to be conducted in person, meaning that participants must be brought together at the same time and place.

Preparing for a focus group with a good discussion outline lays the groundwork for success and sets appropriate expectations for everyone participating in the group. The provided sample outline for a discussion guide is a starting point for adaptation.

This sample focus group discussion guide demonstrates an approach to organising a focus group session.

Introduction and Welcome

Explain Purpose Prior to conducting an in-depth product evaluation, we want to gather input from you regarding your needs for this and similar products.

General Instructions for Session:

• No right or wrong answers
  
• Your opinion is what counts
  
• Provide candid input
  
• Lasts about 1 hour
  
• Confidentiality guaranteed.

1. What do you consider to be the most useful product on your desktop? Why? (Surface preferences and essential workflow needs based on responses)

2. How would you go about replacing that product if it weren't available anymore?

3. We're evaluating products for our market research collection. What are the products you are most interested in for this collection? Why? (Listen for products on our evaluation list)

4. Straw poll: On the flip chart is a list of product features. When I read out loud the feature you think is most important for a market research product, please raise your hand. (Look for clusters; generate additional discussion around important features)

5. Let's look at different ways you use the output of a market research product in your work. Talk us through a typical project and the ways market research is essential, important or just 'nice to have' as part of the project. (Let members share and note needs around workflow integration, frequency of usage, level of detail required, etc)

6. Summarise priorities and thank participants for their time.

Tips for Successful Focus Groups

• Determine your purpose - What do you need to know that you don't know?
  
• Select the proper group size (usually 6-10), composition, and time
  
• Establish a recording method
  • Recording
    
  • Note-taker
    
  • Flip charts
    
  • Other method

• Recruit participants based on purpose
  • Bringing together 'like minds' is more effective than striving for diversity in each group
    
  • Be sensitive to internal politics

• Plan for no-shows
  

• Craft your questions ahead of time
  • Consider unique value of group experience - what can you get in a group that you can't get from a survey?
    
  • Pre-test your questions to be sure they are clear and targeted
    
  • Identify which questions you are willing to drop if time runs short

• Manage the group
  • Encourage quiet participants
    
  • Listen for tone
    
  • Keep to time
    
  • Reflect back what you hear to ensure understanding
    

• Write results as soon as possible following the session to capture insight.

Workplace Observation

Surveys and focus groups are both needs assessment tools that rely on users' self-reported needs and preferences. But users are not always accurate when they self-report things like the amount of time certain tasks take or the number of steps a project takes. Workplace observations can be a helpful needs assessment tool when decision- critical information demands accuracy in these areas. An external observer can time tasks, record steps and ask clarifying questions about tasks that the worker may not even be aware of doing.

Workplace observations are time-consuming and are used much more rarely than surveys or focus groups. For significant cost-savings related to worker time and tasks, however, they are more reliable than self-reported data.

This sample workplace observation outline is designed to capture information about workflow and hidden costs of search associated with time:

Database log-ins during 4-hour period (add hash mark for each log-in):

Duration of sample search sessions (time with stopwatch):

First Steps, Next Steps

A needs assessment is part of a whole approach to product evaluation that starts with defining evaluation criteria, continues through methodology design and data gathering, and ends with purchase recommendations. Incorporating a needs assessment helps ensure that the product evaluation actually meets user needs. Most business products will do what they say they will do - a content management system will manage content; a premium content database will provide access to premium content. The needs assessment connects a product's features and functionality directly to user needs, enabling decision- makers to choose products best suited to the enterprise.

Related FreePint links:

• FUMSI <http://www.freepint.com/fumsi/>

« Blog