Product Evaluations: Completing a Needs Assessment
Jinfo Blog
31st March 2007
Item
Over the course of doing business, every organisation encounters the need to conduct product evaluations. Sometimes evaluations are no more complex than taking advantage of a free trial period. For larger purchases, however, as well as products that require the input of many different stakeholders, it's important to approach product evaluation with the same rigour and attention to detail as any other research project.
A needs assessment is a critical step in creating a product evaluation plan that truly incorporates the unique needs of an organisation. This article, adapted from the recently published FreePint FUMSI report, Product Evaluation Report and Tool Kit, offers three approaches to building needs assessment tools:
- Surveys
- Focus Groups
- Workplace Observations
Surveys
Surveys are effective for gathering input from stakeholders, particularly those based in many different locations. Surveys can be conducted electronically through Web-based tools or the old fashioned way, on paper. If you haven't written surveys in the past, you may be surprised at how much planning and thought must go into crafting one to ensure you get good information.
This sample survey is designed to elicit user needs regarding a specific type of information. The survey could easily be implemented through a Web-based tool, making data collection and analysis much easier for the project manager.
Financial Information Needs Assessment The research centre is in the process of evaluating two database products to license for internal use. In order to help us make the right decision, we ask you to take 5 minutes to complete this brief survey about your needs and preferences for such a product. If you have any questions about this process or how the data will be used, please contact the research centre at extension 3353.
How frequently do you need or request financial information?
- Multiple times every day
- Once a day
- Three or four times a week
- Fewer than three times a week
When you access or request financial information, how critical is the data to your ability to perform your job?
- Essential - I can't work without it
- Important but not absolutely essential * Nice to have but not a real problem if missing
Are you familiar with the following database products?
Product | 1 - not familiar | 2 | 3 | 4 - very familiar | Tick if you have ever used |
Donaldson's Reports | |||||
Davidson's Reports |
The following list of features is among the evaluation criteria for these products. Please rate the following features based on how important they are to your work with financial information:
Feature | 1 - not important | 2 | 3 | 4 - essential | N/A |
Downloadable reports - spreadsheet format | |||||
Downloadable reports - other format | |||||
Customisable taxonomy and indexing | |||||
Ability to associate work with project codes | |||||
Integration with news database(s)/results | |||||
Graphical reporting within product (eg, charts, maps, etc.) | |||||
Desktop access | |||||
User-defined access to results (eg, personal folders for saved work) | |||||
Online training | |||||
Live chat support |
What other features not named above are important to you in a financial database product or service?
Quick Tips for Crafting Surveys
- Be Brief: Your survey should be as short as possible and ideally
take fewer than 10 minutes to complete.
- Question Arrangement: Be sure to create logical flow to the order of
your questions - don't jump from one topic to another without
raising a few 'signposts' to guide your respondents.
- Simplify your language: 'What is the frequency of your usage of the
information centre over the last 30 days?' is better understood as,
'About how many times have you sent a request to the information
centre in the last 30 days?' Relax your grammatical standards if
the questions sound too formal.
- Assure a common understanding: Write questions that everyone will
understand in the same way.
- Avoid leading questions: Leading questions demand a specific response. Here's an example of a leading question:
Which of our premium databases is your preferred resource for news items?
As written, the leading question assumes that the premium database collection is the preferred resource - rather than the open Web, for example.
- Avoid 'double-barrelled' questions: Double-barrelled questions are difficult to answer and impossible to interpret. Here's an example of a double-barrelled question:
Is the Info Update newsletter interesting and useful?
The newsletter may be very interesting but not very useful.
- Limit open-ended questions: The more open-ended (or 'fill in the blank') questions you include in your survey, the less likely respondents are to provide thoughtful and meaningful responses, and the less likely you are to get a high response rate. Open-ended questions also require much more work in analysis and interpretation.
Types of Questions and How to Use Them:
Type of Question | Example | Results in ... | Challenges |
Rating scale | Please indicate on a scale from 1 - 5, where 1 = 'not at all' and 5 = 'completely' your agreement with the following statements:[list] | Numbers that you can average or analyse across a spread (i.e., how many selected 1, how many selected 2, etc.) | Respondents may have different internal measures for how they rate items - one person's '3' may be another's '5.' Unless your survey has been vetted by a statistician, be sure to keep averages and spreads in perspective with other sources of data. |
Multiple option lists | What were your reasons for enrolling in the online training programme? Please check all that apply: | Spread of responses from all respondents across specific categories of measurement (e.g., 75 % of respondents selected Option 1) | Can create some 'muddiness' in the results if respondents select lots of options - difficult to tell what's really important to them; also be sure to include an 'other' category. |
One-choice option lists | What was the primary reason you signed up for the Expert Search Workshop? Select only one: | Indicator of specific preferences and/or needs | Forces respondents to single out a response, when they might prefer to select several; often helpful to combine with a multiple option question first, followed by a one-choice option question. |
Open-ended questions | What other topics would have been helpful to have covered in the programme? | Respondents' unique and specific input | Reviewing and tabulating responses can be difficult, if responses do not easily fall into identifiable groups; many respondents skip these questions entirely because they take more time and/or thought than they want to put into the survey. |
Open-prompt questions | What day of the month do you prefer for:Receiving the Info Update newsletter: [entry]Onsite Seminars: [entry]Project billing reports: [entry] | Respondents' unique and specific input on particular areas or questions | Reviewing and tabulating responses is time-consuming (though less so than for open-ended questions). |
Focus Groups
Like surveys, focus groups offer an opportunity to gather input directly from stakeholders regarding their wants and needs. One advantage of focus groups is that they create a forum for give-and- take. The group process can be creative, and a good focus group can surface needs that an evaluator did not even consider.
On the other hand, focus groups require more coordination of schedules, and reviewing results can be time-consuming since so much of the data is unstructured. And although more and more researchers are conducting online, asynchronous focus groups, it is still much more common for focus groups to be conducted in person, meaning that participants must be brought together at the same time and place.
Preparing for a focus group with a good discussion outline lays the groundwork for success and sets appropriate expectations for everyone participating in the group. The provided sample outline for a discussion guide is a starting point for adaptation.
This sample focus group discussion guide demonstrates an approach to organising a focus group session.
Introduction and Welcome
Explain Purpose Prior to conducting an in-depth product evaluation, we want to gather input from you regarding your needs for this and similar products.
General Instructions for Session:
- No right or wrong answers
- Your opinion is what counts
- Provide candid input
- Lasts about 1 hour
- Confidentiality guaranteed.
1. What do you consider to be the most useful product on your desktop? Why? (Surface preferences and essential workflow needs based on responses)
2. How would you go about replacing that product if it weren't available anymore?
3. We're evaluating products for our market research collection. What are the products you are most interested in for this collection? Why? (Listen for products on our evaluation list)
4. Straw poll: On the flip chart is a list of product features. When I read out loud the feature you think is most important for a market research product, please raise your hand. (Look for clusters; generate additional discussion around important features)
5. Let's look at different ways you use the output of a market research product in your work. Talk us through a typical project and the ways market research is essential, important or just 'nice to have' as part of the project. (Let members share and note needs around workflow integration, frequency of usage, level of detail required, etc)
6. Summarise priorities and thank participants for their time.
Tips for Successful Focus Groups
- Determine your purpose - What do you need to know that you don't
know?
- Select the proper group size (usually 6-10), composition, and time
- Establish a recording method
- Recording
- Note-taker
- Flip charts
- Other method
- Recording
- Recruit participants based on purpose
- Bringing together 'like minds' is more effective than striving for
diversity in each group
- Be sensitive to internal politics
- Bringing together 'like minds' is more effective than striving for
diversity in each group
- Plan for no-shows
- Craft your questions ahead of time
- Consider unique value of group experience - what can you get in a
group that you can't get from a survey?
- Pre-test your questions to be sure they are clear and targeted
- Identify which questions you are willing to drop if time runs short
- Consider unique value of group experience - what can you get in a
group that you can't get from a survey?
- Manage the group
- Encourage quiet participants
- Listen for tone
- Keep to time
- Reflect back what you hear to ensure understanding
- Encourage quiet participants
- Write results as soon as possible following the session to capture insight.
Workplace Observation
Surveys and focus groups are both needs assessment tools that rely on users' self-reported needs and preferences. But users are not always accurate when they self-report things like the amount of time certain tasks take or the number of steps a project takes. Workplace observations can be a helpful needs assessment tool when decision- critical information demands accuracy in these areas. An external observer can time tasks, record steps and ask clarifying questions about tasks that the worker may not even be aware of doing.
Workplace observations are time-consuming and are used much more rarely than surveys or focus groups. For significant cost-savings related to worker time and tasks, however, they are more reliable than self-reported data.
This sample workplace observation outline is designed to capture information about workflow and hidden costs of search associated with time:
Database log-ins during 4-hour period (add hash mark for each log-in):
Duration of sample search sessions (time with stopwatch):
Subject of search | Time started | Duration | Next action following search |
First Steps, Next Steps
A needs assessment is part of a whole approach to product evaluation that starts with defining evaluation criteria, continues through methodology design and data gathering, and ends with purchase recommendations. Incorporating a needs assessment helps ensure that the product evaluation actually meets user needs. Most business products will do what they say they will do - a content management system will manage content; a premium content database will provide access to premium content. The needs assessment connects a product's features and functionality directly to user needs, enabling decision- makers to choose products best suited to the enterprise.
Related FreePint links:
- FUMSI <http://www.freepint.com/fumsi/>
- Blog post title: Product Evaluations: Completing a Needs Assessment
- Link to this page
- View printable version
- Product Evaluations: Completing a Needs Assessment
Saturday, 31st March 2007
AI Roundtable
Community session
24th October 2024
Mitigating risk, evaluating sources of market research; AI roundtable; and vendor training of end users in the pharma sector
Blog posting
2nd October 2024
October 2024 Update
YouTube video
3rd October 2024
- Mitigating risk, evaluating sources of market research; AI roundtable; and vendor training of end users in the pharma sector
2nd October 2024 - Retrieval Augmented Generation and other industry analysis; How to evaluate market research; Vendor takeover strategies
3rd September 2024 - Content purchasing developments, Asia-Pacific Community session, and AI agents
1st August 2024
- The future of news provision (Community) 19th November 2024
- Asia-Pacific Community session (Community) 6th November 2024
- AI Roundtable (Community) 24th October 2024