Jinfo feedback - market research report providers
Jinfo Blog

By Robin Neidorf

Abstract

We answer a customer query on how to evaluate the value for money and quality of off-the-shelf market research reports.

Item

Jinfo Consulting recently answered a customer query about a publisher of market research reports.

Our customer had never heard of this particular provider and needed to make a quick decision about the value of the report in question - a niche report in the pharmaceutical field, projected at 30 pages and costing around US$4,500. For that amount, the publisher was also offering up to 40 hours of analyst time to customise the data.

"We've had these questions from [internal] customers before, and it's very hard for us to evaluate the reports if we don't know the provider," she explained. "The customer wants the report so we purchase it, but then it turns out to be worthless. The customer is upset with us for not knowing this ahead of time, and we're out a fair bit of money.

"How do we make a sound decision, quickly, when we don't have time to do a lot of research?"

She sent me the materials the publisher had provided, along with copies of emails they exchanged about methodology and coverage.

Jinfo's response

Asking about methodology is certainly the first step in vetting a research provider; the publisher's description of the methodology was pretty vague, however, with claims of "proprietary algorithms" preventing provision of more detail.

So I took a closer look at the samples, marketing materials, website and other online resources. 

Here's our quick list of tasks for getting an initial sense of the value of a market research provider:

• Writers and researchers: Does the firm name their analysts? Do reports have bylines on them?
  
  
• Glassdoor: What do employees have to say about the business? How long have they been there? What job titles do they have?
  
  
• LinkedIn: How does the company describe themselves? Look at employees - how long have they been there? Do you see senior-level people who have been there three or more years, with the appropriate training to do the analysis you need?
  
  
• Website, collateral, samples: Are specific sources named? Do they provide insight as to their methodology, or is the description pretty generic? Are there typos? Do samples include bylines? Are there customer testimonials about the value and accuracy of the reports? 
  
  
• Gut reaction on pricing: Is the price a reasonable offer for the value they seem to be promising?

In this case, I found red flags everywhere:

The website did not list analysts or provide any validating information about the quality of the people doing the research. I found numerous typos and broken links when I dug beyond the initial pages.

Employee comments and listings on Glassdoor and LinkedIn were primarily for marketing staff, most of whom described their role as relating to search engine optimisation: they represent a significant investment in making the website discoverable via search engine. 

The few analysts I found as employees on LinkedIn were relatively junior - in the first three years of their careers. Most employees I found on either site indicated they had been with the company less than a year.

The collateral and sample materials included typos but not bylines. There were no testimonials from satisfied customers. One brochure claimed that "advanced analysis using Excel" was part of their process - an odd claim to make in 2017, when advanced market research techniques would be relying on more sophisticated tools.

Finally, my gut reaction on pricing was negatively impacted by the promise of 40 hours of analyst time. If all they provided was the analyst time for the quoted price of $4,500, the hourly rate comes to US$112.50. Either they don't expect many customers to use this time, or these are extremely junior analysts who will be conducting the work.

This quick review took around 20 minutes to complete, with a bit more time to document the results for our client. She was able to bring this information to her internal customer, who was grateful to not waste the money.

Further discussion on the topic

At the Content Buyers' Breakfast hosted by the Leadership and Management Division at the SLA annual conference last June, we had a very interesting discussion about exactly these types of publishers and the concerns of content buyers.

Some of the concerns raised:  

• Methodology: If you are building business plans, developing new strategy or making any funding requests based on the report, you need to understand the methodology in detail; most of these publishers will only provide vague descriptions, stating that their processes are confidential.
  
  
• Analysts: When consultation time is included in the purchase, the quality of the analysts tends to be of a relatively low standard

• Bias: When they offer deeper analysis of datapoints or slices of data, some content buyers/customers have heard a pretty scary question, "What do you WANT the data to say?" That is, they're shaping the data and analysis to meet a desired outcome in order to please a customer, rather than conducting a proper, unbiased analysis.

Bringing these concerns to an internal customer can kick off valuable discussions about what they truly need and what they're willing to spend in order to get it. What would make US$4,500 "money well-spent"? You might be able to come up with a different source or approach to meet their needs, thus performing as a strategic partner rather than a transaction centre.

Have a question for Jinfo Consulting? Learn more about our capabilities, then start the conversation today.

« Blog