In the ever-growing list of “trophies for everyone” industry awards, it is becoming more difficult to decipher real awards from paid advertisements. You should be skeptical if you don’t understand an award process or know the perspective of award sources. What methodology was used? Were solutions tested? What criteria was examined? In this article, I will expose “open industry secrets” and provide tips to help you enhance your investigative prowess.

Introducing Critical Thinking

Before we examine the most commonly touted industry analyst accolades, let’s sharpen our critical thinking skills. What is critical thinking? Critical thinking is the ability to think clearly and rationally. It includes the aptitude to engage in reflective, independent thinking. Someone with critical thinking skills is able to do the following:

  • Understand the logical connections between ideas
  • Identify, construct and evaluate arguments
  • Detect inconsistencies and common mistakes in reasoning
  • Solve problems systematically
  • Recognize the relevance and importance of ideas
  • Reflect on the justification of one’s own beliefs and values

Critical thinking is a disciplined process of skillfully conceptualizing, applying, analyzing, synthesizing, and evaluating information. It considers clarity, accuracy, precision, consistency, relevance, sound evidence, good reasons, depth, breadth, and fairness.

Foundation for Critical Thinking

Source: The Foundation for Critical Thinking

People who think critically are keenly aware of the innately flawed nature of unquestioned human thinking. Emotion, prejudice, bias, complexity, and vested interest do distort personal reality. Critical thinkers recognize these challenges and understand everyone makes mistakes.

Critical thinking can be applied to many areas of our lives – not just analyzing the analysts. We do live in a world filled with fake news, propaganda and conflicting reports. The World Economic Forum predicts critical thinking will rise to become the second most important skill in the Fourth Industrial Revolution.

Future of Jobs Report, World Economic Forum

Source: Future of Jobs Report, World Economic Forum

Questioning Sources

As you start applying critical reasoning to evaluation and award processes, continually develop a checklist of questions to assess evaluation methodology, reasoning and detect potential biases. Here is a partial list of questions that I have created from my own investigations.

  • Who is the source? What is the motive? How are they measured? (Note numerous analyst firms measure analysts performance based on paying vendor and subscriber ratings of analyst calls, reports and other work that they do. This makes it difficult for them to be truly unbiased or share negative market intelligence.)
  • Is the source qualified? Can they impartially review vendors? Has the source received anything of value from the vendor?
  • How is the firm the source works for structured? How do they earn a profit? Were payments made to the firm by evaluated vendors? Why? Were payments required to be included in an evaluation, report or survey?
  • How were vendors involved in the process? Were vendors required to get all the survey respondents? How were respondents verified as actual clients?
  • Were all relevant vendors considered? Was a significant vendor omitted? What inclusion criteria, assessment criteria and process was used? (Note numerous lower tier analyst firms and media groups allow any vendor to be included in evaluations, surveys or reports. The vendors later pay for distribution rights of awards based on countless angles of the findings. Thus, everyone can win an award for use in marketing and sales pitches. Vendors that don’t fare well or make it into top tier analyst reports heavily promote these types of awards for everyone.)
  • What information, data and evidence was collected? How was it gathered?
  • Does the evaluation, survey or report award sample accurately reflect the larger population or does it merely reflect the opinion of vendor curated happy clients and filtered responses? (Note numerous “crowd sourced” surveys and review websites are actually not crowd sourced. Vendors often pay to be listed and then ask happy clients to fill out online forms. You can usually detect this if you see many responses posted over a short timeline.)
  • Did sources conduct “hands-on” tests of solutions or did they merely rely on vendor presentations and/or vendor curated happy client feedback?
  • Did the assessment address topic depth, breadth and complexity? What assumptions were made?
  • Do other sources conflict with the evaluation findings? (Note when you see vendors touting <insert topic> awards, perform a Google search to see how many other companies were also given similar awards. One media group I know sells Top 20 <insert topic> awards aka ads for $3,000 USD with no review criteria. If there is no review criteria, it is a fake award.)

As you delve into these questions, you will uncover a wide variety of “open industry secrets” and known issues with awards that are often overlooked or misunderstood. While almost all sources, reports and awards do provide value, it is important to understand award context for appropriate use and weighting in a vendor evaluation process. Never rely on an award as a shortcut to replace your own personalized vendor shortlist or selection process.

It is well for the heart to be naive and the mind not to be. -Anatole France

Honestly, I find vendor client references, case studies, and testimonials to be far more compelling than  lower tier industry analyst firm or media group awards. Many of the big companies are extremely cautious in providing those types of references. If they do, they had to jump through legal hoops to make sure what they said is true. If someone publicly shares experience or provides a testimonial, you can contact them or their peers to verify the information yourself. I also find asking for feedback on LinkedIn reveals priceless insights that you’ll likely never get anywhere else.