Grappling with Estimative Language

How do you determine whether or not to place judgments such as “probably” or “likely” in your products?  

As analysts, you owe your clients your best judgments and a possible way forward. After reviewing all of the available information and considering alternatives, it is your job to give your readers some idea of the certainty surrounding your judgment(s). This is often done with terms such as “likely,” “almost certainly,” “probably,” and “might.” 

Efforts to standardize the range of certainty have met with limited success. Sherman Kent, considered the pioneer of intelligence analysis tradecraft and methodology, attempted to apply a methodology when determining levels of confidence, and several U.S. intelligence agencies published probability range tables throughout the years to determine the best approach. Following the September 11th attacks, intelligence agencies across the world published charts attempting to tackle this issue. 

Here is a chart from Intelligence.gov:

For comparison, here is the U.K.’s probability scale:

These charts, and many others, are best used as a guide to show the relationship of the terms to one another. One difficulty in selecting a one-size fits all policy when it comes to probabilistic language is the way in which someone interprets the words themselves–what do you mean when you say terms such as likely, probable, might, etc? Another complication involves the recipient of the report –your reader might interpret your judgment in a manner consistent with or supportive of their desired preference or outcome. For example, your client might translate a term, such as “likely,” as meaning 70 or 80 percent if that is a desired outcome, but as only 50 to 60 percent if the outcome is not desired. 

When determining probabilistic language, try not to focus on precision, but keep in mind the following best practices:

• You can view these terms on a sliding scale–use your best judgment and the research in front of you to make your decision on the best terminology. 
  • If you have solid evidence and past history to suggest something is “likely to” happen going forward, go ahead with that judgment. If you are on the fence, or if it would take something to make that judgment shift, perhaps you should instead say “probably” or “might” (see my caution below on the use of “could” or “might”). And it is always helpful to highlight those shifts if they are important caveats (see below). 
• When reviewing your own reports, ensure you have the evidence to back up your judgments. 
  • For example, if you say social media activity is “likely to” increase in the coming month because of a certain issue, but if your evidence suggests the issue is likely to go away in a few days, perhaps your “likely” should be “probably” or “might.” 
    • Or you could instead say–”Social media mentions probably will increase in the coming month if the issue remains a prominent headline in mainstream media.” 
• Speaking of caveats–these are really helpful, particularly if you are on the fence about your judgment. Or if you want to let your client know that if something moves in a certain direction, your judgment will also shift. 
• If possible, try not to use “could” or “might” because these terms do not provide as much useful information for your reader. Anything could happen. Nonetheless, if your client asks a specific question you do not have a great response based on the available evidence, using “could” or “might” is an acceptable response. 
  • For example, if your client asks if a rival company which has expressed little interest in working with them in the past is open to future engagement, you could say something like “Company X might be open to engagement if you present them with the following opportunities…”
  • When using these terms it is always helpful to add a bit more context if you can and let your reader know what might shift the “could” or “might” to a more favorable outcome. This also allows your client to track the accuracy of your assessment over time, monitoring the situation to see if the judgment is growing stronger or weaker based on new information.
• Avoid using two probabilistic phrases in the same sentence.
  • For example, by saying something “might happen because the following conditions are likely to occur.” This is confusing for your reader. Instead, you could say it “might happen because the following conditions are present.”
  • Another example: A presidential election might occur because the National Assembly is likely to approve a new vote. Again, which is it? It seems like a pretty good chance the election will occur if the National Assembly is likely to approve a new vote (if that is a condition for a new election). Instead you could say, “A presidential election likely will occur because the National Assembly is set to approve a new vote.” 
• If you are in a situation where you are struggling between two judgments–do you think something probably will happen or is likely to happen–you can think in percentages to help you make a determination. 
  • But remember, analytic judgments do not convey a solid degree of mathematical precision. Analytic judgments are exactly that–judgments. Judgments which are based on solid evidence and facts, your expertise, and past history and examples. 
• Another helpful way to convey uncertainty in our judgments is to list up front the key assumptions that underpin your analysis. Letting your client know the key assumptions which guide your interpretation of evidence and reasoning about the particular problem at hand, is a helpful way to demonstrate why you think what you think and if something changes, the impact it could have on your analysis.

And always remember…your client is paying for your analysis. As long as you have the research to back up your arguments and judgments, you are on solid ground!

Reference: Critical Thinking for Strategic Intelligence Analysis” by Katherine and Randolph Pherson.