Paul Schoemaker

Making the right decision depends on collecting the relevant data. But even good data can lead you astray if your critical thinking skills aren't sharp.

Our increasingly uncertain world places a great premium on critical thinking, which is about asking the right questions, challenging assumptions, seeing more angles and being aware of thinking biases. For instance, put yourself in the shoes of NATO leaders as they ponder Russian threats to Ukraine. Or closer to home, should supply chain managers rethink their adherence to just-in-time inventory in view of protracted supply gridlocks? And, do you consider current high US inflation to be transitory--linked to fiscal stimuli and Covid-related economic dislocations-- or something more structural and lasting?

That's a lot to ponder. So, to sharpen your critical thinking skills, here are seven test questions. My answers and explanations follow.

1. A mother reads a scientific study that 17-year-old boys in the U.S. have fewer automobile accidents per 1,000 trips than 16-year-old boys. This prompts her to wait a year before letting her just 16-year-old son drive, to make him safer. Is her reasoning correct?

2. Earlier U.S. Census studies have found that the number of annual divorces filed equals about half the number of new marriages registered each year. These findings have been the basis for the commonly claimed fact that about half of all new marriages will end in divorce. Is this inference correct?

3. You play a game of tennis against a slightly better opponent and decide to put some money on the match. You are both equally fit in terms of stamina and physical endurance. Are your chances of winning the match the same whether you play just one set or the best out of three?

4. You attended a large conference where one of the European speakers had an accent that sounded more Swedish than German to you. But you were not quite sure, so you make a bet with your friend about whether the speaker is from Germany or Sweden. Assume the accent is all you have to go by since you don't recall the speaker's name nor any other biographical information. Which nationality do you bet on and why?

5. You are playing golf and reach an interesting par 3 hole where the tee is 10 feet higher than the green below, which happens to be completely flat. There is no wind, fog or rain. How should you play it: hit the shot as if the green is level with the tee area? Or try to identify all the factors that could affect the shot's eventual length due to the 10-foot difference in elevation. You may recall from physics that a ball or bullet will follow a parabolic trajectory if there is no air friction..

6. Several studies about the benefit of bicycle helmets have shown that fewer severe head injuries occur - for most kind of falls or spills - if you are wearing a helmet. Does it follow therefore that making helmet use mandatory will further reduce head injuries in cycling?

7. During the early part of World War II, many English bombers were being shot down by the Germans. To reduce their high casualty rate, the Royal Air Force (RAF) decided to reinforce its bombers with armor. But where? Since metal was scarce, they had to be smart about which surface areas to strengthen. Their statistical analysis of the bullet holes in planes that returned revealed a very uneven pattern of locations where planes had been hit. Should the RAF reinforce its bombers where bullet holes were most numerous, evenly throughout the plane, or elsewhere?

ANSWERS:

1.We don't know if the driving improvement is due to 17-year-olds being more mature or due to them having an extra year of driving experience under their belts. If the latter, waiting a year won't help. Probably both effects operate, so waiting may be wise.

Lesson 1: Always consider multiple hypotheses when explaining a fact.

2. In a stable population with zero growth, the cross-sectional statistic on divorces filed in any one year may be a good estimate of the chance that any random marriage will end in divorce. But, if the population is growing, or if values and marital expectations are changing, and if the ages or income levels at which people marry shift, then the cross-sectional data may be misleading about the longitudinal pattern of divorce.

Lesson 2: It is easy to lie with statistics; are you comparing apples and oranges?

3.The longer you play, the more the final result will conform to the law of averages. Since this favors the better tennis player, your chances decline if you play three sets. If you were to play Roger Federer, say, your best chance to beat him is to play just one point and hope he misses. And even that is a long shot; so just forget about playing any further than one point.

Lesson 3: Lay your bets according to the underlying statistics; consider base rates. The race may not always go to strongest or fastest, but that is nonetheless the way to bet.

4. Since Germany is more than eight times larger than Sweden (84 million people vs 10 million), you should bet on Germany unless you are very sure about your impressions about the accent you heard or have good reasons why German speakers would be less common.

Lesson 4: Understand the Law of Averages and be sensitive to sample size: as the old saying goes, one swallow does not a Spring make.

5. First, you should hit a golf shot that does not go as far as normal since the ball will travel longer in the air and thus farther. But other, mostly secondary, factors may also come into play due to the effect of the altitude difference. For example, the ball will hit the green at greater speed and thus will bounce up higher (especially if the green is cold and hard). Also, since the angle of impact on the green will be steeper, the overall length of subsequent bounces may be less. Finally, any back-spin your swing may have produced will reduce how far the ball travels through the air. Even though most of the factors mentioned suggest that you should aim your ball less far horizontally, your optimal aim will depend on the size of these other factors, some of which are correlated.

Lesson 5: Multiple effects usually operate in complex cases; try to consider all.

6. Over the decades, helmet use has increased in the U.S. but head injuries did not decilne as much as expected. Possible explanations include: improved bikes make riders go faster; wearing a helmet creates a false sense of security and more risk-taking on the road riders wear their helmets incorrectly (too much forehead exposed); the type of people riding bikes may have shifted; traffic and road conditions may have changed. Experts especially blame the false sense of security that helmets give to bikers and car drivers. Behavioral changes often lessen the benefits of safety actions, as has been seen with seat-belts laws, insurance, diet advice, anti-smoking remedies and Covid-19 vaccinations. Humans may suffer from unconscious risk-homeostasis, as captured in the adage that the sum of sins tends to remain constant.

Lesson 6: Look for unintended consequences that may undermine your initial goal.

7. The initial thinking was to reinforce the areas with the most bullet holes. But when the RAF asked statistician Abraham Wald to examine the surviving bombers for damage patterns, he came to a different conclusion. Wald reasoned that the surviving planes had not been damaged fatally by the random bullets and thus suggested reinforcing in places showing the fewest bullet holes. These were the most vulnerable, he argued, since few bombers apparently survived those bullet shots. The RAF followed his counter-intuitive advice and improved the survival rate of its bombers and crews.

Lesson 7: Ask what data you aren't seeing and why; there could be a selection bias, in this case known as a survivorship bias.

Inc