By Jack Hart
The Oregonian
Figures don't lie, goes the adage, but liars can figure.
Which is precisely why we need to be a lot more skeptical of
the numbers we publish.... Especially numbers that supposedly measure things
such as nutrition, environmental hazards and health risks.
One gripe about modern newspapers is that they carry an unrelenting
onslaught of threatening -- and often-contradictory -- information. This
study says salt causes high blood pressure ... that study says it doesn't.
This scientist says an earthquake will strike ... that scientist says the
prediction is nonsense. This nutritionist says you should avoid butter
... that one says you shouldn't bother. And so on, a daily deluge of threats
that leave readers feeling badgered, bullied and bewildered.
Who can blame them for throwing down the paper in disgust?
But there are ways to sort through all those claims. Victor Cohn's
"News and Numbers" is a guide to dealing with the conclusions reached by
all the scientists, doctors and demographers who are out there counting,
weighing and timing everything in sight. All of them have vested interests
in proving their points. Sometimes they're liars who figure. Sometimes
they're just humanitarians blinded by their own beliefs. Once in a while,
they're actually right.
We have "News and Numbers," in our Writer's Library. And we also
have "Reporting on Risk," another Victor Cohn book that suggests ways for
getting past the numbers to find the truth. Cohn, now a senior writer at
the Washington Post, was the Post's science editor. So he brings a journalist's
perspective and skepticism to the subject. One or both of his books are
essential reading for every one of us who deals with science, medicine,
nutrition or any other field heavily influenced by scientific research.
Copy editors who deal with wire stories on research results should
be especially wary. We pass along too much dubious information that serves
only to frighten readers despite sketchy or nonexistent evidence.
* * *
The point is well illustrated by the catchy story we put on A1
a couple of months ago, the one that claimed left-handed Americans lived,
on average, nine years less than right-handers. The study no doubt got
'em talking, and it's great when we can get something that lively and provocative
on the front page. But the L.A. Times-Washington Post version we published
on March 4 gave little basis for deciding whether or not there was any
truth to this startling -- and frightening -- claim.
The story was based on a study scheduled for publication in the
New England Journal of Medicine, which gave it a certain amount of credibility.
And it did hedge the basic claim, quoting several scientists and doctors
who warned of possible sampling error and inherent improbability.
What it didn't do was give the basic facts Cohn says should accompany
any such story. Unlike many stories on scientific studies, it did describe
the sample -- death certificates for 1,000 recently deceased folks from
the San Bernardino area. It didn't say how many of them were left-handed,
although the story did reveal that about 10 percent of the overall population
is left-handed.
If we project from the national figures to this sample, we can
conclude that we gave front-page play to a startling claim based on figures
for about 100 lefties in a single American city. Generalizing to the whole
American population of left-handers -- something like 25 million people
-- must involve a huge amount of risk. Normally, you would need a sample
of about 3,000 before you could safely generalize to such a huge universe.
Unfortunately, we didn't report the amount of risk involved in
this particular generalization. So we left out a statistic that Cohn calls
for in every story on based on sampling.
We express statistical risks in two basic figures: margin of
error and confidence level. The margin of error is the amount -- plus or
minus -- that the sample figure is likely to vary from the actual figure
for the whole population. And the confidence level is the likelihood that
it varies by that much. The usual confidence level for reputable scientific
research is 95 percent. That is, the scientist is 95 percent sure that
his numbers are within a certain amount of the numbers for the whole population,
sometimes known as the universe.
Here's our suggested form for reporting those two numbers in
conversational terms: "The chances are 95 out of 100 that the actual figures
are within X amount of the sample figures."
In this case the sample measured life expectancy. So the margin
of error would have been expressed in months or years.
* *
*
We published a similar A1 story last December 13. The story also
was from the Times-Post service and also reported on the results of a study
due to be published in the New England Journal of Medicine. It, too, made
a startling claim: For the first time, according to the lead researcher,
scientific evidence linked colon cancer to a diet heavy in animal fats.
The basic claim was that women who ate beef, pork or lamb every
day as a main dish had a risk of developing colon cancer 2.5 times higher
than women who consumed red meat less than once a month. On the surface,
the evidence seemed good. The scientists based their study on what appeared
to be a huge sample -- nearly 90,000 women -- and they studied the women
in the sample over a six-year period, which meant they relied less on the
respondents' memories.
But our report still lacked certain critical information. Once
again, it lacked margins of error and confidence levels. Even more importantly,
it lacked the crucial base figure. It told us that women who ate red meat
regularly ran 2.5 times the risk of women who hardly ever at red meat.
But what was the basic risk? If it was tiny, then a risk 2.5 times as great
was still tiny. But we nonetheless gave front-page play to the story, suggesting
the risk was significant. Our story may, in fact, have persuaded large
numbers of readers to change their eating habits significantly. And for
what?
By the way, that's a flaw in a high percentage of our medical
stories. We say that a such and such a behavior involves X times the risk
of another behavior. But we seldom say what the risk actually is. And that
number is critical to readers who want to make up their own minds about
the risks they're running. Smoking and cancer stories are notorious for
just this kind of loose end.
A number buried deep in this particular story suggests that it
hardly deserved the credence we gave it. During the entire six years of
the study, only 150 cases of colon cancer turned up among the 120,000 women
in the sample group. (The story never explained the difference between
that figure and the figure given for the actual sample size.)
Presumably, only some of those 150 cases qualified as women who
ate red meat every day. And only some qualified as women who ate red meat
less than once a month. Of 150 women picked at random, how many would eat
red meat less than once a month? Five? Ten? Fifteen at the most?
Let's say 15. And let's say that 37 of the women -- 2.5 times
as many -- ate red meat every day. So do some simple math. If you eat red
meat every day, your chances of developing colon cancer over a six-year
period are 37 out of 120,000 -- something like 1 in 3,500. And your chances
if you eat virtually no red meat? One in 8,000.
Of course, it isn't quite that simple. And we're working on the
basis of multiple assumptions forced by the sketchy information the story
provided. Furthermore, your risk over the vulnerable part of your adult
lifetime will be higher than your risk over six years. And how much faith
can we place in odds that apparently were calculated on the basis of 15
cases out of 120,000 women?
But put all that aside for a moment. Accept the odds we've calculated.
Do they sound like the kind of risk that would persuade you to have no
more than one steak a month for the rest of your life?
(Hart is a managing editor at The Oregonian.)
----------------------------------------------------------------------------
Chances Are....
Some added cautions concerning risk....
þ Never report relative risk by itself. Telling readers
that smoking triples their chances of lung
cancer
tells them almost nothing.
þ Make sure that every story on risk contains a figure
revealing the actual risk that something will
happen
to readers. Tell them, for example, that the
actual
chance that an adult male in Portland will be murdered
in any one year is one in 50,000. Or that the chance
a home in Southeast Portland will be burglarized in any
one year is one in 350.
þ Put actual risk in context. Compare it to something
that readers will recognize. The old chestnut
about
being hit by lightning is tired, but effective.
If you
want to cite an even more remote risk, note
that one
American was actually hit by a meteor, which makes the
odds something like one in 700 million. Higher
risks
can be calculated from local crime or medical
statistics.
----------------------------------------------------------------------------
A Healthy Skepticism
By Jack Hart
"I told them how I saw a sea of a million black men, and yet just think, 10 to 20 times that many died on the slave ships to America." (10/22/95, D1)
The claim that as many as 20 million slaves died on the infamous
Middle Passage has been in the air recently. It was, unfortunately, only
a matter of time before it showed up in The Oregonian.
It belongs to the same class of unquestioned claim as the recent
assertion that wife beatings went way up on Super Bowl Sunday. It's a rhetorical
point that has innate appeal to those who would use it for rhetorical purposes.
But like the Super Bowl claim, it is pure fiction. We should
always be skeptical of such claims. And we should avoid passing them along
without correction, even in direct quotes.
The figure is immediately suspect because it is so large relative
to world populations during the 150 years when the slave trade thrived.
At the time of the Revolution the largest American city, Boston, had only
50,000 residents. In 1810, two years after the legal importation of slaves
into the United States ended, the total American population had reached
only 7.2 million. And only 1.2 million of those were slaves. So a figure
of 10-to-20 million deaths on the Middle Passage is way out of reasonable
proportion from the start.
In fact, the African American Almanac reports that the best estimates
of TOTAL slaves brought to all points in the Americas -- not just the United
States -- was 10 to 20 million. The best estimate of how many died aboard
the slave ships was 15 percent -- 2 to 3 million.
That's still a terrible number, of course. And this observation
is in no way intended to demean it. The point is that a newspaper should
get its numbers right. And it should never uncritically accept numbers
promoted by ANY interest group in the promotion of ANY cause.
----------------------------------------------------------------------------
News by the Numbers
By Jack Hart
In the grand scheme of things, most journalists rank numbers somewhere
below cockroaches. If the truth be told, a good number of us chose journalism
as a college major because it allowed us to avoid math courses.
Something about the inclination to write flatly contradicts
an interest in things quantitative. We're into words, not numbers.
And yet, as Victor Cohn points out in "News and Numbers,"
we're ultimately forced to deal with math anyway. "We journalists like
to think we deal mainly in facts and ideas," says the former Washington
Post science editor, "but much of what we report is based on numbers."
Anybody who thinks about it has to agree. Government budgets,
economic forecasts, environmental studies, unemployment figures, housing
starts -- the whole flow of daily news is shot full of numbers.
That gets more true every day. The march of science and the computerization
of government mean that counting things becomes more and more fundamental
to the process of daily life. Just compare the quantity of numbers generated
by the 1990 census compared to any census that's come before.
The problem with that, from our point of view, is that most of
us aren't terribly good with numbers. Our distaste for figuring led us
to avoid the study and practice that gives more mathematically oriented
folks such ease with numerical procedures. And maybe we just lack the talent.
Plenty of journalists received verbal SAT scores that were 30 or 40 percentile
points above their quantitative scores.
But that doesn't relieve us of any journalistic obligation. Numbers
are critical to public policy-making, and if we don't do a good job reporting
them, we abdicate a large part of our watchdog role.
Furthermore, many of our readers aren't any better with numbers
that we are. That sad fact reinforces our duty to do a solid job of sifting
through the numbers generated in the public arena, extracting the newsworthy
figures and presenting them in the clearest, simplest way possible. The
last thing we want to do is confuse readers with cloudy statistics and
quantities that don't allow for easy, number-by-number comparisons.
Besides, numbers bog down the text. Loading a story up with numbers
almost guarantees low readership. And so does a story that forces readers
to do their own math to figure out what we're saying. Few members of our
audience turn to The Oregonian because they're craving a good story problem.
Most of all, we want to get our numbers right. Far too often
our attempts to deal with numbers just don't add up.
That said, here are some general principles that should help
us do a better job of what most of us like doing least:
Do the Arithmetic
Rivera was 46 on July 4. A native of New York, he was graduated from Brooklyn Law School and practiced law, representing the poor, until 1960. (XX,XX)
Anything's possible with Geraldo, but it doesn't seem likely that was practicing law at the age of XX. The other problem with this passage, of course, is that the writer didn't tell us when the king of the trash journalists started practicing law. The word "until" suggested that we were reporting a range. But we should never report only one end of a range.
Visualize the Thing the Numbers Describe
The John Day drawspan bridge has a center section that pivots 180 degrees to allow boats to go through. (6/25/'89, B5)
Yeah? Think about it. It will take a damned fast boat to get past that bridge.
Use Comparable Forms
Three-quarters of college students reported monthly alcohol use in 1990, compared with four-fifths in 1980. (3/31/'91)
Quick! Which is bigger -- three-quarters or four-fifths? How much
bigger?
We should never force readers through such mental gymnastics.
Three-quarters is 75 percent. Four-fifths is 80 percent. Once we start
comparing apples with apples, the comparison starts to make some sense.
And it also reveals that college drinking habits didn't differ
all that much between 1980 and 1990. Depending on the sample size, in fact,
the apparent difference may have been a statistical artifact. But we didn't
report the sample size or the margin of error. So who knows?
You'll pay some fees -- about 5 percent of each $100 you put into a stock fund for instance.... (12/30/'90, E1)
Well ... better overkill than no kill at all. But the fact is
that a percentage is a way of simplifying comparisons by converting everything
to base 100. If we express something in terms of a percentage, there's
no need to break it down per 100 of the original quantity. It's simply
5 percent of everything you put into a stock fund.
The reverse is also true. If you express things in terms of 100,
you have no need for a percentage. It's $5 for each $100 you put into a
stock fund.
Make It Meaningful
Police and fire rescue crews diverted traffic around the roadblock, which was estimated at 1,000 cubic yards, or about the size of a house. (Rick Bella, 3/5/'91, B1)
Hooray! Here's a writer sympathetic enough to readers to understand
that most of them have absolutely no idea how much 1,000 cubic yards of
landslide might be. The simple comparison with a volume we all understand
made the size of the slide understandable.
Let's remember that most of us don't really understand how much
a billion is, or 5,000 acre-feet, or 20,000 board feet or any number of
other large and/or unusual quantities. Give readers a break by expressing
the unfamiliar in terms of the familiar.
Use Only What You Need
In all, conservation groups sought to halt 421 proposed national forest timber sales containing 2.5 billion board feet of timber. That's more than half the 4.8 billion board feet the U.S. Forest Service plans to sell on the 13 national forests in Oregon and Washington this year. (3/17/'89, D1)
The point of this passage was to indicate the scope of the environmentalist attack on logging. So what are the critical figures? To some degree, that's a judgment call. But you can make a good argument that all readers really needed to know was that the conservationists wanted to stop more than 400 sales containing more than half of all the timber the Forest Service planned to sell in Oregon and Washington. Adding all the other figures just detracts from the central point without contributing anything terribly meaningful.
Minimize Density
Stacey King scored 23 points and made a key three-point play with 1:01 left Wednesday night as top-ranked Oklahoma beat Kansas State 86-82, the Sooners' 28th consecutive victory. (2/23/'89, D8)
In Renton and Everett, Wash., 27 new Boeing Co. commercial airliners
roll out of mammoth buildings each month as the company tallies a 1,528-plane
order backlog. In Gresham, in a much lower profile building, about 1,940
Boeing Commercial Airplane Co. workers produce about 7,500 different parts
for those planes, each part destined for a particular plane scheduled to
roll off the line sometime in the following 18 months.
The Gresham workers are a small portion of the commercial airplane
unit's 60,000 employees and even a more minuscule percentage of the entire
Boeing work force of 157,000, of whom 101,000 work in the Seattle area.
The Seattle work force should grow slightly to 102,000 by mid-1990....
(5/7/89, D1)
A 30-year-old Molalla man has been sentenced to 20 years in prison with a 10-year minimum in the death of a 16-month-old boy who died of head injuries three days after the baby was in his care. (raw copy, 8/15/'90)
The $6.8 million corrections funding bill would pay for a half-dozen prison construction projects, including about $205,000 for planning a 500-bed medium-security prison that will be the first phase of a new prison complex that may eventually house as many as 3.000 inmates. (4/21/'89, D1)
As a general rule, more than three numbers in a paragraph turns
the copy into sludge. Usually, you won't have a problem if you use only
the numbers you absolutely need. The Boeing story is a case in point. Who
really needs all that detail about numbers of planes, parts and employees?
But the main point here is that when you must use more than just
a couple of numbers, you spread them out. Keep the density down to a number
or two per paragraph.
Note, too, that all but the last of the examples listed here
was a lead. Filling a lead with numbers is like nailing a locked gate on
the front of the story.
Simplify
... the Golden State will have 52 representatives in the House, seven more than in the 1980s. That's the largest delegation, and, at 12 percent, the largest share of House seats held by a single state. (12/27/'90, A1)
Numbers have one great virtue. They're almost infinitely malleable.
They can be collapsed, converted and simplified in a dozen different ways.
For many news developments, in fact, you only need one hard number. Everything
else can be expressed in terms of comparison with that base. In other words,
you express things in ordinal terms -- explaining that some things are
bigger or smaller than others without necessarily explaining exactly how
much bigger or smaller.
The story on California's congressional delegation, for example,
built everything around the base number 52. It didn't include the number
of California representatives in the 1980s. And it didn't include the total
number of House members. Instead, it expressed things in relative terms.
The writer did the work for us, and we only had to read the numbers that
counted.
Avoid Excessive Detail
In 1984-85 the first full year of programs, the district spent $112,000. The budget for this year is $326,109 -- $6.15 for each of the 52,996 district students. That's .11 percent of the district's total $310 million general fund budget.... (9/18/'89, A10)
... the "dent" in the sun will be larger in the Northwest, where 54 percent of the sun's diameter will be blocked by the moon in Eugene, 55 percent in Portland and 56 percent in Seattle. (3/2/'89, E1)
The fairview Budget Committee Tentatively approved a $1,730,592 budget Monday..., one that is 1.63 percent larger than last year's $1,702,716 budget. (4/11/'89, B2 [East])
We're often excessively precise. Does a dollar or two -- or $2,000
-- really make that much difference in a million-dollar budget? Wouldn't
we be better off giving round numbers that readers can understand instead
of burying them in an avalanche of detail?
Instead of the smothering detail in the first example, for instance,
we could have said that the district's $325,000 budget is three times larger
than in was five years ago, but that it's still only a little more than
$6 for each of the district's nearly 53,000 students and less than a thousandth
of the district's general-fund total.
And what's a percentage point when it comes to the sun's surface?
Suffice it to say that, when viewed from the Northwest, the moon will block
more than half the sun.
And that Fairview approved a $1.7 million budget Monday, up less
than 2 percent from last year's.
Be Fair
In January Joe Uris, a longtime Portland activist, contributed
a Forum piece that analyzed the way the Portland School District cooked
the charts to make improving black student scores on achievement tests
more dramatic than they really were. It was, said Uris, "a classic example
of distortion through the use of a statistical model."
Uris was absolutely right. What the district did -- and what
we passed along to our readers -- was to create a graph that ran from 30
percent to 80 percent on the vertical axis instead of 0 percent to 100
percent. That made the line depicting the increase in scores over time
seem steeper.
Furthermore, the district graph distorted the time periods depicted
by the horizontal axis. It collapsed a seven-year period of gradual improvement
in black scores and made it approximately equal to a two-year period of
minimal improvement. That increased the slope of the already distorted
improvement curve even more.
The result of all this? We passed along school district propaganda,
even though we had the figures we needed to create a more accurate graph.
We're used to the way politicians and bureaucrats hedge their
words to make themselves look good. Because we're on guard and savvy to
the politics of the language, we usually protect readers from the most
self-serving versions of the truth.
But we're much less critical when it comes to quantified information.
In our hearts we know that XX. But when it comes to actually doing the
math, we're often at less than our best. Still, our fundamental obligation
is to bring the same skepticism to numbers that we do to all the news,
especially now that so much of the news is numbers.
-------------------------------------------------------------------------
Last month. Prof. Emerson Hoogstraat shared a little math lesson
for newsies. Unfortunately, his treatise barely tapped the subject's surface.
As other readers repeatedly point out, our transgressions go way beyond
the few examples he so diplomatically cited. Among them:
þ Numbers that just don't add up:
1) Deck: The Pentagon proposal would cut nearly 200 ships....
Lead: The Navy has proposed slashing its current 452-vessel
fleet to 340 ships....
(5/17/93, A9)
Take 340 away from 452 and you get 112, which is a long way from "nearly 200." Besides, a vessel can include anything from a captain's gig to a battleship. Theoretically, you could slash a 452-vessel fleet to 340 ships without eliminating a single vessel that actually qualified as a ship.
2) The Oregon Comprehensive Firearms Act went into effect in 1990.... In 1989 there were 11,743 concealed weapons permits in Oregon. In the first year of the law, the number of permits went from a few hundred to 13,240. Through December 1993, the number had shot up to 40,040. During January 1994, 1,192 more were issued. (2/27/94, A16)
Hmmmmm.... 11,743 in '89. A few hundred at the beginning of '90. More than 13,000 by the end of '90. A grand total for the last month of '93, but a monthly total for the first month of '94. What is going on here?
3) Last week, Tri-Met's board of directors adopted a plan that calls on the agency to add 125,000 new riders a day by 1998. It would be a big jump. (4/9/93, C2)
It certainly would be. As William Burr, one of our more faithful correspondents, points out, 125,000 riders a day from the date of publication until the beginning of 1998 totals 216,375,000 new riders.
4) Oregon farmers plant about 25,000 acres of Blue Lake bush beans each year, harvesting 2.5 times the yield of Wisconsin, which leads the nation in bean production. (8/29/94, B6)
Well, maybe the writer meant that Wisconsin was harvesting lots of other kinds of beans. But she still managed to confuse at least one readers, who wrote to find out what was going on.
5) Rail statistic: "Each year, 3 million to 4 million American women are battered by their husbands or boyfriends, making battery the most significant cause of injury. -- U.S. Surgeon General.
Text statistic: According to the National Coalition Against Domestic Violence ... more than half of women are battered some time in their lives; more than one-third are battered repeatedly every year.... (1/31/93, L1)
So which should we believe. On first thought, the government source seems likely to be more responsible than the advocate. A second thought should clinch the decision: If what the NCADV says is true, something like 100 million American women are battered every year. Let's hope they all plan to ride Tri-Met.
6) According to the Job Corps, which tracks former students for six months, 67 percent of the graduates get jobs and 17 percent go on to higher education. Those percentages include the nearly four in 10 students who quit the program in the first three months or are kicked out for infractions like using drugs or carrying a weapon. (2/2/1/93, A4)
Not only do these numbers not add up, they don't even make sense. How can figures for graduates include figures for students kicked out in the first three months?
þ Mangled survey terms or results:
1) ... the study's results are accurate within 3 percentage points. (8/24/93, A11)
Nobody can make such a claim. The researchers probably reported that the study's margin of error was 3 percentage points. That means there is a known likelihood (usually 95 out of 100) that the true results are with 3 percentage points -- plus or minus -- of the results actually obtained.
2) However, Wolraich said these three differences, while statistically significant, could still have occurred as a matter of chance. (2/3/94 [1st edition], A1)
Of course they could have occurred by chance! Any result based on sampling can occur by chance. At the 95 percent level of confidence, 5 out of every 100 findings will occur by chance.
3) Random samplings of audiences at a dozen or so of his speeches in four states over the past two months indicates that most are made up of longtime supporters. (3/16/93, A3)
A "random sampling" is a strictly scientific sample. Survey researchers
spend huge amounts of money trying to approximate random samplings.
The writer who produced this example meant what the researchers
call an "accidental sample." The most graceful way to say that for a general
audience "a casual sample," as in "a casual sampling of audiences."
þ Mislabeled statistics:
1) To evaluate this level of debt, consider that for a corporation, a debt-to-equity ratio of 40 percent is considered pretty good. (2/22/94. B5)
A ratio and a percentage are not the same thing. Percentages are
all built from a base of 100. Ratios are simply comparisons of two numbers.
"The ratio of potatoes to onions was 2-1."
In this case, the writer presumably meant debt equal to 40 percent
of total equity. In that case, the equity-debt ratio would be 60-40,
or 3-2.
þ Numbers without context:
1) It appears to me, assuming that reporting patterns have stayed the same ... certainly we are more safe today than we have been in the past in Oregon," said Willhite, 47, who has been forecasting state prison-bed needs or analyzing crime data since 1987. "So we are doing a better job somehow." (3/23/94, B1)
Oregon's crime rate may be down, but the conclusion that the authorities
are therefore doing a better job is nonsense. For one thing, young men
commit the vast majority of crime. So the most important statistic when
it comes to the level of crime is the percent of the population that consists
of men from 18 to 35. Crime rates go down when an especially large
cohort of men passes through that age range and into their mellower middle
years. That's what's happening now that the last of the Baby Boomers are
into their 30s. So the crime rate could well be falling even if the authorities
are doing a much worse job.
Besides, as Willhite himself pointed out in the same story, the
Oregon's population growth includes an influx of well-educated professionals.
They don't commit much crime (at least the kind that shows up in FBI statistics),
and that factor alone could reduce the crime rate.
þ Incomplete statistics:
1) We ran a cancer-scare story from the New York Times with this lead: Women who do not smoke but are married to men who do have a small but increased risk of developing lung cancer, a study has confirmed. (6/8/94, A16)
We could have chosen from four additional wire stories on the same subject. Several key numbers failed to match:
AP: The study found that the risk of developing lung cancer for the women with spouses who smoke was about 30 percent higher over a lifetime than for those with nonsmoking spouses."
L.A. Times: "In the largest and most comprehensive study of its kind, researchers in four states reported Tuesday that long-term exposure to secondhand cigarette smoke can increase the risk of lung cancer in non-smoking women by as much as 86 percent....
Newsday: The EPA report consolidating data from 30 studies worldwide and concluded that a woman who lives with a smoker is 1.19 times as likely to develop lung cancer as a woman living with a non-smoker. In the Fontham-led study the comparable figure is 1.29.
The confusion over the numbers is interesting enough. But what's
even more flabbergasting is the fact that the key number is totally missing
from all the stories.
If you were a women who lives with a smoker, the only number
relevant to whether you leave or stay is your ACTUAL chance of developing
lung cancer, not your chance relative to someone living with a non-smoker.
If the chance is one in a thousand, what difference does a difference of
19 percent, 29 percent, 30 percent or 86 percent actually make. At worst,
hanging around the same old house increases your odds of getting lung cancer
to something less than two in a thousand.
We often leave the critical number out of disease-risk stories.
What readers need to know is not their relative risk, but their actual
risk. Let's give it to them.
2) Sales average $1,113 per machine -- and that's a lot of tickets. (9/4/91, D3)
Sales average $1,113 over what period of time? The number is nearly meaningless without that context.
þ Conclusions that go way beyond the numbers:
1) Headline: At lunchtime, it never fails: Men network, women
run errands.
Text: Researchers at the University
of California,
Irvine, found that 51 percent of women said they
shopped or ran errands at lunch. Only 39 percent
of men claimed to do the same. (8/28/94, L5)
Nothing in the text supports the conclusion in the headline. Nobody asked men what they were doing instead of errands. The percentage of women who report running errands hardly justifies the claim that "it never fails."
þ Numbers that don't match:
1) Although estimates vary, taxol costs about $22,000 an ounce. It takes 9,000 pounds of dry bark to produce a pound of taxol. (10/21/91, B1)
We regularly ask our readers to do the math instead of doing it
ourselves. When we cite numbers, let's make them easily comparable. If
taxol is $22,000 an ounce, we don't care how much bark it takes to make
a pound. We care how much it takes to make an ounce.
There are 16 ounces in a pound. Ergo, it takes a little over
560 pounds of bark to make an ounce. Or, to run at it from a different
direction, 9,000 pounds of bark will produce a pound of taxol worth $352,000.
2) Each gallon of the chemical produces 5.4 kilograms, or 12 pounds, of heroin, Van Horn said. The drug is commonly sold wholesale by the "piece," which is 25 grams and typically sells for about $2,500. Each gram of heroin makes 20 user doses that sell on the street for $20 each, Van Horn said. At $20 per dose, a "piece" of heroin would make 500 users doses and sell for a total of $10,000 retail, and a kilogram would be worth $400,000. (6/27/91, B8)
So how much is a pound worth?
-----------------------------------------------------------------------------
The Numbers Crutch
By Jack Hart
These days lots of our sources grind out prodigious quantities
of numbers. They apply dozens of different measures to their activities,
and then fill news releases with the endless columns of figures that result.
But the numbers themselves seldom mean much. Interpreting the numbers to
find the news is our responsibility.
Simply passing all those numbers along to readers is no better
than uncritically passing along any kind of press-release information.
Such practices undercut our standing as tough-minded, independent sources
of information.
They also damage our readability. Nothing kills reader interest
quicker than an apparently endless stream of numbers. Our job is to intercept
the deluge of statistics spewing out of publicists, trade organizations
and governments. We should screen those figures, sifting out the dross
and reducing what remains to the most meaningful, newsworthy numbers.
We failed to do that in the following examples:
The number of new single-family building permits slipped slightly
in * January, to 117, with a total valuation of $10.7 million, compared
with 170 permits with a valuation of $15.5 million issued in January 1990.
For the year, the county issued a total of 2,480 single-family
permits, compared with 1,812 in 1989. Multi-family permits showed a much
more modest increase -- 1,062 compared with 1,036 the previous year.
Residential sales, though, jumped 21.3 percent, from 4,683 in
1989, to 5,679 last year, according to Benchmarks, a Vancouver appraisal
service that tallies real estate transactions.
The total volume of those sales showed an even greater gain,
from nearly $374.6 million in 1989 to more than $531.8 million last year
-- an increase of almost 42 percent.
The Multiple Listing Service of Clark County reported it had
a total of 2,467 active listings as of Feb. 15, of which 595 were existing
homes and 386 were new houses. Alicia Leeb, systems coordinator for the
service, said scores of new listings were coming in weekly as interest
rates have come down.
The listing service showed 586 homes had been sold during the
preceeding six months -- an average of more than three a day. The average
selling price was $86,354, and the average time on the market was 60 days.
(3/3/'91, H1)
* *
*
The U.S. Agriculture Department estimates farmers nationwide will
ship more than 35 million trees this season, up from the previous high
of 34.3 million trees sold in 1988. The volume has increased by as much
as 2.5 percent annually for the past 15 years, at least twice the growth
rate of the U.S. population.
In Oregon, the No. 1 producing state, thousands of workers last
week began to cut, pack and ship a crop estimated at 7.5 million trees,
up nearly 5 percent from 7.2 million in 1988, said Bryan Ostlund, executive
secretary of * the Northwest Christmas Tree Association.
Michigan, No. 2 in volume, will harvest 6.5 million; Wisconsin,
4.8 million; and Washington, 3.5 million. The biggest market for Oregon
trees is California.
Prices haven't changed much since last year.
Farmers are receiving $7 to $11 from wholesale buyers for each
top-grade Douglas fir, 6 to 7 feet tall, compared with $8 to $11 last year,
Ostlund reported. The price varies by quantity and quality, he explained,
and some smaller growers lacking a strong relationship with a buyer are
getting $1 less this season.
The same tree will retail for an average of $15 to $21, he said,
compared with $16 to $21 in 1988. The tree may sell for $12 to $18 at a
cut-it-yourself farm.
Wholesale buyers are paying $13 to $19 for the same size and
grade of noble fir. Consumers will pay $24 to $35 at a retail lot, or $17
to $27 at a you-cut farm, also little or no change from 1988. (11/19/'89,
F1)
* * *
In 1986, for example, Americans built an estimated 19,000 log
houses worth* in excess of $2 billion. More than 84 percent of them were
earmarked as primary residences, according to Log Homes magazine's 1988
log-homes buyers guide published in December.
The popularity of log homes was revived in the 1970s, spurred
first by a renewed awareness of the environment by those who were seeking
an alternative lifestyle and later fueled by a growing acceptance of factory-built
log-home kits for do-it-yourselfers.
Today, the log-home industry is booming, with an estimated 350
log-home manufacturing companies and custom builders carving out a significant
niche in the market. Of the 371,000 custom-built, single-family homes that
the government says were built in 1986 in the United States, more than
5 percent of them were log homes, according to industry figures.
The average size of a log home built in 1986 in the United States
was 1,873 square feet, with 44 percent of the owners surveyed by Log Homes
magazine reporting that their homes cost more than $70,000 to complete,
excluding land.
In terms of log-home production in 1986, Oregon ranked 12th,
with 584 units built for a 3.03 percent share of the overall log-homes
market. However, only 317 of those homes remained in the state, which ranked
Oregon 25th in log-home construction.
By comparison, the state of Washington ranked fifth in log-home
production, with a total of 1,018 units produced in 1986, and was 16th
in log-home construction, with 447 units built in that state in '86. (6/5/'88,
H1)
--------------------------------------------------------------
The Perils of Polling
By Jack Hart
During the past six months, 1,700 of the stories we published
in The Oregonian's news sections contained some reference to polls or surveys.
Surprised? Don't be. We're more typical than not. When Arnold
Ismach, dean of the University of Oregon journalism school, asked students
to check five days' worth of news stories in 14 papers, they discovered
that nearly 1 in 5 had some quantitative research element. Usually, that
means a poll or a survey.
Survey research is a big deal these days. It influences elections,
helps determine government policy and shapes our view of the world. The
volume just keeps growing. More and more, survey research makes news, and
not just on regular news beats. Surveys play an essential part in arts
and music criticism, lifestyle coverage, medical writing and business reporting.
Now that we're headed into the heat of the election season, we'll
be faced with an especially intense onslaught of polls. It follows that
we should be as savvy and skeptical of survey research as we are of more
traditional news sources.
We have good reason to be. Even respectable pollsters are often
way off the mark. Early polling on Washington state's initiative to limit
terms of political office showed 70 percent support. It failed. Missouri
surveys on an initiative to increase school taxes showed 2-1 support. It
failed, too. And recent polls on political races involving Jesse Helms,
Texas Gov. Ann Richards and David Duke were off by large margins, as well.
We'd do a better job if we knew more about polling and included
more safeguards in our reporting.
The AP Stylebook lists several suggestions for handling survey
research, and anybody who hasn't checked it lately should consult the "Polls
and Surveys" entry before writing or editing a poll story. In addition,
you should pay special attention to these key questions:
þ Is it really a survey? We're not terribly careful when it comes
to how we use the terms "poll" and "survey." Those two words are, for the
most part, interchangeable, although "polls" usually appear in the context
of a political campaign. The important point is that both involve scientific
sampling as a way of estimating figures for a larger population. Man-on-the-street
interviews are neither polls nor surveys. Neither are call-in votes. Remember,
too, that haphazard or self-selected samples like those are not random
samples, which are scientifically selected so that every respondent has
a known probability of being included.
Note that a huge self-selected sample is usually less accurate
than a much smaller probability sample. The infamous 1936 Literary Digest
poll, which picked Alf Landon for president, drew a sample of more than
2 million.
That doesn't mean that we shouldn't do man-on-the-street interviews,
call-in programs or our own non-scientific studies. After the Nathan Thomas
shooting, we contacted 12 police departments to check their records on
use of deadly force. That wasn't a scientific survey. But it sure was good
reporting.
The important thing is that we don't represent non-scientific
samples as anything more than what they actually are. So we shouldn't ever
refer to one of our phone-in reader-participation efforts as a poll or
a survey.
And don't forget that counting everybody, as the State of Oregon
did when it checked to find out how many homeless people were in shelters
on a single night, isn't a poll or a survey either. It's a census. We overlooked
that fact we when we headlined the story, "One-night survey finds
3,200 people homeless." (12/18/91, C1)
þ Who sponsored the poll? One of the stylebook's most important
points is that we should always identify the poll's sponsor, which bears
directly its credibility.
An AP report we published this fall (11/15/'91, A15) is a case
in point. The story's main conclusion? "Americans who think they'll be
sitting pretty when it comes to covering health costs during their retirement
may instead have to turn quickly to welfare." The source? A study conducted
by Northwestern National Life Insurance Co., which had an obvious self-interest
in the subject.
On its face, the "study" is nonsense. The only survey-research
result reported was that of workers over 40, "most believe they will be
well-prepared for retirement." The insurance company also reported that
only a minority of private companies provide after-retirement medical insurance.
Then they proceeded to wave the bogeyman of welfare around while they conveniently
ignored the fact that virtually all retirees are covered by Medicare, Medicade
and, often, some form of personal health insurance. We helped them out
by passing along a one-source story rife with conflict of interest.
If some local insurance salesman had walked into the office with
a similar claim, we'd have laughed him right out into the street.
At least we named the sponsor, even though we failed to highlight
its direct self-interest in the study's outcome. All too often, we allow
sources to toss around completely anonymous poll results.
þ Do the data justify the conclusions? Not only do we often
swallow anonymous polls or dubious surveys from self-interested sponsors,
but we also have a weakness for inflated conclusions that go way beyond
a study's hard data. When we reported that a UC San Diego study of 242
upper-middle-class women showed that those with jobs had lower cholesterol
and blood-sugar levels, our headline concluded that Work is healthy. (1/30/92,
D1)
Tell that to a coal miner.
þ Is our math OK? We also have a hard time reporting numbers accurately. For example, we frequently confuse percentages and percentage points. We reported that the number of full professors in the Oregon State System of Higher Education had increased 5 percent over 10 years. (1/26/92, B4) In fact, the number had increased from 6 percent of the total to 11 percent, an increase of 5 percentage points. Assuming that total number of full professors was fairly stable, that's an increase of nearly 100 percent.
þ Is our report balanced? When we do report numbers accurately, we sometimes fail to keep them in perspective. The rate of pregnant Oregon women carrying the AIDS virus doubled from 1989 to 1991, and our story's headline and lead focused on that fact. But the total number of infected mothers-to-be was 34, a tiny, tiny percentage of the total number of both pregnant women and infected Oregonians. Each case is a tragedy. But was this angle the news in this story?
þ What were the actual questions? We almost always leave out the
exact questions used in a poll or survey, even though the national association
of professional pollsters asks that they always be included. (See sidebar
on page XX.) Sometimes question wording is critical. Lyndon Johnson flim-flammed
the press in 1968 when he released a poll that showed him leading four
Republicans in New York state. What most news stories failed to note was
that the poll questions didn't mention Nelson Rockefeller, the state's
favorite son.
Differences in question wording also explain why the president
usually gets a better job-performance rating from Gallup than from Harris.
Gallup asks respondents whether they approve or disapprove of the job the
president is doing. Harris asks respondents to choose a rating from among
four choices ("excellent," "pretty good," "only fair," "poor"). That spreads
out the responses and puts fewer into the positive categories.
Inept or malicious question wording can influence results, too.
Watch out for double-barreled questions, imprecise questions and loaded
questions.
þ What's the margin of error? Probably the best-known source of
error in virtually all polls is simple chance -- the likelihood that the
poll sample doesn't accurately reflect the entire population. We express
that likelihood as the level of confidence and the margin of error. Usually
pollsters reveal that there's a 95 percent chance (the confidence level)
that the poll results are within plus or minus X percentage points (the
margin of error) of the actual values for the whole population. (For a
fuller explanation of the margin of error, see the sidebar on page XX.)
Both the stylebook and the professional pollsters say we should include
both figures in every poll or survey story.
But we often leave them out, even when they're essential to interpreting
the results. For example, when we reported the results of a New York Times/CBS
News poll on the Clarence Thomas nomination, we said that 22 percent of
the men and 18 percent of the women polled were against the nomination.
But we reported no margin of error. Chances are that it was at least 3
percent. That means the figure for the men could have been as low as 19
percent, and the figure for women could have been as high as 21 percent.
So it's well within the realm of possibility that, in fact, more women
than men were against the nomination.
þ When was the poll taken? Public opinion is volatile, and it
often keeps changing right up to election day. That's why it looked as
though David Duke would do much better than he actually did.
Early polls are especially likely to mislead on initiative campaigns
because supporters generate most of the early publicity. Once the opposition
gets organized, things can change in a hurry. Which is why the early reports
on public opinion in the Washington state term limits initiative were so
far off the mark....
þ What was the response rate? Lots of commercial mail surveys
draw an extremely low percentage of returns. And the lower the rate of
return, the more likely that the responses represent some systematic error.
Maybe most of the replies came from those with the most extreme opinions.
Or from the richest respondents. Or from Republicans.
In general, be suspicious of response rates below 70 or 80 percent.
Lots of other variables affect survey results, and no news story
will mention them all. We don't want to bore readers by mentioning every
little thing that went into a poll. But we should nonetheless ask plenty
of skeptical questions before we decide that a poll is worth reporting.
Remember, for example, that respondents sometimes lie when they
have good reason to do so. Only 19 percent of Louisiana's voters admitted
that they cast ballots for David Duke in last fall's primary. But Duke
got 32 percent of the vote.
The order of the questions matters, too. One survey asked respondents
to agree or disagree with the statement that "it's all right for Japan
to impose import quotas." The rate of positive responses was far higher
when the statement followed on that said "it's O.K. for the United States
to impose import quotas."
Intensity matters as well. If a small group believes something
passionately, that can have more impact than if a lot of people feel lukewarm
about something. Jesse Jackson almost always gets more votes than
the polls predict because his supporters are the most likely to vote.
Lots of other things go into the mix. The kind of interviewing
-- phone, mail or in-person -- may have an impact. And a skeptical reporter
will always ask whether the folks who administered the questionnaires were
audited to make sure they didn't make up answers or bias results. We should
also always ask whether a survey turned up anything that the pollster isn't
reporting. Even accurate results can be misleading if they only reveal
part of the picture.
And, lastly, we should remember that survey research isn't the
same thing as common sense.
Philip Meyer, an early proponent of using survey research methods
in journalism, points out that no computer ever managed a political campaign.
And he cites an old story about Harvard professor Edward Banfield
and his comments to the students in his voting-behavior seminar. No matter
what computer tables told them about the peculiarities of Chicago voters,
Banfield told his students, they still would know less than Mayor Daley.
……………………………………………………………………………………………………
Full Disclosure on Polls
By Jack Hart
The National Committee on Published Polls has compiled this checklist, which specifies the minimum amount of information that a newspaper report on a poll or survey should contain:
1. The identity of the survey's sponsor.
2. The exact wording of the questions asked.
3. A definition of the population sampled.
4. The sample size and the response rate.
5. The margin of error.
6. Which results are based on only part of the sample, such as probable
voters, those who have heard of the candidate and so on.
7. How the interviews were conducted -- in person at home, by phone,
by mail, on street corners or whatever.
8. When the interviews were conducted.
Obviously, we should use our own judgment. We wouldn't, for example,
use all that information in a passing reference to some poll result. Nor
would we include the exact wording of every question on a two-hour questionnaire.
But we should be aware of why the N.C.P.P. thinks each of the eight items
is important, and we should use good news judgment to include those that
are most relevant to individual news stories.
Note, too, that responsible public opinion pollsters are obligated
to provide this information and to blow the whistle on politicians or others
who warp their poll results. The American Associate for Public Opinion
Research holds members to this vow:
We shall maintain the right to approve release of our findings, whether or not ascribed to us. When misinterpretation appears, we shall publicly disclose what is required to correct it, notwithstanding our obligation for client confidentiality in all other respects.
For further details on what we should customarily include in stories
on survey research, see the "Polls and Surveys" entry in the AP Stylebook.
……………………………………………………………………………………………………
How Polls Work
By Jack Hart
Even politically sophisticated journalists sometimes have a hard
time understanding how a couple of thousand individuals can accurately
represent the entire U.S. population. But they can, so long as the pollster
draws a good sample. The experts have proved their ability to predict voting
patterns, within a couple of percentage points, time and time again.
George Gallup likes to explain it with his soup analogy. One
spoonful of soup can accurately represent the taste of the whole pot so
long as everything is well-stirred.
The stirring's the key. That's what introduces the random element
that's so important to scientific sampling. Randomness, in turn, is what
brings the laws of probability into play. And the laws of probability are
what make accurate polling possible.
Everything depends on what's called the "sampling distribution."
The idea is this: If you draw repeated samples out of the same population,
they distribute themselves in a normal (bell-shaped) curve around the average
for all possible samples. And that average will exactly equal the average
for the whole population.
That's what probability theorists call the Central Limit Theorem.
Let's say you're sampling heights from a sample of 100 men who,
on average, are 6 feet tall. You take repeated random samples of 10, drawing
every one of the hundreds of unique samples possible.
You'll get more samples with average heights close to six feet
because there are more of those men and you're more likely to select them.
You'll get damned few samples that average 5 or 7 feet because you're unlikely
to get many extremely short or tall men in any given sample. So the total
"sampling distribution" will form a normal curve that falls away on both
sides of the point representing the average of all samples, which is exactly
6 feet. Like this:
______________________________________________________________
Illustration Here
______________________________________________________________
The magic of this is that the average for the sampling distribution
-- 6 feet -- is exactly the same as the average for the whole population.
Always.
Of course, you don't actually draw the entire sampling distribution.
The beauty of this thing is that you don't have to.
Every sampling distribution forms a normal curve. And for every
normal curve, you can calculate the "standard deviation," which describes
the shape of the normal curve and is based on the total amount that each
individual differs from the average for the whole group. It tells you,
in other words, how flat or steep any given normal curve happens to be.
The critical fact of nature is that in every normal curve about
two-thirds of the things being measured will be within one standard deviation
of the average. About 95 percent will be within two standard deviations.
So every time we draw a random sample, there's a 95 percent chance
that it falls within two standard deviations of the average for all samples,
which is the same as the average for the whole population.
Voila! We have everything we need to figure out how much faith
we should have in our sample.
First of all, we know that we can be 95 percent confident that
our results are within plus or minus two standard deviations of the true
figures for the population we've sampled. That 95-percent figure is what
we call the confidence level.
Second, we can calculate the value of one standard deviation
for this sampling distribution. Once we have that, we can calculate the
number of percentage points that two standard deviations represent in either
direction. That's the margin of error. So we have the two figures -- the
confidence level and the margin of error -- that are critical to assessing
the value of any poll.
In the case of our example, let's say that we draw one random
sample of 10 and that the average man in that sample is 5-foot-10. Let's
also say that we calculate the standard deviation in heights for a sampling
distribution of 10-man samples drawn from a 100-man population. Let's say
it happens to be 2 inches. That means that 95 percent of the possible samples
will average between 5 feet 8 inches tall and 6 feet 4 inches tall. So
we can say that the chances are 95 out of a hundred that the average height
in our sample -- 5-foot-10 -- is within plus or minus 4 inches of the true
average height for the whole population.
That doesn't mean we won't get an occasional wild sample that's
way off the mark. We will, 5 percent of the time. And it doesn't mean that
other sources of error won't bias our results. We could have done a bad
job measuring, for example.
But we can be absolutely certain of the confidence level and
the margin or error. Those result from the Central Limit Theorem. And they're
just as reliable as the Theory of Relativity, the formula for pi or any
other law of the universe.
And that's why George Gallup has been able to call every presidential
election within 3 percentage points since 1952.
……………………………………………………………………………………………………
Reporting the Margin of Error
By Jack Hart
Big stories on polls and surveys should always report the confidence
level and the margin of error. Those two figures give readers what they
need to determine whether two numbers are significantly different. If a
poll determines that 49 percent of voters favor Candidate A and 51 favor
candidate B, for example, how confident are we that Candidate B is actually
leading?
We're pretty good about mentioning the confidence interval in
poll stories. In fact, more than 500 stories have done so in the past three
years. We're much less consistent in explaining the margin of error in
a way that makes sense to readers.
If you're a little murky on the concept yourself, see "How Polls
Work" on page XX. If you don't know and don't care -- or even if you do
know and sorta care -- here's a foolproof formula for reporting the confidence
level and the margin of error.
The chances are 95 out of 100 that the results are within plus
or minus XX percentage points of the true value for the population sampled.
……………………………………………………………………………………………………
Thinking Versus Doing
By Jack Hart
Most of the polls and surveys we report -- especially at this
time of year -- deal with attitudes toward political candidates or issues.
That's fine. One of the things we do is let folks know which way the wind
is blowing.
But we sometimes overlook the value of survey research as a tool
for other things.
For example, opinion surveys can reach beyond public affairs
to sample thinking that reveals lots about our lifestyles and values. Our
rape series reported some fascinating research on how men and women regard
certain sexual behaviors.
But the most revealing surveys usually deal with behavior, not
opinion. They're more reliable, for one thing. And they're a firmer foundation
for building public policy.
The New York Times/CBS News poll that came out during the height
of the Clarence Thomas hearings put an edge on the debate by revealing
that 4 of 10 women said they'd been sexually harassed at work. The annual
survey on high school drug use provides a good baseline for measuring progress
in the war on drugs. And the recent survey of high school sexual activity
was directly related to AIDS-prevention stories and the condom-distribution
controversy.
Sometimes surveys paint a more accurate picture of society than
measures that traditionally dominate news stories. Every year, for example,
we make a big deal of the FBI's unified crime statistics. They generally
show crime on the upswing.
But reported crime isn't a reliable measure of actual crime because
reporting practices change. The best measure of the national crime problem
may be the National Crime Victimization Survey from the Bureau of Justice
Statistics. That annual study, which asks a scientific sample of Americans
what kind of crime they've experienced in the past year, produces a picture
radically different from the FBI's.
According to the crime victimization survey, all American crime
is down about 25 percent since 1973.
……………………………………………………………………………………………………
Learning More
By Jack Hart
If you want a more thorough understanding of survey research, including election polling, we have several useful books on hand. They include:
G. Cleveland Wilhoit and David H. Weaver, "Newsroom Guide to Polls and Surveys."
Victor Cohn, "News and Numbers: A Guide to Reporting Statistical Claims and Controversies in Health and Related Fields."
Victor Cohn, "Reporting on Risk: Getting It Right in an Age of Risk."
Philip Meyer, "Precision Journalism, Second Edition."
The county library has several more texts on the subject. Among the most useful is:
Norman Bradburn and Seymour Sudman, "Polls and Surveys: Understanding
What They Tell Us." (San Francisco: Jossey-Bass, 1988).
……………………………………………………………………………………………………