Statistical Significance: Making Sense of Numbers

Like a lot of writers and artists, I had an aversion to math when I was younger. Then, I took a statistics class in high school, and it changed the way I viewed the study of mathematics, its real-world application, and, perhaps most profoundly, my own ability to understand analytical concepts.

It sounds incredibly geeky, but once I learned the difference between mean, median and mode, and how frequently they’re confused, I never looked at statistics in a news report, election result, textbook chapter, or research paper in the same light.

Lately, I’ve read more than a handful of articles and press releases that misuse or leave out statistics gathered via surveys or studies.

As professional writers, we occasionally need to delve into the world of math and statistics. Whether you’re preparing a press release, writing an article for the company website, or reporting the news, your facts may rely on figures.

What you don’t want is to be associated with that old saying about lying with statistics. Your credibility rests on your transparency with numbers, especially when the goal is something like press coverage, promotion or a fact-driven news story.

Here are some basics for writing with numbers:

Include the number of subjects who participated in a survey or study
This figure is an absolute requirement (and I’ve seen it missing from more than three recent press releases or website posts in as many months). It is, after all, the starting point for any survey or study and it provides the reader with an ability to judge how relevant the data might be.

Note that in any study or survey there’s a percentage of answers or results that must be discarded: a participant chose not to answer one or more of the questions or results from a lab test were unclear. From a pool of 400 participants, a plus or minus error rate of 1 – 2 percent might be acceptable, but if only 10 people took the time to fill out a survey, even one incomplete or botched entry makes a huge difference to the quality of results.

Frankly, if the participating group is as small as 10, the survey or study probably doesn’t hold a lot of weight scientifically. This hasn’t stopped reporting of such results, but frequently what the reader isn’t told, because it would cast doubt on the validity of both the results and the assertions made about them, is how many people were in the study group.

In scientific and medical research, groups that small are often part of preliminary studies, which is why there is so much regulation around reporting findings that may not have any bearing on larger populations of patients.

Provide a breakdown of participant groups
The more details you provide, the more credible your story becomes. It doesn’t necessarily make the survey or study more credible, but you give your reader the ability to assess the information based on their own understanding of the subject and related facts.

Participant details (as long as you are sharing non-identifying, unconfidential information) can include things like gender, age group, political affiliation, economic strata, professional experience, blood type, etc.

Share types of questions asked, specific information or samples gathered from subjects
Certainly, there’ll be articles where this counts as too much detail, but in a scientific study it might be essential to understand that blood samples were taken within a certain time period following the administration of medication.

For surveys, sharing a greater level of detail comes in handy when highlighting a particularly notable response. Knowing the question helps the reader analyze the answers.

You get bonus points for including the scale used to score a survey (and for using more complex scales – five response options rather than two or three – when conducting a survey in the first place).

Be clear about the scope of findings
Avoid at all costs “universalizing” results. This is a mistake I see frequently. Unintentionally or not, extrapolating findings from the original study group to a large population (for example, using one exit poll to predict an election) misrepresents the scope of the results. It’s striving for a significance that isn’t there.

A survey or study has a specific number of participants; the findings refer to the original population, especially with only one study and no further research to verify the original results.

Results must be qualified with statements like “among people who took the survey,” “according to survey respondents,” “in the study,” and, for large studies, the data may need to be further broken down to percentages within the various subgroups (such as, “x% of the women/seniors, etc., in the study population experienced reactions”).

Avoid calling survey or study methodology “scientific”
Like extrapolating data, appending the word “scientific” to any old survey or study is striving for a kind of credibility the research may not have earned.

The scientific method has since the 17th century provided objective, measurable, repeatable standards and techniques for investigating subjects and gleaning new information. Wikipedia offers a more comprehensive explanation here, but suffice to say, if a study wasn’t conducted by an agency with no investment in the outcome (objective), didn’t start with a hypotheses against which results could be compared and contrasted, proved or disproved (measurable), and wasn’t repeated to ensure reliability of reporting, calling it “scientific” is a misuse of language. Speaking of which…

Understand key terms like “statistical significance”
The word “significant,” when attached to study data, does not translate as “key,” “momentous” or “important.”

“Statistically significant” is a phrase statisticians use when a result is “unlikely to have occurred by chance.” Which also doesn’t mean that it’s important simply because it’s unlikely. The unlikely result would need to be repeatable and measured through additional objective means (at the very least) in order to determine its real-world significance.

Do you have a mathematical pet peeve? What examples of exaggeration have you noticed when it comes to writing with numbers? Feel free to enumerate in the Comments.

Writing that inspired me this week:

“Figures often beguile me, particularly when I have the arranging of them myself; in which case the remark attributed to Disraeli would often apply with justice and force: ‘There are three kinds of lies: lies, damned lies, and statistics.’”
~ Mark Twain, “Chapters from My Autobiography”