Here's How This Sausage Was Made
Hopefully this helps the sausage maker with their credibility problem.
Welcome back to TITLE-ABS-KEY(“science journalism“), a newsletter about science journalism research. In the previous issue, I watched, once again, as computer scientists tried to help journalists do their job.
This time, it’s the communications researchers who are eager to help! That is, I’m taking another step back from science journalism to all journalism and looking at a fundamental concept behind the craft (see also my earlier thoughts on source credibility).
Today’s paper: Masullo, G. M., Curry, A. L., Whipple, K. N., & Murray, C. (2022). The story behind the story: Examining transparency about the journalistic process and news outlet credibility. Journalism Practice, 16(7), 1287-1305. DOI: 10.1080/17512786.2020.1870529.
Why: transparency about the journalistic process is something that has fascinated me ever since I did a comprehensive unpacking of a wild story about Russian scientists in North Korea (in Russian, now available only on the Wayback Machine) for a class I was teaching. The unpacking, which explained what I had done for the story and why, was just as interesting and probably more useful for readers than the story itself: it’s unlikely they will ever need to enforce sanctions against a rogue state, but they might need advanced Twitter sleuthing or hacks in using multiple search engines.
Abstract: This three-study project sought to test how news organizations can boost perceptions of credibility of their news outlet by improving journalistic transparency. Three experiments (Study 1: n = 753; Study 2: n = 599; Study 3: n = 321) tested whether adding a transparency box to news stories that explained why and how journalists covered each story would improve credibility perceptions. Study 1 used a mock news site, and the box did not increase credibility perceptions. Study 2 made the box more prominent and used the real audiences of two news sites, USA TODAY and the Tennessean, and the box increased credibility perceptions. Study 3 used a less prominent box on the real audiences of three McClatchy newspapers, and the box did not increase credibility perceptions regardless of where it was placed on the news story. Results suggest transparency boxes may have limited influence on news outlet credibility, but they must be very prominent to have an effect and do not work consistently.
Okay, so this is a slightly different and, in retrospect, more logical take on process transparency: if readers can see how the story was written and why, they will arguably find it and the source more credible.
Well, except that earlier snark analysis in this newsletter, cited above, actually suggests that source credibility has a lot to do with perceptions and priors that the reader is bringing into the evaluation. And again, a theory of change that goes “we give readers more information —> readers start trusting us more“ has that slight whiff of the information deficit model. The reason why readers do not find media sources credible is hardly because they don’t know how they operate.
And that brings us to the ultimate problem: if a prior for a reader is that journalists are lying liars who lie, why on Earth would they find the disclosures themselves credible? (In most cases it’s probably not that grotesquely extreme, but I have a hunch that no amount of transparency about this recent feature will make any of its critics believe I am not in fact a paid shill peddling Western propaganda. I suspect it’s not a transparency issue, you know?)
The paper starts with a description of the fake news problem in the US media – but not the one that’s about disinformation running rampant in the fringe media ecosystem, but the one where Trump has reclaimed the term ‘fake news’ for any reporting or analysis he does not like. So the US media are really struggling with credibility issues, the authors write, and with proving they are not biased and inaccurate.
With that in mind, there’s quite a lot of research on what makes these issues worse. Some of the factors, such as low-quality content or errors in stories, can be remedied and controlled by the media; others, such as uncivil online comments posted on stories – readers shitposting and trolling, pretty much – and relative credibility of the communication channel, eg. TV versus online media, are largely external. I mean, you can always get rid of comment sections on the pages themselves, but unless you also leave social media completely, you can’t escape the incivility.
Less research, the authors seem to argue, is available for factors that make things better. To address that gap, they run experiments to test whether explaining to news consumers how news is produced is one of those factors.
I think I’ve met the only people on the planet who may not have heard of the sausage principle?…
But if I am being less facetious, it’s not entirely unreasonable to think that transparency boosts credibility. If we are asked to accept something as true, and if that something is even marginally important and not self-evident to us, we’d probably like to know how the source of the claim arrived at that conclusion. “Not self-evident” is important here; confirmation bias is very real, so conceivably for things that neatly fit our worldview, we won’t be that interested in the logical steps.
For the purposes of this study and my analysis, I think that is a double-edged sword. On the one hand, if people are faced with a claim that goes against their existing beliefs and are thus interested in the logic leading to it, then maybe showing that logic would genuinely help a reasonable person to see the source as more credible. On the other hand, the reasonable people we are dealing with in the fake news context are – in their minds – very reasonably convinced the mainstream media are lying to them. So seeing a claim that contradicts their beliefs, coming from the media, would likely just reinforce that conviction and thus would not require any additional thought.
The paper goes on to talk about transparency as a journalistic norm as well as earlier attempts to use transparency to boost credibility, with “decidedly mixed” results. Interestingly, they note that the idea that transparency would help news organizations is a result of extrapolation from research on people: when people disclose information about themselves, other people trend to trust them and like them more.
I wonder if the issue here is twofold. First, most people probably do not assume other people talking to them are pursuing some non-obvious interests that shape absolutely everything they say; it would be a very taxing thought to live with. So I don’t know if the extrapolation really works here.
Second, I suspect that transparency as journalists practice it – updates, links to sources, ‘methods’ sections, correcting and explaining errors, allowing user comments, labels – is needed mostly for credibility with other journalists and for an internal sense that you are upholding your end of the invisible contract with the audience. You are simply doing your job by showing your sources and admitting it when you were wrong, but the readers en mass don’t necessarily care that much?… I know that sounds odd, but in my experience, journalists are disproportionately more concerned with these things than audiences are.
That may be why the signal hardly comes through the noise in studies, i.e. even if transparency measures do have an effect, it’s usually small.
(Interestingly, the paper points out that there are much deeper dives into transparency such as the readers’ ombudsmen, but those are notably less popular among media orgs, and Margaret Sullivan’s Newsroom Confidential can give you a good idea why.)
But the ‘transparency box,’ i.e. a box that explains why a story was done, showed some promise in earlier research, so the authors set out to experiment with it. A quick word on the box: I honestly think journalists should produce those boxes for themselves, even if they don’t add them to published stories, especially if the story did not really come from a rabidly competitive pitching exercise. It would likely illuminate so many curious things in the newsroom.
RQ1: Will disclosing the journalistic process of reporting a story to the news audience boost credibility perceptions about the news outlet?
To answer this question, the authors ran three studies with a total of 753 people. Here are the studies:
In #1, people from a demographically representative sample were randomly assigned to read one of two news stories posted on The News Beat, a mock news site, with or without a transparency box. They then rated The News Beat on a five-point scale for transparency, credibility etc; here,
exposure to the transparency box did not produce significant changes in news outlet credibility perceptions.In #2, real news outlets, USA TODAY and The Tennessean, were used in the same setup, this time with those newspapers’ actual audiences – and the papers designed transparency boxes themselves using guidelines from the authors. Now, this time
exposure to the transparency box increased perceptions of news outlet credibility, with a modest effect size.In #3, researchers wondered whether it matters where the box is in the story – and had people read real news stories from The Wichita Eagle, The Sacramento Bee, and El Nuevo Herald with either no box, a box in line with the text, or a box at the bottom. Alas, the effect went away: no matter where you placed the box, it did not significantly help credibility.
Overall, our findings tentatively suggest that adding a transparency box that explains how journalists did a story may have some limited potential to improve perceptions of the credibility of a news outlet — but the box must be prominent to have any effect.
That is a fairly generous interpretation of what happened that the authors themselves immediately hedge with a lot of caution and limitations. Notably, they point out that the box might work as a “cognitive heuristic,” i.e. it’s not really the contents of the box but the mere presence of it that matters. It would be interesting to run this analysis with a sleek-looking box with intentionally dense and obfuscated text that in reality would imply something outrageous, like the reporter having interviewed turkeys for the story.
The authors argue that since their research has shown no downsides to the box, news organizations should consider adding a transparency box to some of their more complicated stories that might raise questions for readers, but they must make the box stand out. They do acknowledge this would be extra work, but see my point above: I do think it would be helpful for journalists, and you don’t necessarily need to publish the boxes – as long as you produce them in good faith and reflect on the results, they’ve done their job.
Overall, this is not a very successful attempt at helping journalists win back their credibility. Then again, perhaps the media needs to be much more strategic about this credibility crisis: for one thing, it’s clearly not the readers lacking any understanding of why and how the stories are made and independently reaching an informed conclusion that journalists are all fake news.