A (News) Worthy Cause
What happens when AI scientists decide to help journalists
Hi there! This is TITLE-ABS-KEY(“science journalism“), a newsletter about science journalism research. In the previous issue, I got existential when I read a paper exploring just why science journalists haven’t all quit yet.
This time, I am once again asking for your support reading a paper on machines in science journalism. But now these machines do not write the news stories — rather, they hack the most sacrosanct of all newspeople concepts, newsworthiness.
Today’s paper: Nishal, S., and Diakopoulos, N. (2022). From Crowd Ratings to Predictive Models of Newsworthiness to Support Science Journalism. Proc. ACM Hum.-Comput. Interact. 6, CSCW2, Article 441 (November 2022), 28 pages. DOI: 10.1145/3555542.
Why: I mean, robots (2). I just told my Russian-language subscribers how ChatGPT had failed my easiest course assignment, and I’ve already covered automation in science journalism in this section.
Abstract: The scale of scientific publishing continues to grow, creating overload on science journalists who are inundated with choices for what would be most interesting, important, and newsworthy to cover in their reporting. Our work addresses this problem by considering the viability of creating a predictive model of newsworthiness of scientific articles that is trained using crowdsourced evaluations of newsworthiness. We proceed by first evaluating the potential of crowd-sourced evaluations of newsworthiness by assessing their alignment with expert ratings of newsworthiness, analyzing both quantitative correlations and qualitative rating rationale to understand limitations. We then demonstrate and evaluate a predictive model trained on these crowd ratings together with arXiv article metadata, text, and other computed features. Based on the crowdsourcing protocol we developed, we find that while crowdsourced ratings of newsworthiness often align moderately with expert ratings, there are also notable differences and divergences which limit the approach. Yet despite these limitations we also find that the predictive model we built provides a reasonably precise set of rankings when validated against expert evaluations (P@10 = 0.8, P@15 = 0.67), suggesting that a viable signal can be learned from crowdsourced evaluations of newsworthiness. Based on these findings we discuss opportunities for future work to leverage crowdsourcing and predictive approaches to support journalistic work in discovering and filtering newsworthy information.
Now, I have to confess something first. When I recently recounted the abject failure of ChatGPT at completing the first assignment in my multimedia science journalism course, I casually noted that automating the news should be relatively straightforward — especially if you exclude the very first step in the process, picking a story. My rationale was that, as anyone who has taught journalism students knows too well, it is genuinely hard to turn that gut feeling into a manageable algorithm.
What I somewhat conveniently forgot was that’s not even how it works anymore. The diabolical and irresistible power of machine learning lies in the fact that you don’t have to tell the model how you figure out what’s newsworthy. It will swallow a dataset of these decisions into that black hole of a black box, and it will return its judgment. No one will be able to tell just how the gut feeling is broken down into parameters that can be evaluated, but you will have some kind of newsworthiness scores that may likely track uncomfortably well with your gut.
(You will then question your self-worth, spiral into despair and burnout, and maybe write a newsletter about it.)
Well, except for the final italics, that was basically a recap of this whole study. But I really want to dive deeper into what they did, so let’s do that.
From its inception more than a hundred years ago as the “Gee-Whiz” reporting of new scientific findings to more modern conceptions of “telling the whole complicated story” science journalism occupies an important role in society, serving both to translate and critique scientific findings that have important bearing on a range of issues, from climate change and global pandemics to the rise of artificial intelligence in social systems.
That’s certainly one way to recap the history of science journalism in one sentence. But I understand that the authors really need to get to the ‘now,’ where science journalists are essentially sandwiched between a “changing media ecosystem” with diminishing resources and a growing terrain science journalists must cover.
Right, I’m sensing a particular angle here...
Our work explores how to help science journalists effectively grapple with monitoring the growing scale of scientific information available in the present environment. To do so we develop a socio-technical approach leveraging crowdsourcing and a machine-learned model for predicting newsworthiness.
I was not wrong! Computational news discovery, the notion scientists put at the center of the paper, is supposed to help journalists deal with the proverbial firehose. So, effectively, the problem of interest with the ‘sandwich’ I mentioned above is that there are just too many papers. No reasonable human being can ever read all those papers, so we will have a machine “read“ them first and pick the newsworthy ones.
That’s an interesting take on the state of science journalism, and especially on “telling the whole complicated story.“ For one thing, suppose I’m generous and assume that the authors do not in fact reduce all science journalism to a conveyor belt of news stories based on single published papers. Yes, papers can be solid hooks for bigger stuff, and they can be the threads you pull on to get to the complications. But that’s not usually about their newsworthiness, and I’m not sure a machine trained to assess newsworthiness will pick up on hints for enterprise reporting potential.
And honestly I wouldn’t say that I’ve ever been truly bothered by my inability to keep up with my beat, even though my beat — policy-relevant science with a Russian twist — is about as wide as they go. It was more, you know, navigating an ocean of disinformation, conflicts of interest and agendas, all while struggling to make a living as a freelancer? Or, at a later point in life where I was freelancing on the side, I was once offered a higher rate to encourage me to write more; I smiled at the assumption that I wasn’t writing more because of the rate and not because I could not physically add more work to my 80-hour week.
But I guess AI can’t help me with any of that (yet), so I’ll have to settle for “reading“ the papers. I mean, can I treat this computational news discovery as yet another layer for my DIY and admittedly a bit slapdash monitoring system, the one that can go further than a string of keywords? Hmmm…. perhaps. But then my question is, if it is trained on some sort of aggregate industry gut feeling and not my gut, will it offer me stories (a) I will want to pursue and (b) I will be able to pitch in line with my ‘unique selling proposition’ as a writer?
At this point I feel slightly ungrateful to the authors who seem to genuinely want to help me effectively grapple with monitoring the growing scale of scientific information available in the present environment. So let’s see how they do that.
Despite what I wrote here about AI not needing your gut-based explanations of newsworthiness, the authors actually do come up with a set of news values derived from expert evaluations of newsworthiness from professional science journalists. They need these to see whether any of the values can be feasibly crowdsourced from laypersons, because that’s the kind of big (bigger?) data you need to train a predictive model.
Here is where I get to my second concern, the first being that we’re looking for lost keys under a street light, i.e. solving the most AI-friendly problem of science journalism that I’m not quite sure causes industry-wide burnout. (That was this newsletter’s least veiled sarcasm so far.)
To me, newsworthiness is not some physical property, like mass or sugar content, that exists independently of the journalists and their audience. That is, the reason crowdsourced ratings of newsworthiness often align with expert opinions on newsworthiness is because journalists are building the news landscape. They both write from an imperfect mental model of their audience and its interests and shape those interests. It’s mind-bending if you try to really focus on any part of this, like an Escher drawing, but also something really obvious and natural to anyone who’s worked as journalist.
What’s more, you now have other forces shaping those audience interests (and by extension the mental model — it’s loopy a feedback loop), namely algorithmic distribution and social media, which the authors file under ‘things the media ecosystem is adapting to.’ And let me just say I don’t like what those forces have been doing to readers’ ideas of newsworthiness!
In short, if we train our AI ‘screeners’ of potentially newsworthy stories on this status quo, represented by a poll of Amazon Mechanical Turk workers no less — are we going to accidentally end up like Amazon and its HR robot who did not like women? Except our problem is likely not going to be that glaringly obvious.
I feel like I’ve spent so much time outlining my general concerns and so little time quoting the paper that this issue feels unusual. So, naturally, what I’m going to do is get to my third quibble! (If you are interested in the nuts and bolts of the predictive model design, it’s best to just read the paper in this case. TLDR: the machine learning algorithm did surprisingly well.)
When the authors introduce the notion of computational news discovery and how it can help journalists, it goes kind of like this:
News discovery is costly;
“Factors extraneous to a potentially newsworthy event” (e.g. staff shortages) can play a significant role in news selection;
(This is a citation from a paper) Scientific output continues to rise, and yet its appearance in news media is less than 0.013% of total articles published [in the same period];
This is happening because (1) there are soooo many papers and (2) scientists fail at communicating interesting results to lay audiences;
We’re going to focus on (1).
Now, do I disagree that “extraneous“ factors play a big role? Nope, and I don’t like that; I’m on the record repeatedly telling audiences I want to have so many science journalists and so much money in it that, for every arcane and bizarre story, there are always at least one matching nerd who is available to cover it and one other nerd happy to commission it.
But this line of thought also felt just a little bit like ‘so you guys seem to suck at Objectively Picking the REALLY Newsworthy Stuff, and that is why there’s so little science in the media — BUT we’re going to build a machine that is never tired or burned out, doesn’t need health insurance or lunch breaks, it will help you find all the stories, AND THEN science will get the coverage it deserves!’
Um, no, I don’t think it works that way. And sorry, but LOL at comparing the total number of papers published and the number of media science stories.
With all of that in mind, I guess I’m still open to testing a ‘smart monitoring‘ system designed this way (expert evaluations → crowdsourced ratings → predictive model). Not least because this system will probably be better at approximating the status quo of audience interests than my (severely limited and fuzzy) mental model.
Interestingly, that first transition, from journalists to readers, dropped almost all news values that the authors deemed too complex and specialist for non-journalists to assess just from the arXiv preprints they were shown, like framing and angles, potential as a hook for a bigger story, or marketability and click-worthiness. So that approximation might even end up ‘cleaner’ than my mental model, which is inevitably colored by déformation professionnelle.
In the Discussion section, the authors circle back to my second concern, in a way. Because expert and crowdsourced assessments do not align perfectly well, the scientists wonder whether the prediction model should be better attuned to expert opinions to be more useful — after all, they are here to help journalists. But then they go on to ask whether instead we might have the audience i.e. the crowd-workers, act as the arbiters of what is "newsworthy", and consequently formulate a model that aims to predict the interests of the crowd.
My thought process here will require some quick context. In the currently-fashionable conspiracy theory around ‘15-minute cities,‘ there is an entire school of thought positing that one day, the concept will inevitably change from ‘you don’t need to go anywhere else’ to ‘you’re not allowed to go anywhere else.’ So I read that section of the paper on arbiters of newsworthiness and instantly thought of a future newsroom where ‘the Model has determined this story is not newsworthy.’
I’m not too worried we’ll get there anytime soon, but it is a little surprising that the closest the authors get to this hypothetical is by acknowledging this pursuit could also encourage the tabloidization of the news media. Yet the very next section is about the risk of paper abstracts changing to perform better in the newsworthiness model. Right, because getting picked up by science journalists is what shapes the academic language right now… (Mediatization of science is real but abstracts are still safe I’d say.)
You know I’ve adopted the meerkat metaphor for this newsletter: imagine that you could get a meerkat commenting on all those fascinating studies of meerkats and their complex social behavior! Most of the commentary would likely be ‘yeah no we were just reeeeally bored that day‘ but it still sounds cool.
So, in this case, I absolutely believe the authors have the best intentions, I truly do, and it’s not like journalists aren’t interested in automation and a new era of computer-assisted reporting. This is useful! And it still reads like a paper by aliens whose understanding of meerkats is mostly shaped by, I don’t know, one stuffed meerkat they put into their spectrometer?
(If our future science news overlords do take over, I now fear this newsletter will not help my case. Oh well.)
That’s it! If you enjoyed this issue, let me know. If you also have opinions on science journalism research or would like to suggest a paper for me to read in one of the next issues, you can leave a comment or just respond to the email.
Cheers! 👩🔬🤖