AoIR 2019

Are Journalism Metrics Trustworthy Enough to Be Used in Editorial Decision-Making?

Aljosha Karim Schapals and Axel Bruns

• 3 Oct. 2019 – Association of Internet Researchers conference, Brisbane

Extended Abstract

Introduction: The Quantification of Journalism

With the shift towards online platforms as the primary medium for news consumption (Newman et al. 2018), journalism has undergone an intense process of quantification, well beyond the measures of audience engagement that were available during the print and broadcast eras. Readers’ page clicks, reading time, arrival and departure pathways, and especially also their engagement across various social media platforms are now measured in intricate detail, and reported back to news organisations in real time.

These metrics may then serve to directly influence future editorial decisions: stories deemed to be successful may receive further follow-ups, while fields that generate a poor return on investment may gradually lose staffing support. Such managerial responses to evidence of audience interest are not new in themselves, but the speed of this potential feedback loop between editorial gatekeeping and audience gatewatching patterns, especially in digital and social media, is genuinely unprecedented (Bruns 2018: 225).

Further, this quantification has extended also to the activities of journalists themselves, whose own social media performance can be similarly measured; as a result, many newsroom “managers … put pressure on journalists to be active social media users” (Hedman and Djerf-Pierre 2013: 371). For some, this is a troubling development: “does journalism now include not only the content but also the journalist herself?” (Hedman 2016: 11; emphasis in original). If so, it may entrench the overrepresentation of white, male, cisgendered staff in the journalistic workforce, as staff diverging from that identity are likely to be at greater risk of trolling and other attacks.

Trust in Journalism Metrics?

Such metrics on journalism and journalists are generally reported back to the newsroom and its managers, now often in the form of live dashboards that enable the tracking of the performance of individual stories and offer options to test the respective popularity of minor variations in story presentation (including accompanying headlines, synopses, or images). But such systems have been criticised on a number of levels. First, the data and analyses provided to newsrooms are not necessarily very sophisticated or reliable: a 2016 report concluded that “contemporary forms of analytics are very good at understanding the main ways in which people used digital media in 2010”, but fail to capture more recent digital and social media practices (Cherubini & Nielsen 2016: 39).

Second, the newsroom staff following these metrics may not have the skills and literacy to understand their meaning or act on their observations; in an economically troubled industry, they also lack the freedom to resist the imperative to continuously chase better metrics: “if the company’s not making money then I might get laid off. … That’s just the way it is” (Web editor qtd. In Tandoc Jr. 2014: 570).

Third, news outlets might seek an advantage over their competitors by manipulating their engagement metrics: as BuzzFeed founder Jonah Peretti has remarked, “the natural inclination, if one metric is seen as the important, true metric … is to game it” (qtd. in Salmon 2014: n.p.). This is especially likely when such metrics are used not only to determine performance internally, but also affect advertising revenue or other external rankings.

Journalism Metrics in Everyday Use

Many such shortcomings of current journalism metrics and their operationalisation in everyday newsroom practice are by now well understood by journalists and editors themselves, and some news organisations have developed more sophisticated practices for capturing, analysing, and responding to the metrics they gather on their stories. But the differing levels of digital and data literacy, of technological sophistication, and of editorial commitment to quality as well as quantity mean that approaches to the incorporation of metrics into editorial decision-making processes remain highly divergent across the industry.

This paper, therefore, reports on the results of a large-scale comparative analysis of newsroom practices in the use of metrics across Australia, Germany, and the United Kingdom. Drawing on a large number of interviews with journalists and editors at large and small, established and emerging, legacy and born-digital news outlets, we develop a comprehensive picture of the incorporation of journalism metrics into editorial practices, with a particular focus on journalists’ and editors’ own assessment of the degree to which they are willing to trust these metrics in their decision-making. We examine both the differences and the commonalities in approaches across the various types of news organisations, and explore the influence of different national media systems and traditions.

What emerges from this work is a new perspective on the trustworthiness of journalistic metrics, in the view of practicing journalists themselves. In the words of one interviewee working for an emerging, born-digital news outlet, “if our job as journalists is to make sense of the world [and] trying to explain [things], then I think it’s completely horrible if everything is just narrowed down to a single score” (personal communication, 17 October 2017). Overall, our preliminary findings indicate a sense of ambiguity about the role journalistic metrics (should) play in editorial decision-making: while some heralded the respective affordances of the technology as allowing them unprecedented insights into audience interactions with journalistic output, others went as far as to suggest that an increased emphasis on such metrics could potentially lead to certain stories not being covered at all – to the detriment of story diversity across a variety of journalistic beats. This paper synthesises these views and concludes with an agenda for future research for an industry that, as a result of the continuing ‘fake news’ debate, is also battling an overall lack of public trust in the media.

References

Bruns, A. (2018). Gatewatching and News Curation: Journalism, Social Media, and the Public Sphere. New York: Peter Lang.

Cherubini, F., & Nielsen, R.K. (2016). Editorial Analytics: How News Media Are Developing and Using Audience Data and Metrics. Oxford: Reuters Institute for the Study of Journalism, University of Oxford. http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2739328

Hedman, U. (2016). When Journalists Tweet: Disclosure, Participatory, and Personal Transparency. Social Media + Society, 2(1). https://doi.org/10.1177/2056305115624528

Hedman, U., & Djerf-Pierre, M. (2013). The Social Journalist: Embracing the Social Media Life or Creating a New Digital Divide? Digital Journalism, 1(3), 368–385. https://doi.org/10.1080/21670811.2013.776804

Newman, N., Fletcher, R., Kalogeropoulos, A., Levy, D.A.L., & Nielsen, R.K. (2018). Reuters Institute Digital News Report 2018. Oxford: Reuters Institute for the Study of Journalism. http://media.digitalnewsreport.org/wpcontent/ uploads/2018/06/digital-news-report-2018.pdf?x89475

Salmon, F. (2014, June 11). BuzzFeed’s Jonah Peretti Goes Long: The Media Mogul (Twice Over) on Being Both Contagious and Sticky. Medium. https://medium.com/matter/buzzfeeds-jonah-peretti-goes-long-e98cf13160e7

Tandoc Jr., E.C. (2014). Journalism Is Twerking? How Web Analytics Is Changing the Process of Gatekeeping. New Media & Society, 16(4), 559–575. https://doi.org/10.1177/1461444814530541