Transparency News

How the scores work

Every step the site runs, in order, with the research or newsroom rule behind it, every context rule, and every listed word. Every number on the site comes from these steps, so anyone can check a score by hand.

Automated assessment. Scores measure selected wording patterns in the outlet's own words. They do not assess factual accuracy, completeness, or which stories an outlet covers.

Method 3.1.0 · Rule book fingerprint (SHA-256) 85d55227ee29cf170a0aba8314199879f2d946e64244edbbcebe477dfc12fe73. Any change to a single rule changes it. · Research notes · Audit · What changed · Source code · Model card

The standard

Unbiased reporting says what happened, who said what, and how we know. Its words match the weight of the event. It names its sources. It labels opinion as opinion. It includes the facts a reader needs to understand the event.

The score measures part of this standard: listed wording in the outlet's own voice, flagged attribution phrases, and headline wording. The About page sets out which parts are measured, partly measured and not measured.

Step 1
Read the outlets' feeds

The site reads each tracked outlet's public news feed (RSS or Atom) several times a day and keeps reports published in the last 120 days. Readers can ask for a new outlet on the request page.

The rule

Feeds are read as TransparencyNews/0.1 (+https://transparencynews.org/about); article pages and front pages as TransparencyNewsBot/2.0 (+https://transparencynews.org/method). The reader checks each site's robots.txt first and skips any page the site asks automated readers to leave alone. Up to 40 article pages are read per update, newest first.

Where an outlet publishes its own plain-text edition, the reader tries it first, under that edition's own robots.txt: NPR stories are read from text.npr.org. When the page's paragraphs give fewer than 150 words, the reader also checks the story text the page itself carries for search engines (the schema.org "articleBody"). The reader never works around a refusal: no other reader name, no proxy, no cached or mirrored copy.

Why, and the backing: Respecting robots.txt follows the Internet standard for automated readers [1]. A site that refuses automated reading is measured on headline and summary only; those results are listed apart and never enter the outlet ranking. Which outlets are read in full is on the coverage map.

Step 2
Separate the outlet's own words from quotes

From each article page the site takes the readable paragraphs of the story. Text inside quotation marks is a speaker's words, reported by the outlet, and is set aside from the score. The site counts the quoted words and lists any listed word that appears inside quotes. A scare quote is different: a phrase of one to three words in quotation marks with no speaker in the same sentence (within 60 characters). The reporter chose to put it there, so since method 3.1.0 its words stay in the outlet's own text and count in full. Scare quotes are also listed on every report page.

The rule

A report needs at least 150 words in the outlet's own voice, read from the full page, to count toward the outlet's score.

What the site keeps. The published record holds counts, matched words and a few words around each match. Since method 3.1.0 the site can also keep the text of each page it read with the outlet's permission, in private storage that is never published. That storage is not switched on yet, so today no text is kept. It is used only to measure reports again after a rule change and to settle disputes. An outlet can ask for its stored text to be deleted through the corrections page. Pages behind a subscription are never read.

Why, and the backing: Newsroom standards hold reporters responsible for their own wording and attribute loaded words to the people who said them [2] [3]. Which quotes a reporter chooses is still an editorial choice, so quoted loaded words are shown beside the score.

Step 3
Recognize wire copy and opinion

A report whose text opens with a wire service's dateline credit, such as "WASHINGTON (AP)", is wire copy. It is shown with a wire tag and left out of the publishing outlet's own score, because the words are the wire service's. A report counts as opinion when the outlet labels it as opinion, by a section in its web address (such as /opinion/) or a category in its own feed. Opinion is scored with the same rules, tagged, and left out of the outlet's news score.

The rule

Wire credits recognized: AP, Reuters, AFP.

Why, and the backing: The dateline credit is how wire services mark their copy [4]. The journalism code of ethics asks outlets to "label advocacy and commentary" [5]; using the outlet's own label keeps the site from guessing intent.

Step 4
Find listed words and phrases

Each report is checked against published word lists. Matching ignores capital letters, counts whole words and phrases only, and tries longer phrases first, so "woke mob" counts once as a partisan label and not again as "mob".

The rule

LayerWhat it counts
Word choiceContentious and delegitimizing labels, partisan-coded terms, expressions of doubt, editorializing, loaded words for "said", puffery, and (experimental) insults
TemperatureWords that turn the heat up beyond what a plain description needs, and words that turn it down (euphemisms that blur what happened or who did it). Both directions cost points.
Flagged attribution phrasesUnnamed sources and vague attributions such as "experts say", when the story gives no reason for anonymity

Partisan-coded terms come in 15 mirror pairs, each worth 3 points, so neither side's vocabulary costs more. Every entry is listed below.

Why, and the backing: Word choice entries follow newsroom style rules [3] [2]. Studies of biased wording found that one-sided terms, factive verbs (such as "admitted") and assertive words carry a writer's view into a sentence [6]. Partisan phrase pairs follow research showing that phrase choice tracks political slant [7]. Unnamed and vague sourcing follows the journalism ethics code [5].

Step 5
Read each match in its context

A match counts only when the reporter is using the word in their own voice. Every match gets one of five classes, and four of them are shown but not counted.

The rule

ClassRuleCounted
NegatedOne of these words appears within 3 words before the match: not, no, never, nor, without, isn't, wasn't, aren't, weren't, isn’t, wasn’t, aren’tNo
AttributedThe same sentence has an attribution verb (said, told, according to, accused and others) with a named speaker. Unnamed and vague sources ("sources said", "critics say") do not count as attribution. Does not apply to loaded words for "said", doubt words, or sourcing phrases, which are the reporter's own choices.No
DiscussedThe word is introduced as a term: "the phrase", "the term", "the word", "the label", "words like"No
LiteralA strong word describing real physical damage or a literal event, listed belowNo
ReporterNone of the aboveYes

Literal uses:

  • explosive followed by device, devices, material, materials, vest, vests, and others
  • devastating followed by earthquake, quake, fire, fires, flood, floods, and others
  • destroyed followed by homes, buildings, houses, vehicles, structures, bridges, and others
  • destroys followed by homes, buildings, houses, vehicles, structures, bridges, and others
  • brutal followed by cold, heat, winter, summer, weather, killings, and others
  • catastrophic followed by flooding, floods, damage, failure, injury, ies, and others
  • massive followed by earthquake, quake, fire, storm, explosion, flooding, and others
  • panic followed by attacks, disorder, button, buying
  • admitted followed by to, into the hospital, intensive care, emergency room, bar, practice, and others
  • admits followed by guilt, guilty, to the charges, crimes, killing, murder, and others
  • conceded followed by the race, election, primary, contest, defeat
  • concedes followed by the race, election, primary, contest, defeat

Why, and the backing: Short negation windows are a tested way to catch negated findings in text without a full language parser [8]. Every match on every report page shows its class and the words around it, so a reader can see where the rule got the context wrong and dispute it.

Step 6
Set each word's severity from its evidence

Each listed word has a severity from 1 to 3 points. For word choice and sourcing, severity follows the newsroom rule behind the entry. For temperature and softening words, which rest on reader research, severity is capped by how strong the research is for that word group.

The rule

Evidence for the word groupHighest severityWords
Direct: a study tested this kind of wording on readers322
Category: studies tested the wider category247
Indirect: related studies only140
None foundExperimental: shown, not counted30
Newsroom rule (word choice, sourcing)As listed131

Effective severity = the lower of the listed severity and the cap for its group's evidence. Insults and name-calling are experimental in this version. An unnamed source whose anonymity the story explains (for example "not authorized to speak publicly") costs 0 points and is still listed.

Why, and the backing: The groups and their studies are on the evidence page, with what each study found, how it applies, and its limits. Explaining why a source is granted anonymity is the practice the major stylebooks ask for [4].

Step 7
Turn points into a rate

Counted points are divided by the number of words in the outlet's own voice, so a long report is not penalized for being long and a short one is not excused for being short.

The rule

rate = counted points ÷ own words × 1,000

Why, and the backing: Putting every measure on the same per-length basis before comparing is the standard first step for any composite score [9].

Step 7b
Find the densest stretch

A rate over the whole story can be lowered by padding: neutral text around a loaded passage. So for a story longer than 200 words, the site also finds its densest 200-word stretch for each layer and uses the average of the two rates.

The rule

rate used = 0.5 × whole-story rate + 0.5 × densest-stretch rate

Stretches of 200 words start at word 1, 51, 101 and so on, plus one stretch ending at the last word. The densest is the one with the most counted points; its rate is its points ÷ 200 × 1,000. A story of 200 words or fewer uses its whole-story rate. Each report page shows the stretch and the words in it.

Why, and the backing: Adding neutral text can still lower the whole-story half of the rate, so padding helps less but does not stop helping. The densest-stretch half cannot be lowered by padding.

Step 8
Place the rate on a 0 to 100 line

Each layer has two fixed ends. A report with no counted words scores 100. A report at the loaded end or beyond scores 0. Between them the line is straight, so every counted word costs the same share of the score wherever it falls.

The rule

layer score = 100 × (loaded end − rate) ÷ loaded end, kept between 0 and 100

LayerLoaded endIn plain terms
Word choice40About one 3-point word every 75 words
Temperature40The same density
Flagged attribution phrases20One unexplained unnamed source every 100 words

Attribution is "not established" when a story has no attribution marker (such as "said" or "according to") and no flagged sourcing phrase, at any length. That layer is then left out of the total instead of scoring 100. (Before 3.1.0 this applied only from 150 words, so a short story with no attribution could score 100 on this layer.)

Why, and the backing: The ends were checked against 50 Federal Register abstracts [10] and 49 left-leaning plus 29 right-leaning opinion articles. Fixed, published ends are the "min-max" normalization method for composite indicators [9].

Step 9
Score the headline on its own

Headline words count only in the headline layer, never again in the story layers. Headlines are short, so the headline has its own scale and counts as at least 25 words.

The rule

headline rate = counted headline points ÷ max(headline words, 25) × 1,000
headline score = 100 × (1 − headline rate ÷ 240), kept between 0 and 100

Shown, not scored: the headline's rate minus the rate of the story's first 80 words, and (experimental) how many names and figures in the headline also appear in the story.

Why, and the backing: Headlines that overstate or slant the story change what readers remember, even after reading the article [11]. Clickbait-style headlines lower readers' trust in the outlet [12].

Step 10
Combine the layers

The layer scores are combined with a weighted geometric mean, shifted by 10: 10 is added to each layer before multiplying and taken away at the end. Layers that are not established are left out and the remaining weights are rescaled.

The rule

overall = (Word choice + 10)0.4 × (Temperature + 10)0.3 × (Flagged attribution phrases + 10)0.2 × (Headline wording + 10)0.1 − 10

A report scoring 100, 100, 100 and 40 gets 92 this way; an ordinary average gives 94. A perfect story with a headline scoring 0 gets 77. Before 3.1.0 there was no shift and each layer counted as at least 1, so that story got 63: one zero cost more than a third of the score. A weak layer still pulls the total down.

Why, and the backing: The geometric mean limits how much a strong layer can make up for a weak one [9]. The United Nations' Human Development Index switched to it in 2010 for the same reason [13]. The shift is a standard fix for zeros in a geometric mean [14]. The audit re-ranks every outlet under other weights to check the order does not hinge on these choices.

Step 11
Score each outlet

An outlet's score averages its own news reports read from the full page (opinion and wire copy left out), pulled toward a starting value when it has only a few reports. Reports measured on headline and summary only are listed apart and never enter this average. The pull can never move an outlet into a better band than its plain average; when it would, the adjusted score stops at the top of the plain average's band. Both numbers are shown, along with a same-event comparison.

The rule

adjusted = (n ÷ (n + k)) × plain average + (k ÷ (n + k)) × μ0

μ0 and k come from the data and are recomputed every quarter: μ0 is the average of the outlets' plain averages, and k is the spread of report scores within an outlet divided by the spread between outlets, kept between 2 and 30. The current values and the date they were set are on the outlets page. They are set only from 10 or more outlets and 150 or more reports; until then μ0 is 94 and k is 5.

90% range, fewer than 30 reports: plain average ± t × standard deviation ÷ √n, each end pulled the same way
90% range, 30 or more: 2,000 re-draws of the outlet's reports, each averaged and pulled; the 5th and 95th percentiles

t is the 95th percentile of Student's t with n − 1 degrees of freedom (2.13 at 5 reports). The re-draws use a fixed seed made from the outlet's id, its number of reports and the method version, so the same data always gives the same range. No outlet score appears below 5 reports or 3,000 words; 5 to 9 reports is provisional.

same-event comparison = plain average + average of (report score − that event's average report score)

Uses events covered by 3 or more outlets, and needs 3 such reports. It compares an outlet with the others covering the same news, so its mix of topics matters less.

Why, and the backing: Pulling small samples toward a common value gives more accurate estimates than raw averages [15]. With few reports a re-drawn range is too narrow, so the range uses Student's t [16]; from 30 reports it uses the bootstrap [17].

Step 12
Group reports into events and order them

Reports join the same event when they share enough distinctive words and names and were published within two days of each other. Names count double; very common political words count half. Each event lists its reports in the order outlets published them, using each outlet's own feed time. Reports less than a minute apart count as a tie. Events within seven days that share words but were not joined are linked as related events.

The rule

match = shared weighted words ÷ weighted words in the smaller report; join when the match is at least 0.5 and at least two distinctive words are shared

Each event shows its grouping confidence: the weakest match that joined a report. Wrong groupings can be reported on any event page and are corrected by hand; every correction is listed on What changed.

Why, and the backing: "Earliest" means the earliest publication time among the outlets this site reads, as each outlet gives it. It does not show who broke the story.

Step 13
Record each measurement

Every measurement records the method version, the word-list fingerprint, the time the page was read, the text extractor version, a fingerprint of the text read (so a later reader can tell whether the page has changed), and a link to the Internet Archive's copy nearest that time. The site asks the Internet Archive to save up to five newly read pages per update.

The rule

When the rules change, reports still in the outlets' feeds are read again. Reports whose page text was kept are measured again from that text, up to 80 per update; until then they are held out of the scores. Reports with no kept text leave the live scores and stay in the archive.

Why, and the backing: The Internet Archive's copy lets anyone check what the page said when it was scored [18], without the site publishing the text itself.

Step 14
Test the method

Tests fixed before the method ran on live outlets check that the rules do not favor one side, and that the method measures what it says. Results are on the audit page, including tests that do not pass and the cases the method gets wrong.

The rule

  • Severity points on each side of the partisan list match within 5%.
  • Every left-coded partisan term has a right-coded mirror with equal severity and the same context rules, so equivalent usage gets equal points.
  • Average score change when party, politician, institution and political-adjective names are swapped is 0.5 points or less, and no single report moves more than 2 points outside published exceptions. The swap includes "Transparency Party" and "Libertarian Party".
  • Diagnostic, not a pass mark: the left-leaning and right-leaning opinion samples are shown side by side. Different samples can legitimately use listed words at different rates, so equal results are not the target. The test that matters is equal treatment of equal usage (mirror pairs and name swap).
  • Every saved report that mentions the Transparency Party is listed with its score and matched words, so readers can check that coverage of the founder's party is scored like everything else.
  • Human panel agreement (Krippendorff's alpha) of 0.80 or higher is firm; 0.667 to 0.80 is tentative; below 0.667 the word is reviewed. Agreement is reported per word group as well as overall.
  • Per-word fairness (Mantel-Haenszel, ETS scale): below 1.0 passes, 1.0 to 1.5 goes to review, 1.5 or above is revised or removed. Groups: passages from left-leaning and right-leaning outlets. Matched on: the panel's overall rating of the passage (1 to 5).
  • Outlet rankings under alternative weights correlate with the default at 0.9 or higher (Spearman).
  • Every method change publishes how outlet rankings moved: the rank correlation of outlet scores under the old and new method on the same reports.

Why, and the backing: Rater agreement uses Krippendorff's alpha [19]. Per-word fairness uses the Mantel-Haenszel test [20]. Weight sensitivity and the before-and-after comparison use Spearman rank correlation [21].

A worked example

A 600-word news story with a 12-word headline. The headline says a senator "slammed" a bill. The story says the bill's backers "claimed" it would cut costs, calls the bill "controversial" and "sweeping", quotes a senator calling it a "sham" (attributed, so not counted), mentions a "devastating" storm (literal, not counted), and cites "people familiar" with the talks without saying why they were not named.

LayerCounted pointsWordsWhole-story rateDensest 200 wordsRate usedLayer score
Word choice (claimed 1 + controversial 1)26003.310.0 (words 1 to 200)6.783
Temperature (sweeping 1)16001.75.0 (words 1 to 200)3.392
Flagged attribution phrases (people familiar 2)26003.310.0 (words 251 to 450)6.767
Headline wording (slammed 1, capped by its evidence)12540.040.083

"claimed" and "controversial" sit at words 41 and 121, and "sweeping" at word 151, so the first 200 words are the densest stretch for both layers. "people familiar" is the only sourcing phrase.

Overall = (83 + 10)0.4 × (92 + 10)0.3 × (67 + 10)0.2 × (83 + 10)0.1 − 10 = 82 82

The headline's word count is raised to the 25-word floor. These numbers come from the same code that scores live reports. Every live report has a page with its own arithmetic.

Shown beside the score: three panels

Three panels on report, event and outlet pages count things the wording score does not. None of them changes a score. Each uses published patterns, so each count can be checked by hand.

What a reader could check

  • Named sources: a capitalized name of two or three words right before or after said, says, told, wrote, testified, added, explained or noted.
  • Unnamed sources: the unnamed-source phrases on the word lists.
  • Primary-record links: links in the story to government (.gov, .mil), court, congressional, Federal Register, SEC, Census, BLS or FEC sites, document hosts, or PDF files.
  • Documents cited: phrases such as "according to the report", "court records", "the complaint says".
  • Checkable specifics: numbers and dates in the text read.
  • Data shown: a table or chart in the page.
  • Correction noted: "Correction:", "Clarification:", "An earlier version of this story".
  • Cites no one: no named source, no unnamed source, no attribution word, no document cited and no primary-record link.

How each party is described

Sentences that mention only one side are counted for that side. The Democratic side is Democratic Party, Democrats, Democratic, Democrat, Schumer, Jeffries, Pelosi, Newsom, Sanders, Ocasio-Cortez, Walz, Harris; the Republican side is Republican Party, Republicans, Republican, Thune, Johnson, McConnell, DeSantis, Cruz, Greene, Vance, Trump. These are the same names the name-swap test exchanges. For each side the site counts passive voice, passive voice with no doer named, loaded attribution verbs (claimed, admitted, insisted and others), hedges (reportedly, allegedly, so-called and others), scare quotes, and titles before a name; plus the share of quoted words, and which side's speaker is quoted first and last.

asymmetry = (pD − pR) ÷ (pD + pR), with p = (count + 1) ÷ (sentences + 2)

Shown only for an event or an outlet, never a single report, only when each side has at least 10 such sentences, and for each measure only when the two sides together have 3 or more cases (with fewer, the +1 in the formula alone would show a lean). Swapping the names must swap the two sides' counts exactly, so every asymmetry changes sign; the audit checks this on every report.

Checked against the official record

Vote counts in a sentence about a vote (such as "passed 61 to 39" in a sentence naming the Senate) are checked against the Senate's roll-call list [51] from four days before to one day after publication: "matches" names the roll call; otherwise the count "does not match the record" for that span. House counts are not checked yet. Bill numbers (H.R. 1234, S. 567) are looked up on Congress.gov [52] and shown with the bill's title and latest action.

How words are added to the lists

People decide what goes on the lists. Candidates come from three places: readers (every report page has a "Report a missed word" link), a word count over the kept text, and, at times, suggestions from an AI model. An AI model never scores a report and never adds a word.

  • Word count over the kept text: words that appear far more often in opinion pieces than in news reports, measured by the weighted log-odds method used to find the words that separate political speech [53]. Words already on the lists are skipped.
  • Review: each candidate is checked against the newsroom rules and the research behind the lists, given a severity by the same evidence rules as every other word, and tested on both sides' vocabulary. Partisan terms are added only with a mirror.
  • Record: every addition is published on What changed with a new method version and word-list fingerprint, and reports are measured again.

The research behind the temperature words

Word groupEvidenceStudies and strength
Combat and impact verbs used for criticism or debateindirect[22] moderate
[11] moderate
Anger and outrage wordscategory[23] strong
[24] moderate
[25] moderate
Catastrophe and disorder wordscategory[26] strong
[27] strong
[28] strong
[29] moderate
War and conflict metaphorscategory[30] mixed
[31] moderate
Mockery and humiliationindirect[32] strong
[23] strong
Delegitimizing labels and accusations in reporter voiceindirect
moved, see evidence page		[33] moderate
[7] strong
[6] moderate
Mental-illness insultsindirect
moved, see evidence page		[34] moderate
Shock and surprise boostersindirect[12] moderate
[25] moderate
Degree and size intensifierscategory[35] mixed
[6] moderate
Rebuke and setback wordsnoneNone found yet
Softening words (euphemism and agent removal)direct[36] moderate
[37] moderate
[38] moderate
[39] strong
[6] moderate
Insults and name-callingcategory[40] moderate
[41] moderate
[42] moderate
[43] moderate
[44] strong
[45] moderate
[46] moderate
[47] moderate
[48] moderate
[49] moderate
[50] moderate

Partisan-coded terms

Listed as mirror pairs with equal severity (3 each) and the same context rules.

More common on the rightMirror more common on the left
radical leftradical right
far-leftfar-right
extreme leftextreme right
leftistsright-wingers
woke mobMAGA mob
ultra-liberalultra-MAGA
pro-abortionanti-choice
death taxtax breaks for the rich
illegal aliensnativists
gun-grabbersgun nuts
open-bordersanti-immigrant
job-killingcorporate greed
Democrat PartyGQP
baby killersforced-birth
climate alarmistsclimate deniers

Name-swap pairs

Used by the name-swap test on the audit page: each report is scored again with these swapped, and a fair score should not move.

Democratic Party ↔ Republican PartyDemocrats ↔ RepublicansDemocratic ↔ RepublicanDemocrat ↔ Republicanliberals ↔ conservativesliberal ↔ conservativeprogressives ↔ conservativesSchumer ↔ ThuneJeffries ↔ JohnsonPelosi ↔ McConnellNewsom ↔ DeSantisSanders ↔ CruzOcasio-Cortez ↔ GreeneWalz ↔ Vanceleft-wing ↔ right-wingHarris ↔ TrumpMSNBC ↔ Fox NewsCenter for American Progress ↔ Heritage Foundationthe Squad ↔ the Freedom CaucusTransparency Party ↔ Libertarian Party

Word lists

The number is the effective severity in points; a listed severity reduced by its evidence cap shows in brackets; "exp." marks experimental words, shown but not counted.

Puffery (12)

Wikipedia "Words to watch": words that praise rather than describe. Source

legendary 1iconic 1visionary 1world-class 1groundbreaking 1renowned 1beloved 1acclaimed 1prestigious 1game-changing 1trailblazing 1storied 1

Contentious label (36)

Wikipedia "Words to watch": value-laden labels need attribution. AP and the BBC use "terrorist" only in quotes or with attribution. Source

terrorist 3terrorists 3extremist 3extremists 3cult 3racist 3racists 3sexist 3bigot 3bigots 3bigoted 3fascist 3fascists 3thug 3thugs 3mob 3regime 3propaganda 3conspiracy theorist 3conspiracy theorists 3controversial 1notorious 1infamous 1embattled 1beleaguered 1disgraced 1witch hunt 2sham 2hoax 2cover-up 2coverup 2rigged 2weaponized 2weaponization 2tyranny 2tyrannical 2

Expression of doubt (6)

Reuters Handbook: "claimed" can suggest doubt. Wikipedia "Words to watch": "so-called", scare quotes and similar words cast doubt. Source

so-called 2purported 2purportedly 2supposedly 2dubious 2claimed 1

Editorializing (57)

Wikipedia "Words to watch": words such as "clearly" and "unfortunately" insert the writer's view. Reuters Handbook: keep judgment out of news copy. Source

notably 1arguably 1essentially 1actually 1utterly 1absolutely 1without a doubt 1happily 1tragically 1aptly 1regrettably 1clearly 2obviously 2of course 2unfortunately 2fortunately 2sadly 2interestingly 2ironically 2remarkably 2needless to say 2undoubtedly 2it should be noted 2shameful 2disgrace 2disgraceful 2disgusting 2pathetic 2absurd 2ridiculous 2outrageous 2insane 2crazy 2idiotic 2stupid 2dumb 2genius 2brilliant 2heroic 2courageous 2cowardly 2corrupt 2evil 2disastrous 2reckless 2disturbing 2alarming 2troubling 2embarrassing 2laughable 2pitiful 2despicable 2vile 2wonderful 2amazing 2incredible 2unbelievable 2

Loaded word for "said" (20)

Wikipedia "Words to watch": "said" is neutral; "admitted", "insisted" and similar verbs imply a judgment. Reuters Handbook: verbs like "refute" imply editorial judgment. Source

admitted 2admits 2conceded 2concedes 2insisted 2insists 2refused to say 2failed to say 2boasted 2boasts 2bragged 2brags 2fumed 2fumes 2seethed 2lectured 2ranted 2rants 2whined 2refuted 2

Insult or name-calling (23)

A report describes people; an insult in the reporter's own voice is a judgment. Experimental: shown on every report but not scored until the rating panel has reviewed it. Research on incivility in news is on the evidence page. Source

moron exp.morons exp.idiot exp.idiots exp.liar exp.liars exp.loser exp.losers exp.clown exp.clowns exp.buffoon exp.buffoons exp.crook exp.crooks exp.traitor exp.traitors exp.scum exp.insanity exp.lunacy exp.lunatic exp.lunatics exp.unhinged exp.deranged exp.

Raises the temperature (94)

Reuters Handbook: use neutral language. Wikipedia "Words to watch": avoid dramatic wording a plain description does not need. Published studies on how this kind of wording affects readers are on the evidence page. Source

slams 1 (2)slammed 1 (2)blasts 1 (2)blasted 1 (2)rips 1 (2)ripped 1 (2)torches 1 (2)torched 1 (2)unloads 1 (2)lashes out 1 (2)lashed out 1 (2)mocks 1 (2)mocked 1 (2)smacks 1 (2)smacked 1 (2)dunks 1 (2)dunked 1 (2)scrambles exp.scrambling exp.hammers 1 (2)hammered 1 (2)erupts 2erupted 2destroys 1 (3)destroyed 1 (3)obliterates 1 (3)obliterated 1 (3)eviscerates 1 (3)eviscerated 1 (3)demolishes 1 (3)demolished 1 (3)shreds 1 (3)shredded 1 (3)bombshell 2 (3)meltdown 2 (3)firestorm 2 (3)frenzy 2 (3)fury 2 (3)furious 2 (3)outrage 2 (3)outraged 2 (3)rage 2 (3)raging 2 (3)war on 2 (3)all-out war 2 (3)crusade 2 (3)onslaught 2 (3)implodes 2 (3)imploded 2 (3)humiliating 1 (3)humiliation 1 (3)humiliates 1 (3)jaw-dropping 1 (3)chaos 2 (3)chaotic 2 (3)lawless 2lawlessness 2nightmare 2out of control 2stunning 1shocking 1explosive 1scathing 1brutal 1devastating 1catastrophe 1catastrophic 1panic 1epic 1very 1extremely 1totally 1truly 1deeply 1hugely 1massive 1massively 1huge 1enormous 1staggering 1whopping 1sweeping 1dramatic 1dramatically 1fierce 1fiercely 1blistering 1searing 1rebuke exp.rebuked exp.rebukes exp.backlash 1major blow exp.blow to exp.

Softens what happened (22)

Wikipedia "Words to watch": euphemisms blur what happened or who did it. Published studies on how softening and agent-free wording affect blame are on the evidence page. Source

officer-involved 2collateral damage 2misspoke 2misspeaks 2enhanced interrogation 2kinetic action 2mistakes were made 2controversial remarks 2misstep 2missteps 2gaffe 2gaffes 2downsizing 2right-sizing 2revenue enhancement 2neutralized 2friendly fire 2raised eyebrows 2raises eyebrows 2sparked debate 2sparks debate 2pushback 2

Unnamed source (16)

Reuters Handbook: anonymous sources are the weakest sources. SPJ Code: reserve anonymity as a last resort and explain why.

sources say 2sources said 2source says 2source said 2according to sources 2a source familiar 2people familiar 2person familiar 2familiar with the matter 2familiar with the discussions 2insiders 2an insider 2reportedly 2it is said 2one official said 2officials said privately 2

Vague attribution (12)

Wikipedia "Words to watch": "experts say" and "critics say" claim support without naming it.

experts say 2experts said 2critics say 2critics said 2some say 2many say 2many are saying 2people are saying 2observers say 2some argue 2many believe 2it is widely believed 2

Known limits

Copyright

The site shows each outlet's headline, a link to the original story, and a few words around each flagged word so readers can judge its context. It publishes counts and matched words, not article text. The text of pages read with the outlet's permission can be kept privately, only to measure again after a rule change and to settle disputes, and is deleted at the outlet's request. Keeping is not switched on yet.

References

  1. Koster, M., Illyes, G., Zeller, H., & Sassman, L. (2022). Robots Exclusion Protocol. RFC 9309, Internet Engineering Task Force. Source
  2. Reuters. Handbook of Journalism: standards and values; neutral language, and anonymous sources as the weakest sources. Source
  3. Wikipedia. Manual of Style: Words to watch. Guidance on puffery, contentious labels, expressions of doubt, editorializing, loaded words for "said", unsupported attributions and euphemisms. Source
  4. Associated Press. AP Stylebook: anonymous sources (explain why a source is granted anonymity) and the dateline credit line for AP copy. Source
  5. Society of Professional Journalists. (2014). SPJ Code of Ethics. "Label advocacy and commentary." "Consider sources' motives before promising anonymity. Reserve anonymity for sources who may face danger, retribution or other harm." Source
  6. Recasens, M., Danescu-Niculescu-Mizil, C., & Jurafsky, D. (2013). Linguistic models for analyzing and detecting biased language. Proceedings of ACL 2013, 1650–1659. Source
  7. Gentzkow, M., & Shapiro, J. M. (2010). What drives media slant? Evidence from U.S. daily newspapers. Econometrica, 78(1), 35–71. Source
  8. Chapman, W. W., Bridewell, W., Hanbury, P., Cooper, G. F., & Buchanan, B. G. (2001). A simple algorithm for identifying negated findings and diseases in discharge summaries. Journal of Biomedical Informatics, 34(5), 301–310. The short negation window used here follows this approach. Source
  9. Nardo, M., Saisana, M., Saltelli, A., Tarantola, S., Hoffman, A., & Giovannini, E. (2008). Handbook on Constructing Composite Indicators: Methodology and User Guide. OECD and European Commission Joint Research Centre. Source
  10. Federal Register (Office of the Federal Register, National Archives). Document abstracts used as the plain reference writing. Source
  11. Ecker, U. K. H., Lewandowsky, S., Chang, E. P., & Pillai, R. (2014). The effects of subtle misinformation in news headlines. Journal of Experimental Psychology: Applied, 20, 323–335. Source
  12. Molyneux, L., & Coddington, M. (2020). Aggregation, clickbait and their effect on perceptions of journalistic credibility and quality. Journalism Practice, 14(4), 429–446. Source
  13. United Nations Development Programme. Human Development Report, Technical notes. Since 2010 the Human Development Index combines its dimensions by geometric mean. Source
  14. The shifted geometric mean (adding a constant before taking the geometric mean, then removing it) is a standard way to keep a geometric mean stable when values can be zero. See, for example, Achterberg, T. (2007). Constraint Integer Programming (PhD thesis, TU Berlin), on shifted geometric means for comparing results.
  15. Efron, B., & Morris, C. (1975). Data analysis using Stein's estimator and its generalizations. Journal of the American Statistical Association, 70(350), 311–319. Source
  16. Student [Gosset, W. S.] (1908). The probable error of a mean. Biometrika, 6(1), 1–25. Source
  17. Efron, B. (1979). Bootstrap methods: Another look at the jackknife. The Annals of Statistics, 7(1), 1–26. Source
  18. Internet Archive. Wayback Machine and Save Page Now. Source
  19. Hayes, A. F., & Krippendorff, K. (2007). Answering the call for a standard reliability measure for coding data. Communication Methods and Measures, 1(1), 77–89. Source
  20. Holland, P. W., & Thayer, D. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer & H. I. Braun (Eds.), Test Validity (pp. 129–145). Lawrence Erlbaum.
  21. Spearman, C. (1904). The proof and measurement of association between two things. The American Journal of Psychology, 15(1), 72–101. Source
  22. Loftus, E. F., & Palmer, J. C. (1974). Reconstruction of automobile destruction: An example of the interaction between language and memory. Journal of Verbal Learning and Verbal Behavior, 13(5), 585–589. Source
  23. Brady, W. J., Rathje, S., Globig, L., & Van Bavel, J. J. (2025). Estimating the effect size of moral contagion in online networks: A pre-registered replication and meta-analysis. PNAS Nexus. Source
  24. Brady, W. J., McLoughlin, K., Doan, T. N., & Crockett, M. J. (2021). How social learning amplifies moral outrage expression in online social networks. Science Advances, eabe5641. Source
  25. Berger, J., & Milkman, K. L. (2012). What makes online content viral? Journal of Marketing Research. Source
  26. Robertson, C. E., Pröllochs, N., Schwarzenegger, K., Pärnamets, P., Van Bavel, J. J., & Feuerriegel, S. (2023). Negativity drives online news consumption. Nature Human Behaviour, 7, 812–822. Source
  27. Soroka, S., Fournier, P., & Nir, L. (2019). Cross-national evidence of a negativity bias in psychophysiological reactions to news. PNAS, 116(38), 18888–18892. Source
  28. Tannenbaum, M. B., Hepler, J., Zimmerman, R. S., Saul, L., Jacobs, S., Wilson, K., & Albarracín, D. (2015). Appealing to fear: A meta-analysis of fear appeal effectiveness and theories. Psychological Bulletin, 141(6), 1178–1204. Source
  29. McLeod, D. M., & Detenber, B. H. (1999). Framing effects of television news coverage of social protest. Journal of Communication, 49(3), 3–23. Source
  30. Thibodeau, P. H., & Boroditsky, L. (2011). Metaphors we think with: The role of metaphor in reasoning. PLoS ONE, 6(2), e16782. Source
  31. Hauser, D. J., & Schwarz, N. (2015). The war on prevention: Bellicose cancer metaphors hurt (some) prevention intentions. Personality and Social Psychology Bulletin, 41(1), 66–77. Source
  32. Rathje, S., Van Bavel, J. J., & van der Linden, S. (2021). Out-group animosity drives engagement on social media. PNAS, e2024292118. Source
  33. Loftus, E. F. (1975). Leading questions and the eyewitness report. Cognitive Psychology, 7, 560–572. Source
  34. Utych, S. M. (2018). How dehumanization influences attitudes toward immigrants. Political Research Quarterly, 71(2), 440–452. Source
  35. Hosman, L. A. (1989). The evaluative consequences of hedges, hesitations, and intensifiers: Powerful and powerless speech styles. Human Communication Research, 15(3), 383–406. Source
  36. Walker, A. C., Turpin, M. H., Meyers, E. A., Stolz, J. A., Fugelsang, J. A., & Koehler, D. J. (2021). Controlling the narrative: Euphemistic language affects judgments of actions while avoiding perceptions of dishonesty. Cognition, 104633. Source
  37. Moreno-Medina, J., Ouss, A., Bayer, P., & Ba, B. A. (2025). Officer-involved: The media language of police killings. Quarterly Journal of Economics, 140(2), 1525–1580. Source
  38. Fausey, C. M., & Boroditsky, L. (2010). Subtle linguistic cues influence perceived blame and financial liability. Psychonomic Bulletin & Review, 17(5), 644–650. Source
  39. Levin, I. P., & Gaeth, G. J. (1988). How consumers are affected by the framing of attribute information before and after consuming the product. Journal of Consumer Research, 15(3), 374–378. Source
  40. Mutz, D. C., & Reeves, B. (2005). The new videomalaise: Effects of televised incivility on political trust. American Political Science Review, 99(1), 1–15. Source
  41. Anderson, A. A., Brossard, D., Scheufele, D. A., Xenos, M. A., & Ladwig, P. (2014). The "nasty effect:" Online incivility and risk perceptions of emerging technologies. Journal of Computer-Mediated Communication, 19(3), 373–387. Source
  42. Otto, L. P., Lecheler, S., & Schuck, A. R. T. (2020). Is context the key? The (non-)differential effects of mediated incivility in three European countries. Political Communication, 37(1), 88–107. Source
  43. Gervais, B. T. (2015). Incivility online: Affective and behavioral reactions to uncivil political posts in a web-based experiment. Journal of Information Technology & Politics, 12(2), 167–185. Source
  44. Van't Riet, J., & Van Stekelenburg, A. (2022). The effects of political incivility on political trust and political participation: A meta-analysis of experimental research. Human Communication Research, 48(2), 203–229. Source
  45. Muddiman, A. (2017). Personal and public levels of political incivility. International Journal of Communication, 11, 3182–3202. Source
  46. Muddiman, A., Pond-Cobb, J., & Matson, J. E. (2020). Negativity bias or backlash: Interaction with civil and uncivil online political news content. Communication Research, 47(6), 815–837. Source
  47. Muddiman, A., & Scacco, J. M. (2024). The influence of conflict news on audience digital engagement. Journalism Studies, 25(3), 278–298. Source
  48. Thorson, K., Vraga, E., & Ekdale, B. (2010). Credibility in context: How uncivil online commentary affects news credibility. Mass Communication and Society, 13(3), 289–313. Source
  49. Skytte, R. (2021). Dimensions of elite partisan polarization: Disentangling the effects of incivility and issue polarization. British Journal of Political Science, 51(4), 1457–1475. Source
  50. Bøggild, T., & Jensen, C. (2024). When politicians behave badly: Political, democratic, and social consequences of political incivility. American Journal of Political Science. Advance online publication. Source
  51. United States Senate. Roll call votes, vote list for each session of Congress. Source
  52. Library of Congress. Congress.gov API, bill records. Source
  53. Monroe, B. L., Colaresi, M. P., & Quinn, K. M. (2008). Fightin' words: Lexical feature selection and evaluation for identifying the content of political conflict. Political Analysis, 16(4), 372–403. Source