Audit · Transparency News

Fairness

Partisan list balance Pass

Severity points on each side of the partisan list match within 5%. Right-coded: 15 terms, 45 points. Left-coded: 15 terms, 45 points. Difference 0.0%.

Covers: method 3.1.0, the published word lists. Run by: this site, automatically, at each update; no independent reviewer yet. If it fails: The list is rebalanced before the next version.

Mirror pairs Pass

Every left-coded partisan term has a right-coded mirror with equal severity and the same context rules, so equivalent usage gets equal points. 15 pairs, 0 missing a mirror.

Covers: method 3.1.0, the published word lists. Run by: this site, automatically, at each update; no independent reviewer yet. If it fails: The unpaired term is removed until a mirror is found.

Name swap Does not pass

Average score change when party, politician, institution and political-adjective names are swapped is 0.5 points or less, and no single report moves more than 2 points outside published exceptions. The swap includes "Transparency Party" and "Libertarian Party". Across 10961 reports, the average change was 0.00 points and the largest 5.0.

Covers: method 3.1.0, 10961 reports. Run by: this site, automatically, at each update; no independent reviewer yet. If it fails: The rule that moved is found and fixed before the next version.

Published exceptions: "Democrat Party" and "ultra-liberal" are partisan-coded labels that contain a swapped word; swapping them to "Republican Party" and "ultra-conservative" removes a flag, which is the intended behavior. Reports that contain either label are left out of the test.

Party-description panel flips with the names Does not pass

Rule: when party and politician names are swapped, the counts for the two sides in the "how each party is described" panel must swap exactly, so every asymmetry changes sign. 7769 of 7788 reports that mention either side pass.

Covers: method 3.1.0, 7788 reports. Run by: this site, automatically, at each update; no independent reviewer yet. If it fails: The pattern that did not swap is fixed before the next version.

Coverage of the Transparency Party Waiting for data

Every saved report that mentions the Transparency Party is listed with its score and matched words, so readers can check that coverage of the founder's party is scored like everything else. No saved report mentions the Transparency Party yet.

Left and right opinion samples Diagnostic

Diagnostic, not a pass mark: the left-leaning and right-leaning opinion samples are shown side by side. Different samples can legitimately use listed words at different rates, so equal results are not the target. The test that matters is equal treatment of equal usage (mirror pairs and name swap). Word choice per 1,000 words: left 12.46, right 18.70. Temperature: left 7.83, right 5.50. Measured under method 2.1.0; recalibration under 3.0.0 is pending.

Remediation: under 2.1.0 this was a pass mark and the site failed it. Method 3.0.0 replaces it as a diagnostic, because equal results between different samples is not the goal; equal treatment of equal usage is tested by the mirror-pair and name-swap tests above.

Weight sensitivity Pass

Outlet rankings under alternative weights correlate with the default at 0.9 or higher (Spearman).

Variant	Outlets	Correlation with default
Default (geometric mean, published weights)	16	1.00
Equal weights (geometric mean)	16	0.99
Published weights (simple average)	16	1.00
Word choice only	16	0.99

Covers: method 3.1.0, reports saved in the live window. Run by: this site, automatically, at each update; no independent reviewer yet. If it fails: The weights are reviewed and the result published.

How rankings moved with this method change

Every method change publishes how outlet rankings moved: the rank correlation of outlet scores under the old and new method on the same reports. From 3.0.0 to 3.1.0: rank correlation 0.93 on the same reports.

Outlet	Plain average under 3.0.0	Under 3.1.0
NPR	99	98
PBS NewsHour	96	93
BBC News	97	94
The New York Times	96	89
The Guardian	94	90
Axios	95
CBS News	96	93
ABC News	98	96
NBC News	97	95
HuffPost	88	86
Fox News	95	92
New York Post	91	89
Washington Examiner	96	93
The Daily Caller	94	91
Breitbart	90	86
The Federalist	90	81
Vox	94	86

Does it measure what it says?

Human panel

Human panel agreement (Krippendorff's alpha) of 0.80 or higher is firm; 0.667 to 0.80 is tentative; below 0.667 the word is reviewed. Agreement is reported per word group as well as overall.

0 ratings from 0 raters. Agreement overall: not enough ratings yet. Within groups: left not enough ratings yet, center not enough ratings yet, right not enough ratings yet. Between groups: not enough ratings yet.

Agreement per word group appears once the panel has rated passages. Until then every word's severity rests on the published research and newsroom rules, not on local validation. The panel is being recruited; panel results will also show how often raters overrule a match.

Stability on held-out events

Reports are split in two by event, so coverage of one event never sits in both halves. Results that hold in both halves are less likely to come from a few unusual events. This checks consistency, not accuracy.

Group	Development half	Held-out half
Outlets rated left-leaning	90.8 (928)	90.5 (1021)
Outlets rated center	94.7 (91)	93.6 (118)
Outlets rated right-leaning	89.4 (1817)	89.7 (2010)
Other or unlabeled	89.7 (1480)	89.5 (1648)
Politics	91.1 (1067)	91.1 (1186)
Courts and crime	91.9 (94)	94.1 (86)
Science, health and climate	94.9 (15)	92.8 (22)
Business and economy	90.8 (52)	89.3 (90)
World	82.7 (128)	84.3 (117)

Average news report score (reports in brackets). Sections are read from each report's link.

Event grouping

Checked against 200 hand-labeled pairs of headlines: of the pairs the method joins, 87% really are the same event; of the pairs that are the same event, the method joins 67%. Labels were made by Claude, an AI model, from the headlines alone, and are pending human review. The sample is half pairs the method joined and half near misses, so close calls are over-represented and these rates are not the rates across all coverage. Rule used for labeling: two headlines are the same event when they report the same specific happening within two days, such as the same debate, rally, court filing, announcement, or campaign ad and the immediate reaction to it. The labels.

Reader understanding

Readers can take a five-question check on what the score means. Results are published here so the site can tell whether its explanations work.

Question	Answers	Correct
A report scores 100. What does that mean?	0	n/a
Does the score check whether a report is true?	0	n/a
Are words inside quotation marks counted in the score?	0	n/a
A grey mark on a word means:	0	n/a
Two outlets have overlapping 90% ranges. What should you conclude?	0	n/a

Challenge set: cases built to break the method

Ten invented test stories, run through the same code that scores live reports. Known limits are listed as limits.

Case	What a good measure would do	Score	Result
A false story told calmly	A trustworthy tool would not rate this as reliable reporting, because the claim is invented.	100	Known limit: the story scores in the top band. The score measures wording, not truth, and says so beside every score.
An accurate disaster report with strong, literal words	Strong words that describe real damage literally should not cost points.	100	Handled: the context rules mark "devastating earthquake", "destroyed homes" and "catastrophic flooding" as literal, so they cost nothing.
An investigation with explained anonymous sources	Anonymous sources whose anonymity is explained, backed by documents, should cost little.	100	Handled: an unnamed source whose anonymity is explained costs nothing (it is still listed), and the documents count as attribution.
An unsupported story with no attribution at all	A story that names no source should not get a perfect sourcing result.	100	Handled: with no attribution found, the attribution layer is marked "not established" and left out, instead of scoring 100.
A partisan story carried entirely by quotes	Quote choice is an editorial decision, so the loaded quotes should be visible even if not scored.	100	Partly handled: quotes are not scored (the reporter is quoting accurately), but every loaded word inside quotes is counted and shown beside the score, with the share of the story in quotes.
A misleading headline with no loaded words	A headline that claims more than the story supports should be flagged.	100	Known limit: the headline uses no listed word, so the headline score is 100. The experimental check of names and figures in the headline cannot catch a wrong verb.
Loaded words diluted with neutral padding	The same loaded sentence should cost the same however long the story is.	84	Known limit: scores are a rate per 1,000 words, so padding a story with neutral text lowers the cost of the same loaded sentence. This is a deliberate choice (long reports are not penalized for length), and each report page lists every matched word regardless of length.
Loaded meaning in words that are not on the lists	Loaded meaning should be caught whatever words carry it.	100	Known limit: "fumbles", "awkward", "baffled" and "exasperated" are not on the lists, so the loaded meaning is missed. Readers can propose additions on the corrections page.
A story about a loaded phrase, not using it	Discussing a term is not using it.	100	Handled: "the phrase" and "the term" before a listed word mark it as discussed, so it costs nothing.
The same wire story run by two outlets	Wire copy should be recognized and counted once, not as each outlet's own writing.	100	Handled: the AP credit is detected, and wire copy is left out of each outlet's own score.

6 of 10 handled; the rest are published limits.