Beyond Aggregate Sentiment — reproducibility repository

Code, taxonomy, and derived results for "Beyond Aggregate Sentiment: Machine Learning-Driven Discourse Indicators for AI News at Scale" (MDPI AI). Six interpretable discourse indicators — Valence, Loss-Salience (LSI), Narrative Drift (NDI), Exposure-Adjusted Sentiment (EASI), Cross-Source Divergence (CSDI), Novelty-Phase Framing (NPF) — are extracted from AI-news headlines, validated against GPT-4o and a three-annotator human-annotated benchmark, and mapped to EU governance instruments.

Layout (scripts use paths relative to this repository root)

code/                    analysis pipeline; run with PYTHONPATH=code from the repo root
  build_corpus.py        ⭐ one-command build: concept links + sentiment → analysis set
  indicators.py          ⭐ the six indicators (canonical)
  validation/            DistilBERT↔GPT-4o validation (step1–step6)
  run_settings.json, gpt4o_prompt.txt
unique_wikitopics.csv    the AI Innovation Taxonomy (425 concepts)   [included]
data/
  concept_links.parquet             doc_id → primary AI concept (+score, duplicate flag)  [included]
  concept_links_multilabel.parquet  doc_id → all AI concepts (robustness variant)          [included]
  sample_news_with_sentiment_final.csv  25-row schema example (headline text withheld)     [included]
  README_DATA.md                    provenance + how to obtain the restricted corpus
outputs/                 scripts read/write here
  validation_tables/     validation-sample results (cross-model, 3-class, confusion, gold)   [included]
  tables_fullperiod/     full-period indicator results (671,913 headlines, 386 topics,
                         July 2022–July 2024)                                                [included]
  figures_fullperiod/    paper figures                                                       [included]
  gold/                  three-annotator human benchmark + adjudicated labels                [included]
  cache/                 analysis-set parquet (built locally by build_corpus.py)   [regenerated — gitignored]
environment.txt          pinned package versions
LICENSE                  MIT (code)

outputs/validation_tables*, outputs/figures_fullperiod, and outputs/gold are pre-populated with the derived results so they can be inspected without re-running. Re-running overwrites them. The concept-link tables are included (they carry no article text), so the only input you must supply is the headline-sentiment table (see below).

Requirements

Python 3.9+ with pandas, numpy, scipy, scikit-learn, pyarrow, matplotlib, statsmodels (+ torch, transformers for the three-class / news-model baselines). See environment.txt.

Data you must supply (Event Registry — not redistributable; see data/README_DATA.md)

Concept links are already included, so the pipeline needs one user-supplied input — the headline-sentiment table, whose article text is subject to Event Registry's terms:

Path (relative to repo root)	Contents / source
news_with_sentiment.csv	full-corpus headlines + DistilBERT sentiment (doc_id, lang, doc_title, src, doc_date, sentiment_label, sentiment_score)
outputs/validation_results.csv	1,000-headline cross-model sample (only for verify_anchors.py Anchor 2 and the validation scripts)

outputs/gold/gold_sheet_adjudicated.csv (the three-annotator human benchmark) is already included.

How to run (from the repo root)

export PYTHONPATH=code            # so scripts can import indicators.py

# 1. Build the analysis set from the included concept links + your sentiment table.
#    → outputs/cache/topic_linked_fullperiod.parquet         (1,996,429 docs → 671,913 analysis, 386 topics)
#    → …_withsyndication.parquet + outputs/tables_fullperiod/coverage_fullperiod.csv (robustness/coverage)
python code/build_corpus.py

# 2. Sanity gate — reproduces the published anchors, stops on mismatch.
#    → Facial_recognition_system LSI 1.75 · Deepfake LSI 3.17 · cross-model rho 0.83
python code/verify_anchors.py

# 3. Indicator tables + paper figures (read outputs/cache/…; env vars override defaults)
TL_PARQUET=outputs/cache/topic_linked_fullperiod.parquet OUT_DIR=outputs/tables_fullperiod \
  FIG_DIR=outputs/figures_fullperiod \
  python code/task3_sensitivity.py     # …and task4_easi_robust.py / task7_ndi_groups.py
python code/rebuild_paper_figures.py ; python code/csdi_figure.py ; python code/easi_scatter.py
python code/lsi_csdi_map.py ; python code/regen_ranked_tables.py ; python code/fullperiod_report.py

# 4. Cross-model / classifier validation (needs outputs/validation_results.csv)
python code/task2_cross_model.py ; python code/task5_medium_band.py ; python code/confusion_by_domain.py
python code/task_threeclass_baseline.py ; python code/task_news_baselines.py

# 5. Human benchmark metrics (uses the included three-annotator gold)
python code/finalize_gold.py outputs/gold/gold_sheet_adjudicated.csv

The concept links in data/ were derived from the raw Event Registry exports using the Wikifier + AI Innovation Taxonomy procedure described in data/README_DATA.md (and the From Code to Concept prior work); they are provided ready-made so the pipeline runs from the single sentiment table above.

Canonical settings

Sampling seed 42; GPT-4o gpt-4o, temperature = 0, top_p/seed unset (code/run_settings.json, code/gpt4o_prompt.txt). "Domain" = a Wikipedia topic; all six indicators are topic-level.

License / citation

Code is released under the MIT License (LICENSE). The derived result tables and figures under outputs/ are made available under CC BY 4.0, consistent with the article's open-access license. The concept-link tables in data/ are derived from Event Registry concept annotations and contain no article text. Concept linking uses the Wikifier + AI Innovation Taxonomy approach validated in prior work (From Code to Concept). If you use this repository, please cite the paper.