This is a partial, UK-focused directory of databases, investigative tools, mapping frameworks, scraping scripts, and data journalism inspiration, created and curated by Matilda Davies, data journalist at The Times & The Sunday Times.
- 🏛️ Politics & Government
- 🏥 Health & Population
- 👮 Policing & Law
- 🚇 Transport
- 💼 Business & Finance
- 🎓 Education
- 🌿 Environment & Energy
- 🌍 Global & Cross-Border
- 🇺🇸 US-Specific Portals
- 🗺️ Global Statistical Offices (Country Data Hubs)
- 🔧 Data & OSINT Tools
- 📈 Web Scraping & Chrome Extensions
- 🔌 DataWrapper & Visualisation Hacks
- 🛰️ Maps & Satellite Imagery
- 📊 R Tools & Tricks
- ✨ Graphics & Visual Journalism Inspiration
Open data, policy, and public spending across UK parliamentary bodies:
-
Gov.uk — Primary portal for published UK public data.
-
ONS — The UK's largest independent producer of official statistics and national statistical institute.
-
Census — Comprehensive regional demographic statistics.
-
Votes in parliament — Visual mapping of votes and divisions in Westminster.
-
HoL Register of Interests — Up-to-date registry mapping peers' business interests.
-
WhatDoTheyKnow — Searchable index of public freedom of information (FOI) requests.
-
TheyWorkForYou — Parliamentary monitoring tool identifying representative voting histories and debate statements.
-
Parli-N-Grams — Visual n-gram generator mapping keyword frequencies in parliamentary debates.
-
Government Tracker – Full Fact — Independent metrics tracing administration pledges and policies.
-
Ofcom — Media, signal coverage, and telecommunication databases.
-
Council Meeting Transcript Search (Bellingcat) — Multi-district audio/text search for local government meetings.
-
Council contracts — Public procurement registers.
Public healthcare analytics, medical evaluations, and regional well-being metrics:
-
NHS Digital — Statistical bulletins, local dashboards, and open clinical databases.
-
NHS England — Detailed clinical metrics, hospital waiting lists, and performance updates.
-
NHS Staff Surveys — Workplace climate evaluations.
-
NHS BSA — High-level prescription databases and operational reporting.
-
NHS Resolution — Official data tracking clinical litigation, negligence claims, and dispute outcomes.
-
CQC — Care Quality Commission assessment data covering adult social care, clinics, and hospitals.
-
OpenPrescribing — Powerful exploration tool querying public GP prescribing behaviors.
-
MHRA Yellow Card scheme — Public drug analysis profiles logging suspected vaccine and pharmaceutical side effects.
-
HERC: Population Health (Uni of Ox) — Curated compilation of open health economics datasets.
-
Family food datasets — Defra's annual estimates of household food, nutrient, and drink intake.
-
Food Standards Agency — Regional hygiene rating files covering restaurants, takeaways, and food shops.
Crime metrics, tribunal files, and operational courts records:
-
Police — Comprehensive regional crime, stop-and-search, and officer-use-of-force database.
-
Prisons data — Key performance tables, staffing ratios, and inmate data for England and Wales.
-
Criminal court statistics — Ministry of Justice reporting on court backlogs and sentencing trends.
-
Courtserve.net — Daily listing dockets from Royal Courts of Justice, Crown, and County courts.
-
Judgments Archive — Comprehensive repository of legal findings and judicial rulings.
Rail, civil aviation, and public transit operations data:
-
Civil Aviation Authority — UK flight data, aviation emissions, and carrier punctuality.
-
Office of Rail & Road — Financial trackers, passenger metrics, and freight safety databases.
-
TfL — Multi-modal transit feeds covering London's open bus, tube, and cycling grids.
Market intelligence, macroeconomics, corporate registries, and labor trends:
-
AnnualReports.com — Global library of corporate annual reports.
-
Bank of England — UK monetary metrics, interest rates, and banking balance sheets.
-
Charity Commission — Non-profit registrations, financial declarations, and trustee logs.
-
360Giving GrantNav — Consolidated engine mapping grantmaker data and charitable donations.
-
Companies House — Free official corporate registry detailing directors, filing history, and accounts.
-
DWP Stat-Xplore — Database detailing housing benefits, universal credit, and pension claims.
-
Financial Conduct Authority — Market compliance and regulatory enforcement data.
-
Nomis — Official labour market statistics, local profiles, and employment metrics.
-
OBR — Economic forecasts, debt assessments, and fiscal reviews.
-
UK Finance — Detailed private banking, mortgage, and payment transaction databases.
-
Employment tribunal decisions — Searchable records of employment disputes and rulings.
-
Indeed Hiring Lab — Open-source data repository mapping market job post indices.
-
Tax rates - country comparison — PwC interactive grid comparing worldwide individual income brackets.
University admissions, regional school performance, and education stats:
-
Dept for Education — Comprehensive hub detailing GCSE/A-level outcomes, exclusions, and funding.
-
Ofqual — Statistics on qualification entries, grading variations, and vocational outcomes.
-
Schools — UK national school register, contact endpoints, and operational status tracker.
-
UCAS — University application trends, placement rates, and applicant demographics.
-
Office for Students — Student outcomes, financial health of universities, and access indices.
Climate tracking, sewage overflows, pollution files, and housing energy efficiency:
-
Air quality & pollution — Live sensor grids tracking air quality indices across the UK.
-
Defra / Environment — Key portal detailing river health, soil chemistry, and marine databases.
-
Energy Performance of Buildings Data England and Wales — Searchable bulk index of domestic/commercial EPC certificates.
-
Ofgem — Interactive data charts covering UK wholesale gas and electric grids.
-
HM Land Registry Open Data — High-level residential transaction and price-paid spatial files.
-
MCS Heat Pumps — Interactive tracker measuring renewable installations.
-
Global Fishing Watch — Live open-access mapping tracker plotting marine traffic and commercial vessels.
-
CITES Illegal Wildlife Trade — Database tracking illicit wild fauna and flora shipments.
-
EA Enforcement Actions — Public log of enforcement citations issued by the Environment Agency.
-
European Environment Agency — Environmental metrics covering 38 European partner states.
-
Rivers Trust: Sewage map — Interactive mapping showing sewage dump duration and locations.
-
Sewage Map — Alternative tracker providing real-time storm overflow alerts.
-
Defra Magic Map — Complex interactive maps detailing environmental and agricultural land designations.
-
UK and regional series — Monthly and annual climate time-series datasets based on the HadUK-Grid.
International trade, immigration patterns, macroeconomics, and ad analytics:
-
Eurostat — Official statistics agency of the European Union.
-
Google Finance — Real-time international market data and financial news.
-
IMF — International Monetary Fund data portals on macroeconomic trends.
-
Inside AirBnB — Unofficial, parsed dataset monitoring Airbnb's community impact across world cities.
-
International Energy Agency — Energy security, supply, and transition statistics.
-
OECD — Highly curated economic and social development database pages.
-
Our World in Data — Research and data tracking complex global issues over long timelines.
-
UN Comtrade — Global bilateral trade statistics repository.
-
UNdata — Single entry point for diverse UN system databases.
-
Waste Data Flow — System used by UK local authorities to monitor municipal waste collection.
-
World Bank — Central catalog of open development indicators, microdata, and financials.
-
Frontex - Migratory routes — Visualizations of irregular border crossings into the EU.
-
Gallup polling — Datasets and world poll statistics made available for public analysis.
-
Meta ad library — Publicly searchable political, election, and social issue ads on Facebook and Instagram.
-
NerdyData.com — Search engine for finding specific source code, HTML tags, or tracking scripts on domains.
-
εxodus — Privacy audit database mapping tracking and permission parameters in Android apps.
-
Telegram Threat Actor channels — Directory of Telegram groups used by cyber threat actors.
-
Public GIS Data Directory — Extensive catalog mapping public geospatial portals.
Electoral tracking, public finances, labor indexes, and federal databases:
-
Bureau of Labor Statistics — USA federal agency tracking inflation, employment, and labor economics.
-
Economic Research Service — USDA economic insights on food, farming, and agriculture.
-
Energy Information Administration — US official energy statistics and consumption trackers.
-
New York Fed — Financial indexes and regional monetary research.
-
St Louis Fed — Federal Reserve economic datasets (FRED) spanning thousands of variables.
-
Executive Orders — Database recording official Presidential declarations and executive orders.
-
KUOW/NPR: Trump lawsuit tracker — Tracker tracing lawsuits involving Donald Trump.
-
Education Stats — National Center for Education Statistics (NCES) database.
-
Bureau of Justice Statistics — Primary US federal agency responsible for reporting criminal justice metrics.
-
CDC — National Center for Health Statistics (NCHS) repository.
-
Deportation data — Curated databases on US immigration enforcement and deportation.
Quick-access portals to official national statistical databases:
- Full list — Comprehensive directory.
| Location | Agency Name | Direct Link |
|---|---|---|
| 🇫🇷 France | INSEE | insee.fr |
| 🇩🇪 Germany | Destatis | destatis.de |
| 🇮🇪 Ireland | CSO | cso.ie |
| 🇮🇹 Italy | Istat | istat.it |
| 🇳🇱 Netherlands | CBS | cbs.nl |
| 🇵🇱 Poland | GUS | stat.gov.pl |
| 🇪🇸 Spain | INE | ine.es |
| 🇺🇦 Ukraine | State Stats | stat.gov.ua |
| 🇦🇺 Australia | ABS | abs.gov.au |
| 🇨🇦 Canada | StatCan | statcan.gc.ca |
| 🇿🇦 South Africa | Stats SA | statssa.gov.za |
| 🇦🇷 Argentina | INDEC | indec.gob.ar |
| 🇧🇷 Brazil | IBGE | ibge.gov.br |
| 🇨🇳 China | NBS | stats.gov.cn |
| 🇭🇰 Hong Kong | C&SD | censtatd.gov.hk |
| 🇮🇳 India | MoSPI | mospi.gov.in |
| 🇯🇵 Japan | SBJ | stat.go.jp |
| 🇰🇷 South Korea | KOSTAT | kostat.go.kr |
Investigative engines, geolocation tools, and flight/marine trackers:
-
Mail merge — Google Apps Script for automated outreach templates.
-
Google Pinpoint — Tool for journalists to parse thousands of PDFs, handwritten notes, and audio files.
-
ClipShare — Archive for TV and radio clips.
-
DocumentCloud — Specialized document analysis, indexing, and embedding tool for journalists.
-
Face search • PimEyes — Face-recognition-based search engine.
-
SunCalc — Calculates sun movement, shadow length, and solar positions at specific coordinates and times.
-
MarineTraffic — Real-time tracking of global commercial vessel traffic.
-
FlightRadar24 — Real-time aircraft transponder flight map.
-
Freedar Flight Radar — Open flight tracker.
-
Telegram - Threat Actor channels — Threat intelligence channel mapping.
-
Nixintel's OSINT Resource List — Comprehensive open-source intelligence bookmark page.
-
Bellingcat toolkit — Essential toolkit for digital detectives and geospatial research.
-
OpenCorporates — Large open database of corporate registry filings.
-
ICIJ Offshore Leaks Database — Search engine mapping offshore assets revealed in corporate leaks.
-
PEP: Politically Exposed Persons of Russia and Belarus — Database tracking public figures, PEPs, and connections inside Russia/Belarus.
-
OCCRP Aleph — Freedom of information and leaked document data index from OCCRP.
Helper utilities to pull structured arrays and troubleshoot client pixels:
-
Bellingcat Filename Finder — Exposes exact filenames of photos uploaded to Google Maps.
-
Instant Data Scraper — Automatically extracts tables or lists from modern web pages to CSV/Excel.
-
SelectorGadget — Simple browser-based CSS selector generation tool for web scraping.
-
Meta Pixel Helper — Tool to audit pixel activity on websites.
-
TikTok Pixel Helper — Tool to troubleshoot advertiser pixel code.
Tips, templates, and libraries to create cleaner news graphics:
-
Charts in tooltips — Embed secondary visualizations inside a chart’s tooltip.
-
Customising tooltips — Customizing the style and structure of tooltip text.
-
E.g. Custom key — Example implementing custom annotations.
-
E.g. Image embed — Example showing inline image assets.
-
E.g. Nav - Buttons — Interactive navigation using buttons in charts.
-
E.g. Nav - Underline — Alternative underline styling for navigations.
-
Flag code lookup table — Directory connecting codes to nation flags.
-
HTML Unicode UTF-8 — Reference guide for using raw Unicode characters and symbols in text/charts.
-
Datawrapper tips (from UnWrapped) — Presentation slides explaining advanced tricks in Datawrapper.
-
FT Visual Vocabulary — Guide for designing visualizations based on reporting intent, matching the FT's legendary style guide.
Shapefile wrangling, geographic editors, and daily satellite capture portals:
-
mapshaper — Command line and UI tool to simplify, crop, merge, or convert geographic files (Shapefile, GeoJSON, TopoJSON).
-
MyGeodata Cloud — Web converter for transforming spatial databases between coordinate reference systems.
-
geojson.io — In-browser editor for drawing and editing GeoJSON shapes on a leaf map.
-
Geodojo.net — Open geographic conversion tools.
-
Google Earth — Standard 3D spatial visualizer.
-
NASA Worldview — Daily, full-resolution earth satellite images spanning environmental and wildfire tracking.
-
Copernicus Sentinel Satellite — Direct access portal for Copernicus imaging.
-
OpenStreetMap — Collaborative world map engine.
Web-scraping, Natural Language Processing, and plotting pipelines in R:
-
How to Use R in DataWrapper — Integrates R processing seamlessly with Datawrapper.
-
Web Scraping Using R and rvest — Complete guide on scraping HTML tables using
rvest. -
UDPipe Natural Language Processing — UDPipe implementation guide for advanced NLP inside R.
-
Analysing US census data — Book on querying the US Census Bureau via tidyverse wrappers.
-
BBC R cookbook — The BBC’s internal documentation detailing how they build chart graphics with R.
-
Introduction to R for journalists — Video and file series produced for the European Journalism Centre.
A curated showcase of world-class investigative reporting, visual desks, and interactive logs:
-
[Times Data] (https://www.thetimes.com/data)