sdmx-dl - easily download official statistics

sdmx-dl is a tool designed to easily download official statistics.
It is mainly based on the SDMX standard but can be extended with other APIs as well.

This project has three parts:

• a Java library that can be incorporated into other projects
• a command-line tool that allows to easily browse and download data
• a web service that serves as a bridge to any application or language

Its documentation is available at https://nbbrd.github.io/sdmx-dl/docs/.

Why?

While being in the information age, it is surprisingly difficult to get free quality statistics, even from official sources.

There are many reasons to that situation:

• Big tech lockup: the huge data collected by big tech companies are locked behind paywalls and the only bits freely available are often redacted to the point of being useless.
• Too many APIs: NGOs and governmental agencies that adhere to the open data principles don't necessarily share a common design for their APIs and therefore require a separate development for each one.
• Time-consuming handling: data retrieval often implies tedious and error-prone manual steps.
• No native support: there are several open standards available but few-to-none are natively supported by mainstream applications.
• No catalog: there are no central catalog to discover data sources.

sdmx-dl is a SDMX-focused attempt to tackle these problems.

Goals & features

• Provides an easy, consistent and reliable data retrieval
• Allows data discovery and automation
• Takes care of technical difficulties and data handling
• Is designed to be used by other applications
• Is bundled with a pre-configured set of data sources covering major international organizations and national agencies (ECB, IMF, OECD, Eurostat, World Bank, ILO, BIS, INSEE, ...)

Quick Start

CLI — download monthly CHF/EUR exchange rates from the ECB in one command (JBang required):

jbang sdmx-dl@nbbrd fetch data ECB EXR M.CHF.EUR.SP00.A

Java library — retrieve data in a few lines with a JBang script:

//JAVA 25+
//DEPS com.github.nbbrd.sdmx-dl:sdmx-dl-standalone:3.0.0-beta.18
import sdmxdl.*;
import sdmxdl.web.*;

void main() throws Exception {
    SdmxWebManager
        .ofServiceLoader()
        .usingName("ECB")
        .getData(KeyRequest.builder()
            .flowOf("EXR")
            .keyOf("M.CHF+USD.EUR.SP00.A")
            .build())
        .forEach(series -> IO.println(series.getKey() + ": " + series.getObs().size() + " obs"));
}

Installing

sdmx-dl CLI runs on any desktop operating system such as Microsoft Windows, Solaris OS, Apple macOS, Ubuntu and other various Linux distributions.

Platform	Command
Windows (Scoop)	scoop bucket add nbbrd https://github.com/nbbrd/scoop-nbbrd.git && scoop install sdmx-dl
macOS / Linux (Homebrew)	brew install nbbrd/tap/sdmx-dl
Any (JBang)	jbang sdmx-dl@nbbrd <command> [<args>]

See CLI installation for all options including Docker, GitHub Actions, Maven, and Gradle.

Developing

This project is written in Java and uses Apache Maven as a build tool.
It requires Java 8 as minimum version and all its dependencies are hosted on Maven Central.

The code can be built using any IDE or by just type-in the following commands in a terminal:

git clone https://github.com/nbbrd/sdmx-dl.git
cd sdmx-dl
mvn clean install

Contributing

Any contribution is welcome and should be done through pull requests and/or issues.

Licensing

The code of this project is licensed under the European Union Public Licence (EUPL).

Related work

This project is not the only one that deals with official statistics.
Here is a non-exhaustive list of related work:

• amattioc/SDMX (SDMX / Java)
• sosna/sdmx-rest4js (SDMX / JavaScript)
• khaeru/sdmx (SDMX / Python)
• rOpenGov/pxweb (PXWEB / R)
• ondata/opensdmx (SDMX / Python)