diff --git a/_data/navigation.yml b/_data/navigation.yml index 4a7ec51e..1ae12afe 100644 --- a/_data/navigation.yml +++ b/_data/navigation.yml @@ -44,13 +44,18 @@ main_menu: url: /astronomy-community/ - title: "For Scientists" submenu: - # - title: "Within SRC & SKAO" - # submenu: - title: "ukSRC Helpdesk" url: "https://uksrc.atlassian.net/servicedesk/customer/portals" blank: true - title: "Science Support" url: /science-projects/ + - title: "Science Tooling" + url: /science-tooling/ + submenu: + - title: "ValSKA" + url: /science-tooling/valska/ + - title: "ValSKA: Case Study" + url: /science-tooling/valska_casestudy/ - title: "SKA Science Verification" url: /science-verification/ - title: "Acknowledgement Statement" diff --git a/_pages/science-tooling.md b/_pages/science-tooling.md new file mode 100644 index 00000000..df615be2 --- /dev/null +++ b/_pages/science-tooling.md @@ -0,0 +1,49 @@ +--- +title: "Science Validation and Tooling: building trustworthy science" +type: pages +permalink: /science-tooling/ +layout: single +header: + overlay_image: /assets/images/SKA-at-Night.jpg + caption: "Image credit: SKAO" +--- + +The Square Kilometre Array (SKA; Figure 1) will produce some of the most ambitious radio astronomy data ever collected. It is designed to help answer frontier questions about the early Universe, galaxy evolution, gravity and dark energy. Turning that data into reliable science, however, requires more than powerful telescopes and sophisticated software. It requires evidence that the analysis methods are trustworthy for the scientific questions we want to answer. + +
+ Composite image of the SKA telescopes +
Figure 1: Composite image of the SKA telescopes, blending real hardware already on site with artist's impressions. From left: An artist's impression of the future SKA-Mid dishes blend into the existing precursor MeerKAT telescope dishes in South Africa. From right: An artist's impression of the future SKA-Low stations blends into the existing AAVS2.0 prototype station in Western Australia. Image credit: SKAO
+
+ +The ukSRC Science Validation and Tooling team develops reusable methods, software and workflows that help scientists test whether SKA and precursor data-analysis methods are reliable for the science they are intended to deliver. Our goal is to make precision radio astronomy results trustworthy, reproducible and auditable, by showing where a method works, where it fails, and which instrument or modelling uncertainties matter most. + + +
+ Validation pyramid +
Figure 2: Validation operates at multiple levels, from checking that simulated data products resemble observations, through testing individual instrument and sky models, to full end-to-end pipeline tests and statistical model validation. By combining these layers, ukSRC can test whether calibration and analysis workflows recover known science signals reliably under realistic modelling uncertainties.
+
+ +## What is validation? + +Validation is the process of testing whether scientific models, simulations, calibration procedures and data-analysis pipelines can be trusted to deliver accurate results. In simple terms, it asks: if we calibrate and analyse data where the scientific result is already known, do we recover that result correctly? + +This is different from verification. Verification asks: “Did we build the system right?” For example, does a pipeline run correctly on test data? Validation asks: “Did we build the right system for the science?” For example, can a pipeline recover a known astrophysical signal from realistic simulated data? + +This distinction matters because modern radio astronomy analyses depend on long chains of sky modelling, telescope modelling, calibration, simulation and statistical inference. Figure 2 illustrates how UKSRC validation works across this chain, from checks on simulated data products and individual model components through to end-to-end pipeline tests and statistical validation of the final inference. This layered approach is important because a workflow may run without errors and still give a biased scientific result if an important instrumental or astrophysical effect has not been modelled or calibrated accurately enough. + +## Why validation matters for SKA science + +Many SKA science goals involve extremely faint signals hidden beneath much brighter foreground emission and complex instrumental effects. For example, 21-cm cosmology aims to study neutral hydrogen from the early Universe, one of the last great frontiers of physical cosmology, comprising the first billion years of cosmic evolution (see Figure 3), and witnessing the birth of the first stars, black holes and galaxies. However, the cosmological signal is far weaker than the radio emission from our Galaxy and other sources. In this regime, the limiting challenge is often not raw telescope sensitivity, but whether we understand the instrument and the analysis well enough to avoid false detections, signal loss or biased measurements. + +Validation provides a way to test this quantitatively. By creating realistic simulated observations, processing them through calibration and analysis workflows, and comparing the recovered outputs with the known inputs, we can measure how reliably the science signal is recovered. These tests can also show how finite precision in instrument knowledge, such as the telescope beam or station response, propagates through calibration into final science products such as images, power spectra, catalogues or cosmological constraints. + + +
+ Schematic timeline of Cosmic History. +
Figure 3: Schematic timeline of Cosmic History. The Epoch of Reionization traces the birth of the first stars and black holes after the Big Bang and their impact on the neutral hydrogen between galaxies that pervaded the Universe at earlier times. Image credit: NASA/WMAP Science Team.
+
+ +## The ukSRC approach +Our approach is modular, reproducible and community-facing. We test individual components, compare simulation tools, run end-to-end pipeline studies, and use statistical methods to assess whether models are robust enough for scientific inference. This allows scientists, instrument teams and programme stakeholders to understand which assumptions matter most and where further modelling, calibration or measurement effort is needed. + +Making validation accessible is central to the programme. We are developing open-source tools, benchmark studies, documentation and reusable workflows that can be inspected, rerun and extended by others. By turning validation into a documented and shareable scientific product, UKSRC helps the community move from “we hope this analysis works” to “we have evidence for where it works, where it fails, and why”. \ No newline at end of file diff --git a/_pages/valska.md b/_pages/valska.md new file mode 100644 index 00000000..59c4d60a --- /dev/null +++ b/_pages/valska.md @@ -0,0 +1,34 @@ +--- +title: "ValSKA: a reusable ukSRC validation framework" +type: pages +permalink: /science-tooling/valska +layout: single +header: + overlay_image: /assets/images/SKA-at-Night.jpg + caption: "Image credit: SKAO" +--- + +A ValSKA validation campaign starts with a science question. What signal are we trying to recover? Which telescope, sky, calibration or modelling uncertainties could affect the result? What metric tells us whether the analysis has succeeded or failed? The campaign then uses controlled simulations and perturbations to test how those uncertainties propagate through calibration, analysis and inference into the final scientific output. + +
+ Conceptual view of ValSKA as a reusable validation framework +
Conceptual view of ValSKA as a reusable validation framework. Science and instrument drivers define validation objectives, which are tested through simulation campaigns, calibration and analysis workflows, and quantitative trust metrics. The shared ValSKA layer provides common infrastructure for configuration, execution, provenance, reporting and reusable artefacts, while allowing science-case-specific modules to be developed for 21-cm power spectra, continuum imaging, calibration, intensity mapping and future SKA workflows.
+
+ +The figure above shows the overall ValSKA approach. Science and instrument drivers define the scope of a validation campaign. These are translated into simulation studies, calibration and analysis workflows, and quantitative trust metrics. A shared ValSKA framework then provides common infrastructure for configuration, execution, provenance capture, reporting and reusable artefacts, while allowing science-case-specific modules to be developed for different SKA and precursor workflows. + +## What ValSKA provides + +The framework is designed to connect the full validation chain: science objectives, simulation campaigns, calibration and analysis workflows, validation metrics, and decision-relevant outputs. In practical terms, this means workflow templates, configuration files, tool interfaces, HPC execution support, provenance capture, summary reports and reusable scientific artefacts. + +The current public demonstrator focuses on 21-cm power-spectrum validation using HERA and BayesEoR. Existing functionality includes orchestration for Bayesian 21-cm inference, built-in Bayesian null-testing, reproducible HPC workflows, unit tests, continuous integration, documentation and command-line tools for preparing, submitting, auditing and reporting validation runs. The software is being developed through the UKSRC GitHub organisation, with the aim of making mature tools and examples available for community reuse. + +## Why this matters + +For science teams, ValSKA provides a clearer route from a validation question to a documented conclusion. For instrument and calibration teams, it provides a way to translate scientific requirements into quantitative tolerances, such as how accurately a beam, station response, array layout or calibration model must be known. For the wider community, it provides shareable workflows and benchmark products that support reproducibility and comparison across methods. + +ValSKA is also being extended beyond its initial 21-cm power-spectrum demonstrator. Current development is moving toward interferometric simulation benchmarking, higher-fidelity instrument and sky models, continuum imaging validation in collaboration with SRCNet partners, and 21-cm intensity-mapping validation with UKSRC collaborators. In these upcoming case studies, calibration is a central part of the validation question: how well must the interferometer be characterised for calibration to support reliable recovery of the target science? + +## Outputs and validation campaign + +A mature campaign will produce a validation report, benchmark datasets, configuration files, provenance records, diagnostic tables and plots, packaged workflow components, containers and, where appropriate, DOI-linked artefacts. These outputs make the basis for scientific trust explicit rather than implicit, and help validation results feed back into science readiness, calibration priorities and instrument-characterisation requirements. diff --git a/_pages/valska_casestudy.md b/_pages/valska_casestudy.md new file mode 100644 index 00000000..0b814d63 --- /dev/null +++ b/_pages/valska_casestudy.md @@ -0,0 +1,32 @@ +--- +title: "A ValSKA Case Study: How well do we need to know the telescope beam?" +type: pages +permalink: /science-tooling/valska_casestudy +layout: single +header: + overlay_image: /assets/images/SKA-at-Night.jpg + caption: "Image credit: SKAO" +--- + +The first ValSKA case study asks a practical question with direct scientific consequences: how accurately must a radio telescope’s beam be known for unbiased 21-cm cosmology? + +A telescope beam describes how sensitive an antenna or station is to different directions on the sky. For low-frequency radio cosmology, beam modelling is especially important because the faint 21-cm signal from the early Universe is observed through much brighter foreground emission from our Galaxy and other radio sources. Small errors in the beam model can affect calibration, mix bright foreground structure into the target science signal, or bias the final interpretation. + +The case study uses HERA, an SKA precursor telescope in South Africa, as a testbed. HERA is designed for measurements of the 21-cm power spectrum from Cosmic Dawn and the Epoch of Reionization. The analysis asks how errors in the beam full width at half maximum, a measure of the beam’s angular size, affect Bayesian power-spectrum inference with BayesEoR. + +## The validation test + +The figure below summarises the validation workflow. The method is deliberately conservative: simulated observations are generated containing foreground emission and instrumental effects, but no cosmological 21-cm signal. The analysis is then repeated while introducing increasing levels of controlled beam-width error. If the analysis reports Bayesian evidence in favour of a 21-cm signal being present in data where no such signal was injected, the validation test fails for that level of beam uncertainty. + +This is a Bayesian null test: the expected answer is no detection. It is designed to identify when imperfect instrument knowledge could produce a spurious signal or bias the scientific interpretation. The result can then be used to set an instrument-knowledge requirement: the level to which the beam must be measured or modelled for the intended science analysis to remain trustworthy. + +
+ Testing how beam uncertainty affects 21-cm science recovery +
Testing how beam uncertainty affects 21-cm science recovery. The ValSKA beam case study uses simulated data with no injected cosmological signal, then repeats the analysis with controlled beam-width errors. If the pipeline reports a spurious 21-cm detection, the test fails for that level of instrument uncertainty. In the current HERA demonstrator, this approach indicates an approximate ±5% beam-width knowledge requirement under the assumptions of the study.
+
+ +## Current findings and next steps + +In the current HERA demonstrator, the tested setup indicates that the beam width needs to be known to approximately ±5% for the analysis to avoid a spurious detection under the assumptions of the study. The exact tolerance is not intended as a universal number for every telescope or analysis. The more important outcome is the method: a reproducible way to translate modelling uncertainty into a science-driven requirement. + +The next stage is to increase realism. Current simulations use Galactic and extragalactic foregrounds with simplified beam models; upcoming work will incorporate more detailed Airy and electromagnetic beam simulations, mutual-coupling effects, simulator benchmarking, and comparisons with observational data. The same validation logic can then be extended to SKA-Low stations, continuum imaging, 21-cm intensity mapping, and other SKA and precursor science workflows where calibration and instrument knowledge directly affect science recovery. diff --git a/assets/images/how-well-fig1.png b/assets/images/how-well-fig1.png new file mode 100644 index 00000000..24ec4825 Binary files /dev/null and b/assets/images/how-well-fig1.png differ diff --git a/assets/images/science-validation-fig1.jpg b/assets/images/science-validation-fig1.jpg new file mode 100644 index 00000000..acd08fad Binary files /dev/null and b/assets/images/science-validation-fig1.jpg differ diff --git a/assets/images/science-validation-fig2.png b/assets/images/science-validation-fig2.png new file mode 100644 index 00000000..f972f567 Binary files /dev/null and b/assets/images/science-validation-fig2.png differ diff --git a/assets/images/science-validation-fig3.jpg b/assets/images/science-validation-fig3.jpg new file mode 100644 index 00000000..7bd5a9c1 Binary files /dev/null and b/assets/images/science-validation-fig3.jpg differ diff --git a/assets/images/valska-fig1.png b/assets/images/valska-fig1.png new file mode 100644 index 00000000..e34024e8 Binary files /dev/null and b/assets/images/valska-fig1.png differ