Add `get_archival_data()` function

[PietrH]

STRAITS

Add a new function get_archival_data() to retrieve processed tag archival data. The user can define tag (tag_serial_number) and data aggregation level (summarized or all). The function retrieves relevant data (files) and returns it to the user as a single data frame.

summarized: only return data from whole hours. Drop all other data points. I could easily add support for other steps in aggregation. Are we sure no averaging or similar takes place?

• fetch csv's and combine them
• fetch them where?
• decompression
• how does the user select the archival data, on what parameters?
• Is a etn database query necessary? Or can it be done purely client based?
• What postgres table contains the links to the csv files?
• What is the best way to programmatically read the csv files without having to manage an authentication cookie.
• Read csv with readr or via httr2? -> httr2 for throttling and retries
• Add lazy evaluation support via duckdb? arrow
• Fix bug: col classes need to match while merging: add schema?

`purrr::list_rbind()` at etn/R/get_archival_data.R:49:3:
! Can't combine `..1$timestamp_utc` <datetime<UTC>> and `..2$timestamp_utc` <character>.

• Adjust throttling
• BUG: running out of working memory when loading multliple large datasets into memory for parsing; Store onto disk and parse one by one?
• Add progress bar argument
• Idea: Break up progress in two steps: fetching data, reading/parsing csv
• Idea: Fetch files sequentially (big files saturate network connections), and use httr2::req_progress() to show downloaded bytes.
• Add aggregation and aggregation argument: via duckdb? need to group by the floor of the datetime, and then aggregate. duckdb is much faster. Let's not take this as extra maintenance for now.
• Test on animal_project_code case sensitivitly
• Test on invalid values for animal_project_code
• Add caching via httr2 for return values: very stable
• Add tests for col classes: hard to predict what cols we'll get back
• Add limit argument for tests?
• Add argument so users can store csv files: path or dir
• Allow partial resume from previously downloaded files
• Name the local files: uuid for now
• BUG: case sensitivity in animal_project_code: BUG: animal_project_code is case sensitive in etnservice::get_archival_data_uuid() etnservice#154 (comment)

Minimal Viable Product

• Describe changes in NEWS
• Add extension to local files
• Deploy etnservice::get_archival_data_uuid() to OpenCPU production
• Add test skips on data domain offline and on no credentials
• Add documentation of returned object, what cols to expect
• Decide: Should invalid values for arguments be checked on the client or via the service? Currently all on service, is this robust?, to client, but postponed see Check for invalid values on the client side #588

Postponed/Extend goals

• Should users be able to read local files without a database query? Without database credentials?
• Don't allow tibble return on very large datasets? How to efficiently count rows of returned csv's? Decide purely on filesize?
• Stop users from trying to read all archival data available to them, this will be millions of records
• Add local caching of uuid values? very stable: cachem over memoise
• return helpful error if a passed path (dir) doesn't exist
• Abstract opening arrow dataset from online hosted files: reusable for public data
• Abstract downloading a file with package defaults (declaring user agent, retries, path handling): can be reused for public data, and for get_acoustic_detections() feather and parquet files
• rename temp_file_paths: not always temporary files
• Abstract out extrating a uuid from a filepath: more readable
• tests: cache http responses?: faster, more consistent, less load on VLIZ, but possibly unacceptably increases package size

Questions

How to detect if a project has archival data -> Will be useful for get_package(), it'll sometimes need to get achival data, how to know that you need to get it.

Requirements

• Add a new function get_archival_data() to retrieve processed tag archival data.
• The user can define tag (tag_serial_number) ~~and data aggregation level (summarized or all). ~~ -> users can aggregate themselves for now, if there is demand we can add aggregation functions later on. You'd need to read all the data anyway.
• The function retrieves relevant data (files) and returns it to the user as a single data frame.

[PietrH]

get_archival_data <- function(tag_serial_number = NULL, data_aggregation_level = c("all", "summarized")){}

[PietrH]

Status 20260203: etn development roadmap, document by Lotte

• What: data pipeline for archival data collected by Data Storage Tags.
• How: Data will be stored as raw (original files), processed (readable files) and summary
  (database table). Summary data are a subset of the raw data.
• Structure: similar to Darwin Core Measurements or Facts:
• date_time
• tag_serial_number
• sensor_type
• sensor_value
• sensor_unit
• Scope: STRAITS, Deadline: December 2026

[PietrH]

Notes form discussion with Peter 20250606:

get_archival_data() is a difficult one, the raw data fetching should be pretty easy. Because the link to these files is present in the database. They should always be in the same format. Then convert to data.frame and return to the user.

Aggregated data will be more difficult, because it's calculated with a query with a mongo-db database. Package will need to be able to run a mongo_db query, not sure how this'll go. = currently blocked.

The mongo-db also contains all of the processed data, so we might not need to go via the file download at all.

[PietrH]

Notes from ETN meeting 20250909

Archival data: first release in production: only works for cfas. 

summary is calculated when requested, a query on the mongo db

the standardized csv is available, stored in mongo. Mongo -> easy path for parquet export. Mongo is built in same partitioning as the parquet files will be.

contract: 
 - add function r package: to get processed tag archival data (not summary data), tag, tag_serial_number, from contract: 
 - > Add a new function get_archival_data() to retrieve processed tag archival data. The user can define tag (tag_serial_number) and data aggregation level (summarized or all). The function retrieves relevant data (files) and returns it to the user as a single data frame.
 -  Vision: query postgres database tags one by one, then combine them one by one. If the user requests summarized data: query mongo db database via the R package.
 - the summaries take around 30 seconds to be calculated via the mongodb at the moment
 - Peter doesn't think the R package should talk to mongo-db
 - there is no summary file, they are calculated on the spot. Stijn: it's possible to calculate them and store them, but how useful would that be? 

[PietrH]

Does a user need to log into MDA (marine data archive: https://mda.vliz.be/archive.php) to download the csv files?

[PietrH]

Does a user need to log into MDA (marine data archive: https://mda.vliz.be/archive.php) to download the csv files?

No, I think these can be fetched from: "https://www.lifewatch.be/etn/archival-data/", files are selected via a GUID: eg: https://www.lifewatch.be/etn/archival-data/file/1980c349-b4df-4b58-a95d-0a6c77a31fc9

But an authentication cookie is passed in the GET request, so you need to login to access this data.

Is there a way around this? What is the preferred way of authentication on the ETN data platform?

[PietrH]

Let's not do data aggregation to start with. Mongo-db could aggregate, but let's not depend on mongo-db. We are not sure it'll still be here on the long term.

The user can aggregate themselves.

[PietrH]

Only support one tag_serial_number at a time, supporting a vector makes it too easy to ask for much much more data than you'd be able to process. Even a single tag is already 1GB+ of data.

Users can call the function in a loop.

[Stijn-VLIZ]

Does a user need to log into MDA (marine data archive: https://mda.vliz.be/archive.php) to download the csv files?

No, I think these can be fetched from: "https://www.lifewatch.be/etn/archival-data/", files are selected via a GUID: eg: https://www.lifewatch.be/etn/archival-data/file/1980c349-b4df-4b58-a95d-0a6c77a31fc9

But an authentication cookie is passed in the GET request, so you need to login to access this data.

Is there a way around this? What is the preferred way of authentication on the ETN data platform?

ETN can make these links unprotected by login, but we will need to add checks if people are allowed to retrieve certain files, question remains how we will do that...