Problem downloading files

[CeciliaNilsson709]

I am having issues downloading and opening monthly csv.gz files from Aloft using Safari.

2024 files work fine:
https://aloftdata.s3-eu-west-1.amazonaws.com/baltrad/monthly/sekrn/2024/sekrn_vpts_202401.csv.gz

But 2023 (and older) don’t download when clicked on, but just open in the Safari browser (showing binary code). If I save the file and double click to unzip, they give an error message:
https://aloftdata.s3-eu-west-1.amazonaws.com/baltrad/monthly/sekrn/2023/sekrn_vpts_202301.csv.gz

The files themselves seem ok though, I can open them in R.

[peterdesmet]

@PietrH is this a result of how we gzipped the data?

[PietrH]

sekrn_vpts_202301.csv.gz : Very strange, it downloads just fine, but if I open it via the download manager it get's opened in my text editor as well.

Can we figure out if this is an archive you or I created Peter?

[PietrH]

I just inspected the request, and the file from 2024 identifies as MIME type binary/octet-stream, while the second one as text/csv, which is probably causing the browser to refer to the wrong application for opening the file..

See: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Type

I'm not sure if this is a server side thing, or happened during file creation...

[peterdesmet]

This issue is annoying for:

1. Pre-2024 files (files created by operational pipeline since 2024 are fine)
2. Safari users
3. Manually unzipping (confirmed on Mac)

But I'm reluctant to fix it because there is a workaround:

1. Downloading in Chrome and unzipping in Terminal with gzip -d sekrn_vpts_202301.csv.gz
2. Getting data via getRad::get_vpts()

... and fixing this would required updating all manually created gz files with the correct MIME type.

@PietrH what do you think? How much work would this be?

[PietrH]

It would be helpful if we could figure out what files would need to be recreated. Recreating the monthly files is trivial if you have a local copy of the data (or a very fast connection).

My hypothesis is that there are two sources for the monthly files, the script I ran on the HPC and some that were manually edited and resaved.

Do you happen to have a local copy of the daily files still? I once gave you one on a hard drive (during the Hirad meeting in Gent).

[PietrH]

I wrote a little gist on how to reprocess a single month via the webaccess:

library(lubridate)
library(getRad)

month_to_collate <- 
  lubridate::interval(lubridate::ym(202205),
                      lubridate::ym(202206) - lubridate::days(1))
radar_to_collate <- "nldhl"

pre_2024_coverage <- aloft_data_coverage() |>
  dplyr::filter(date < lubridate::ymd("2024-01-01")) |>
  dplyr::mutate(month = lubridate::month(date))

aws_domain <- "https://aloftdata.s3-eu-west-1.amazonaws.com"

pre_2024_coverage |>
  dplyr::mutate(vpts_file = file.path(aws_domain,
                                      source,
                                      "daily",
                                      radar,
                                      lubridate::year(date),
                                      glue::glue("{radar}_vpts_{year}{month}{day}.csv",
                                                 year = format(date, "%Y"),
                                                 month = format(date, "%m"),
                                                 day = format(date, "%d"))
  )) |>
  # only collate specific month and radar, you could use something like dplyr
  # group_map or a for loop here instead if you want to process multiple.
  dplyr::filter(date %within% month_to_collate,
                radar == radar_to_collate) |>
  dplyr::pull(vpts_file) |>
  purrr::map(~readr::read_csv(.x,
                              col_types = readr::cols(.default = "c"),
                              show_col_types = FALSE,
                              progress = FALSE),
             .progress = TRUE) |>
  purrr::list_rbind()

[PietrH]

I also wrote a few lines that might help us figure out what monthlies are causing trouble:

test_monthly_url <- 
  "https://aloftdata.s3-eu-west-1.amazonaws.com/baltrad/monthly/chwei/2021/chwei_vpts_202101.csv.gz"

get_mimetype <- function(urls){
  purrr::map(urls, httr2::request) |>
    purrr::map(~httr2::req_method(.x, "HEAD")) |>
    httr2::req_perform_parallel() |>
    purrr::map_chr(httr2::resp_content_type)
}

get_mimetype(test_monthly_url)
#> [1] "text/csv"

^{Created on 2025-06-19 with reprex v2.1.1}