Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
3286b03
docs: fix typos and improve documentation
terryrankine Mar 17, 2026
773f085
ci: fix MinIO image reference for CI tests
terryrankine Mar 17, 2026
edb59f9
fix: respect --profile flag with shared config session
terryrankine Mar 17, 2026
d04203c
fix: check exclude/include filters before rejecting non-regular files
terryrankine Mar 17, 2026
d53ea47
fix: prevent nil pointer dereference in shouldOverride
terryrankine Mar 17, 2026
68c2547
feat: add Tencent Cloud Object Storage (COS) support
terryrankine Mar 17, 2026
a359daf
fix: use shellquote for special characters in sync-generated commands
terryrankine Mar 17, 2026
f2ef696
perf: skip unnecessary stat call when progress bar is disabled
terryrankine Mar 17, 2026
2589055
fix: improve sync error handling and cancellation propagation
terryrankine Mar 17, 2026
91f45bd
fix: retry on expired session token instead of failing immediately
terryrankine Mar 17, 2026
9aee5e5
feat: automatic multipart copy for objects larger than 5 GiB
terryrankine Mar 17, 2026
6138e7e
vendor: bump aws-sdk-go v1.44.298 → v1.55.5
terryrankine Mar 17, 2026
e39b82a
ci: add codespell workflow and fix typos
terryrankine Mar 17, 2026
223b8f9
fix: correct region assignment bug in test helper
terryrankine Mar 18, 2026
1e41921
fix: use max instead of min in argument count error message
terryrankine Mar 18, 2026
64e54f4
fix: use atomic loads for progress bar counter reads
terryrankine Mar 18, 2026
a2a4333
fix: hold locks when reading stat maps in Statistics()
terryrankine Mar 18, 2026
e6fb1ba
fix: guard against nil pointer dereference in metadata access
terryrankine Mar 18, 2026
aedd1cc
fix: add defer cancel() and replace os.Exit in sync command
terryrankine Mar 18, 2026
3658f54
fix: replace os.Exit and panic with proper error handling in copy
terryrankine Mar 18, 2026
e9d08ac
fix: return valid file handle in dry-run mode
terryrankine Mar 18, 2026
8446d78
ci: update GitHub Actions to latest versions
terryrankine Mar 18, 2026
ad3e90a
fix: add nil guard to parallel.Run with clear panic message
terryrankine Mar 18, 2026
ecff4c6
fix: use %w for error wrapping in s3 storage
terryrankine Mar 18, 2026
018902a
docs: add unreleased section to CHANGELOG
terryrankine Mar 18, 2026
8078699
fix: apply source/destination region overrides in sync command
terryrankine Mar 18, 2026
ff4fd4e
perf: add pooled buffers for S3 upload and download managers
terryrankine Mar 18, 2026
5570022
feat: add --start-after flag for ls command pagination
terryrankine Mar 18, 2026
f41bf88
feat: add --log-progress flag for log-friendly progress output
terryrankine Mar 18, 2026
e9c1b36
feat: add riscv64 architecture support
terryrankine Mar 18, 2026
afa3158
feat: add --addressing-style flag for virtual host style endpoints
terryrankine Mar 18, 2026
11b13bf
feat: add --log-file flag for logging to file
terryrankine Mar 18, 2026
9080e28
chore: upgrade to Go 1.24 and bump extsort to v1.4.2
terryrankine Mar 18, 2026
65c80ab
fix: merge user metadata instead of overwriting retry ID
terryrankine Mar 18, 2026
3e4e429
fix: protect LogProgressBar.lastUpdate with mutex
terryrankine Mar 18, 2026
40228af
fix: make log.Close() safe to call multiple times
terryrankine Mar 18, 2026
db386b9
fix: cache progressbar type check before object loop
terryrankine Mar 18, 2026
6c7b161
fix: drain extsort error channels inline instead of in goroutines
terryrankine Mar 18, 2026
7269233
docs: add issue tracker mapping 136 open issues to our fixes
terryrankine Mar 18, 2026
de03428
fix: add B suffix to humanized byte sizes below 1K
terryrankine Mar 18, 2026
4e631d3
fix: allow rm to delete S3 directory objects (keys ending in /)
terryrankine Mar 18, 2026
1002ca2
fix: sync --delete now respects --exclude and --include filters
terryrankine Mar 18, 2026
2524fc9
test: add unit tests for previously untested fixes
terryrankine Mar 18, 2026
9c08554
test: add TestShouldStopSync for sync error classification
terryrankine Mar 18, 2026
5eb5111
test: add TestFormatBytes and TestFormatDuration for log progress
terryrankine Mar 18, 2026
7e82d00
test: add TestFilesystemCreateDryRun for dry-run file handles
terryrankine Mar 18, 2026
70e9377
fix: guard against nil ModTime in shouldOverride
terryrankine Mar 18, 2026
f45fa6b
chore: upgrade staticcheck v0.4.7 → v0.6.0 for Go 1.24 compat
terryrankine Mar 18, 2026
d20f951
fix: suppress SA1019 deprecation warnings and fix Windows test
terryrankine Mar 18, 2026
920d789
fix: remove unused 'to' parameter from multipartCopy
terryrankine Mar 18, 2026
f8e8ed1
refactor: migrate extsort from legacy SortType to Generic[T] API
terryrankine Mar 18, 2026
4557392
fix: gofmt formatting and codespell false positive
terryrankine Mar 18, 2026
7fec15a
changelog: prepare for v2.4.0 release
terryrankine Mar 18, 2026
6bf45d6
ci: make GCS test optional in goreleaser workflow
terryrankine Mar 18, 2026
5da5c10
fix: use step-level conditional for GCS test secrets
terryrankine Mar 18, 2026
d3f0914
chore: migrate goreleaser config to version 2
terryrankine Mar 18, 2026
3b9147f
fix: resolve all goreleaser deprecation warnings
terryrankine Mar 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
3 changes: 3 additions & 0 deletions .codespellrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[codespell]
skip = vendor,go.sum,.git
ignore-words-list = keypair,ist,cancelation,hypen,bu,te,filetest,re-used
29 changes: 13 additions & 16 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,8 @@ jobs:
strategy:
matrix:
go-version:
- 1.22.x
- 1.21.x
- 1.20.x
- 1.24.x
- 1.23.x
os:
- macos
- ubuntu
Expand All @@ -22,8 +21,8 @@ jobs:
name: build (${{ matrix.os }}/go-${{ matrix.go-version }})
runs-on: ${{ matrix.os }}-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-go@v2
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}

Expand All @@ -33,9 +32,8 @@ jobs:
strategy:
matrix:
go-version:
- 1.22.x
- 1.21.x
- 1.20.x
- 1.24.x
- 1.23.x
os:
- macos
- ubuntu
Expand All @@ -45,7 +43,7 @@ jobs:
runs-on: ${{ matrix.os }}-latest
services:
minio:
image: ${{ (matrix.os == 'ubuntu') && 'bitnami/minio:2023.7.18' || ''}}
image: ${{ (matrix.os == 'ubuntu') && 'bitnamilegacy/minio:2023.7.18' || ''}}
ports:
- 45677:9000
options: >-
Expand All @@ -57,8 +55,8 @@ jobs:
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minioadmin
steps:
- uses: actions/checkout@v2
- uses: actions/setup-go@v2
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}

Expand All @@ -69,16 +67,15 @@ jobs:
strategy:
matrix:
go-version:
- 1.22.x
- 1.21.x
- 1.20.x
- 1.24.x
- 1.23.x
os:
- ubuntu

runs-on: ${{ matrix.os }}-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-go@v2
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: ${{ matrix.go-version }}

Expand Down
14 changes: 14 additions & 0 deletions .github/workflows/codespell.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
name: codespell
on:
push:
branches:
- master
pull_request:
branches:
- master
jobs:
codespell:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: codespell-project/actions-codespell@v2
12 changes: 6 additions & 6 deletions .github/workflows/docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,27 +8,27 @@ jobs:
steps:
-
name: Checkout
uses: actions/checkout@v2
uses: actions/checkout@v4
-
name: Set up QEMU
uses: docker/setup-qemu-action@v1
uses: docker/setup-qemu-action@v3
-
name: Set up Docker Buildx
uses: docker/setup-buildx-action@v1
uses: docker/setup-buildx-action@v3
-
name: Login to DockerHub
uses: docker/login-action@v1
uses: docker/login-action@v3
with:
username: ${{ secrets.DOCKER_USERNAME }}
password: ${{ secrets.DOCKER_TOKEN }}
-
name: Build and push
uses: docker/build-push-action@v2
uses: docker/build-push-action@v6
with:
context: .
file: ./Dockerfile
push: true
platforms: linux/amd64,linux/arm64,linux/ppc64le,linux/arm/v6,linux/arm/v7,linux/386
platforms: linux/amd64,linux/arm64,linux/ppc64le,linux/riscv64,linux/arm/v6,linux/arm/v7,linux/386
tags: |
peakcom/s5cmd:latest
peakcom/s5cmd:${{ github.event.release.tag_name }}
28 changes: 17 additions & 11 deletions .github/workflows/goreleaser.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,38 +8,44 @@ jobs:
name: gcs-test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-go@v2
- uses: actions/checkout@v4
- uses: actions/setup-go@v5
with:
go-version: 1.22.x
- run: make test_without_race
go-version: 1.24.x
- name: Run GCS tests
if: env.S5CMD_ACCESS_KEY_ID != ''
run: make test_without_race
env:
S5CMD_ACCESS_KEY_ID: ${{ secrets.S5CMD_GCS_ACCESS_KEY_ID }}
S5CMD_SECRET_ACCESS_KEY: ${{ secrets.S5CMD_GCS_SECRET_ACCESS_KEY }}
S5CMD_ENDPOINT_URL: https://storage.googleapis.com
S5CMD_IS_VIRTUAL_HOST: 1
S5CMD_REGION: ${{ secrets.S5CMD_GCS_REGION }}
S5CMD_I_KNOW_WHAT_IM_DOING: 1
- name: Skip GCS tests
if: env.S5CMD_ACCESS_KEY_ID == ''
run: echo "GCS secrets not configured, skipping"
env:
S5CMD_ACCESS_KEY_ID: ${{ secrets.S5CMD_GCS_ACCESS_KEY_ID }}

goreleaser:
needs: [ gcs_test ]
runs-on: ubuntu-latest
steps:
-
name: Checkout
uses: actions/checkout@v2
uses: actions/checkout@v4
with:
fetch-depth: 0
-
name: Set up Go
uses: actions/setup-go@v2
uses: actions/setup-go@v5
with:
go-version: '1.22'
go-version: 1.24.x
-
name: Run GoReleaser
uses: goreleaser/goreleaser-action@v2
uses: goreleaser/goreleaser-action@v6
with:
version: '1.18.2'
args: release --rm-dist
version: latest
args: release --clean
env:
GITHUB_TOKEN: ${{ secrets.GORELEASER_TOKEN }}
35 changes: 23 additions & 12 deletions .goreleaser.yml
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
version: 2

before:
hooks:
- make build
Expand All @@ -14,30 +16,39 @@ builds:
goarch:
- 386
- amd64
- arm
- arm64
- ppc64le
- riscv64
ignore:
- goos: darwin
goarch: 386
- goos: darwin
goarch: riscv64
- goos: windows
goarch: ppc64le
- goos: windows
goarch: riscv64

archives:
- name_template: '{{ .ProjectName }}_{{ .Version }}_{{ .Os }}-{{ .Arch }}{{ if .Arm }}v{{ .Arm }}{{ end }}'
replacements:
darwin: macOS
linux: Linux
windows: Windows
386: 32bit
amd64: 64bit
- name_template: >-
{{- .ProjectName }}_{{ .Version }}_
{{- if eq .Os "darwin" }}macOS
{{- else if eq .Os "linux" }}Linux
{{- else if eq .Os "windows" }}Windows
{{- else }}{{ .Os }}{{ end }}-
{{- if eq .Arch "amd64" }}64bit
{{- else if eq .Arch "386" }}32bit
{{- else }}{{ .Arch }}{{ end }}
{{- if .Arm }}v{{ .Arm }}{{ end }}
format_overrides:
- goos: windows
format: zip
formats: [zip]

checksum:
name_template: '{{ .ProjectName }}_checksums.txt'

snapshot:
name_template: "{{ .Tag }}"
version_template: "{{ .Tag }}"

changelog:
sort: asc
Expand All @@ -49,10 +60,10 @@ changelog:
- Merge branch

brews:
- tap:
- repository:
owner: peak
name: homebrew-tap
folder: Formula
directory: Formula
name: s5cmd
description: Parallel S3 and local filesystem execution tool
homepage: https://github.com/peak/s5cmd
Expand Down
34 changes: 33 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,36 @@
# Changelog
## v2.4.0 - 18 Mar 2026

#### Features
- Added automatic multipart copy for S3 objects larger than 5 GiB. ([#856](https://github.com/peak/s5cmd/issues/856))
- Added Tencent Cloud Object Storage (COS) support. ([#809](https://github.com/peak/s5cmd/issues/809))

#### Improvements
- Upgraded aws-sdk-go from v1.44.298 to v1.55.5, adding EKS Pod Identity support. ([#769](https://github.com/peak/s5cmd/issues/769))
- Skip unnecessary HeadObject call when progress bar is disabled. ([#793](https://github.com/peak/s5cmd/issues/793))
- Updated GitHub Actions to latest versions (checkout v4, setup-go v5).
- Added codespell CI workflow for typo detection. ([#701](https://github.com/peak/s5cmd/issues/701))
- Dropped Go 1.20 from CI matrix (EOL).

#### Bugfixes
- Fixed `--profile` flag to work with SSO and assume-role credential sources. ([#847](https://github.com/peak/s5cmd/issues/847))
- Fixed nil pointer dereference in `shouldOverride` when stat returns no object. ([#838](https://github.com/peak/s5cmd/issues/838))
- Fixed non-regular files producing errors when they match exclude patterns. ([#776](https://github.com/peak/s5cmd/issues/776))
- Fixed shell quoting for object keys with special characters in sync commands. ([#761](https://github.com/peak/s5cmd/issues/761))
- Fixed sync error handling to stop on network and serialization failures. ([#698](https://github.com/peak/s5cmd/issues/698))
- Fixed expired session tokens to retry with fresh credentials instead of failing. ([#683](https://github.com/peak/s5cmd/issues/683))
- Fixed MinIO CI image reference (`bitnami` → `bitnamilegacy`). ([#840](https://github.com/peak/s5cmd/issues/840))
- Fixed missing subcommand in `cp` help example, typos, and documentation. ([#857](https://github.com/peak/s5cmd/issues/857), [#828](https://github.com/peak/s5cmd/issues/828), [#783](https://github.com/peak/s5cmd/issues/783), [#774](https://github.com/peak/s5cmd/issues/774), [#781](https://github.com/peak/s5cmd/issues/781))
- Fixed data races in progress bar counter reads and statistics map access.
- Fixed nil pointer dereference in metadata retry ID access.
- Fixed resource leak in `Select()` when S3 API call fails.
- Fixed `os.Exit(1)` in error goroutines replaced with context cancellation.
- Fixed `panic()` in copy command replaced with error return.
- Fixed dry-run mode returning zero-value file handles that panic on use.
- Fixed argument count error message showing min instead of max.
- Fixed context leak from missing `defer cancel()` in sync command.
- Fixed test helper assigning access key ID as region value.

## v2.3.0 - 16 Dec 2024

#### Breaking changes
Expand Down Expand Up @@ -129,7 +161,7 @@
- Added `select` command. It allows to select JSON records from objects using SQL expressions. ([#299](https://github.com/peak/s5cmd/issues/299)) [@skeggse](https://github.com/skeggse)
- Added `rb` command to remove buckets. ([#303](https://github.com/peak/s5cmd/issues/303))
- Added `--exclude` flag to `cp`, `rm`, `ls`, `du` and `select` commands. This flag allows users to exclude objects with given pattern. ([#266](https://github.com/peak/s5cmd/issues/266))
- Added `--raw` flag to `cp` and `rm` commands. It disables the wildcard operations. It is useful when an object contains glob characters which interfers with glob expansion logic. ([#235](https://github.com/peak/s5cmd/issues/235))
- Added `--raw` flag to `cp` and `rm` commands. It disables the wildcard operations. It is useful when an object contains glob characters which interferes with glob expansion logic. ([#235](https://github.com/peak/s5cmd/issues/235))
- Added `--cache-control` and `--expires` flags to `cp` and `mv` commands. It adds support for setting cache control and expires header to S3 objects. ([#318](https://github.com/peak/s5cmd/pull/318)) [@tombokombo](https://github.com/tombokombo)
- Added `--force-glacier-transfer` flag to `cp` command. It forces a transfer request on all Glacier objects. ([#206](https://github.com/peak/s5cmd/issues/206))
- Added `--source-region` and `destination-region` flags to `cp` command. It allows overriding bucket region. ([#262](https://github.com/peak/s5cmd/issues/262)) [@kemege](https://github.com/kemege)
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM golang:1.22-alpine AS build
FROM golang:1.24-alpine AS build
COPY . /s5cmd/
RUN apk add --no-cache git make && \
cd /s5cmd/ && \
Expand Down
17 changes: 11 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ binaries for Linux, macOS and Windows.

For macOS, a [homebrew](https://brew.sh) tap is provided:

brew install peak/tap/s5cmd
brew install s5cmd

### Unofficial Releases (by Community)
[![Packaging status](https://repology.org/badge/tiny-repos/s5cmd.svg)](https://repology.org/project/s5cmd/versions)
Expand Down Expand Up @@ -136,7 +136,7 @@ then filtering the results in-memory. For example, for the following command;

s5cmd cp 's3://bucket/logs/2020/03/*' .

first a `ListObjects` request is send, then the copy operation will be executed
first a `ListObjects` request is sent, then the copy operation will be executed
against each matching object, in parallel.


Expand Down Expand Up @@ -315,8 +315,7 @@ more details and examples on `s5cmd run` are presented in a [later section](./RE
Will copy all the matching objects to the given S3 prefix, respecting the source
folder hierarchy.

⚠️ Copying objects (from S3 to S3) larger than 5GB is not supported yet. We have
an [open ticket](https://github.com/peak/s5cmd/issues/29) to track the issue.
ℹ️ Objects larger than 5 GiB are automatically copied using multipart copy.

#### Using Exclude and Include Filters
`s5cmd` supports the `--exclude` and `--include` flags, which can be used to specify patterns for objects to be excluded or included in commands.
Expand Down Expand Up @@ -393,6 +392,12 @@ rm s3://bucket/2020/03/19/file2.gz
mv s3://bucket/2020/03/18/file1.gz s3://bucket/2020/03/18/original/file.gz
```

> **Note**
> If your S3 object keys contain characters that are normally interpreted as wildcards (`*`, `?`, `[`), use the `--raw` flag in your commands to prevent wildcard expansion:
> ```
> cp --raw 's3://bucket/path/file[1].gz' .
> ```

#### Sync
`sync` command synchronizes S3 buckets, prefixes, directories and files between S3 buckets and prefixes as well.
It compares files between source and destination, taking source files as **source-of-truth**;
Expand Down Expand Up @@ -499,7 +504,7 @@ will output

cp s3://bucket/pre/file1.gz s3://another-bucket/file1.gz
...
cp s3://bucket/pre/last.txt s3://anohter-bucket/last.txt
cp s3://bucket/pre/last.txt s3://another-bucket/last.txt

however, those copy operations will not be performed. It is displaying what
`s5cmd` will do when ran without `--dry-run`
Expand Down Expand Up @@ -571,7 +576,7 @@ path-style.
`s5cmd` uses an exponential backoff retry mechanism for transient or potential
server-side throttling errors. Non-retriable errors, such as `invalid
credentials`, `authorization errors` etc, will not be retried. By default,
`s5cmd` will retry 10 times for up to a minute. Number of retries are adjustable
`s5cmd` will retry 10 times for up to a minute. Number of retries is adjustable
via `--retry-count` flag.

ℹ️ Enable debug level logging for displaying retryable errors.
Expand Down
Loading
Loading