Skip to content

Conversation

@dnslin
Copy link

@dnslin dnslin commented Dec 29, 2025

Description / 描述

Implement a new multipart upload API that supports client-side chunked uploads with concurrent upload capability.

实现新的分片上传接口,支持客户端分片并发上传。

API Endpoint: PUT /api/fs/multipart?action=upload|complete

Features / 功能:

  • Uses File-Path header (consistent with /api/fs/put and /api/fs/form)
  • Two-step workflow: upload chunks → complete
  • Auto-initialize session on first chunk upload
  • Support concurrent chunk uploads
  • Idempotent chunk upload (re-uploading same chunk won't fail)
  • Support Overwrite and As-Task headers
  • Session auto-cleanup after expiration (2 hours)

Request Headers:

Header Required Description
File-Path Yes File path (URL encoded)
X-File-Size First chunk Total file size
X-Chunk-Size First chunk Chunk size
X-Chunk-Index Yes Chunk index (0-based)
X-Upload-Id After first Upload session ID
Overwrite No Overwrite existing file (default: true)
As-Task No Async upload on complete (default: false)

Motivation and Context / 背景

The existing upload APIs (/api/fs/put, /api/fs/form) don't support resumable uploads for large files. This new multipart upload API allows clients to:

  • Split large files into chunks and upload them concurrently
  • Resume interrupted uploads by re-uploading missing chunks
  • Better handle unstable network conditions

现有的上传接口不支持大文件的断点续传。新的分片上传接口允许客户端:

  • 将大文件分片并发上传
  • 通过重新上传缺失的分片来恢复中断的上传
  • 更好地处理不稳定的网络环境

Relates to #XXXX (如果有相关 issue 请填写)

How Has This Been Tested? / 测试

  • Manual testing with Python async client
  • Tested concurrent chunk uploads (4 parallel uploads)
  • Tested idempotent chunk upload (re-upload same chunk)
  • Tested session expiration cleanup
  • Tested complete upload with As-Task=true
  • Compiled successfully with go build

使用 Python 异步客户端进行手动测试,测试了并发上传、幂等性、会话过期清理等功能。

Checklist / 检查清单

Copilot AI review requested due to automatic review settings December 29, 2025 08:10
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR implements a new client-side multipart upload API that enables chunked file uploads with concurrent upload support and resumable capabilities. The implementation provides a two-step workflow (upload chunks → complete) with automatic session initialization and cleanup.

Key Changes:

  • New PUT /api/fs/multipart endpoint supporting upload and complete actions
  • Session-based chunk management with 2-hour expiration
  • Idempotent chunk uploads allowing retry of failed chunks
  • Support for concurrent chunk uploads via shared session state

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 11 comments.

File Description
server/router.go Adds the new /multipart endpoint with upload rate limiting and authentication middleware
server/handles/multipart.go Implements HTTP handlers for chunk upload and completion actions with header-based API design
internal/fs/multipart.go Core multipart session manager with chunk storage, merging, and cleanup logic
internal/model/multipart.go Data structures for multipart sessions, chunk metadata, and API responses

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

dnslin and others added 4 commits December 29, 2025 16:24
Co-authored-by: Copilot <[email protected]>
Signed-off-by: DnsLin <[email protected]>
Co-authored-by: Copilot <[email protected]>
Signed-off-by: DnsLin <[email protected]>
Co-authored-by: Copilot <[email protected]>
Signed-off-by: DnsLin <[email protected]>
Co-authored-by: Copilot <[email protected]>
Signed-off-by: DnsLin <[email protected]>
@dnslin
Copy link
Author

dnslin commented Dec 29, 2025

@copilot open a new pull request to apply changes based on the comments in this thread

Copy link
Member

@KirCute KirCute left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

能不能开始上传的时候就构建FileStream并调用驱动的上传,这样做至少有两个好处:

  1. 可以在首个分片的上传请求里携带文件哈希,在驱动触发秒传,如果秒传成功,后续分片就都不用上传了
  2. 可以同时进行用户终端->OpenListOpenList->网盘两个上传过程,这两个过程分别吃服务器的下行和上行带宽,不冲突,可以很大程度上减少总的上传时间

@hhsw2015
Copy link

这个是不是就解决了套了cloudflare域名后 上传大文件会报错的问题了

@dnslin
Copy link
Author

dnslin commented Dec 31, 2025

能不能开始上传的时候就构建FileStream并调用驱动的上传,这样做至少有两个好处:

  1. 可以在首个分片的上传请求里携带文件哈希,在驱动触发秒传,如果秒传成功,后续分片就都不用上传了
  2. 可以同时进行用户终端->OpenListOpenList->网盘两个上传过程,这两个过程分别吃服务器的下行和上行带宽,不冲突,可以很大程度上减少总的上传时间

空闲的时候我会补足这部分的实现

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Module: Server API and protocol changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants