Integrate upstream vosfs and fsspec-cli as canfar data

[shinybrar]

Status

Implementation complete on feat/vosfs-support. All child tickets #151-#155 are closed, every implementation wave passed GPT-5.6 Sol High review, and the final GPT-5.6 Sol xHigh mega-review/fix suite is clean. vosfs v0.6.0 and fsspec-cli-v0.5.0 are normal CANFAR dependencies. Cross-source mv and recursive removal remain deliberately unsupported.

User story

As a CANFAR user, I want one canfar data command group that can operate on the named storage services configured for any Science Platform Server and on my local filesystem, so I can inspect and move data without CANFAR owning another VOSpace protocol stack or command implementation.

Problem

The previous specification and unmerged feat/vospace branch implemented a VOSpace backend, path grammar, transfer engine, CLI suite, FUSE command, and output conventions inside CANFAR. That ownership is now obsolete:

• vosfs owns the asynchronous OpenCADC VOSpace fsspec backend.
• fsspec-cli owns the embedded POSIX-shaped command library.
• CANFAR already owns Science Platform Server selection, registry discovery, Authentication Records, runtime credential precedence, and CLI composition.

The remaining work is the CANFAR integration boundary. The upstream command research and implementation wave is complete, with one rejected capability and one host-policy gate that CANFAR must reflect explicitly.

Proposed solution

flowchart TD
    Registry["IVOA registry resource records"] --> Config["Server.storage mapping"]
    Auth["CANFAR Authentication Record"] --> Factory["Private per-invocation source factory"]
    Config --> Factory
    Factory --> VOSFS["vosfs.VOSpaceFileSystem"]
    Local["local via AsyncFileSystemWrapper"] --> App["fsspec-cli App"]
    VOSFS --> App
    App --> CLI["canfar data"]

Loading

CANFAR owns only:

1. storage configuration and registry pairing;
2. credential materialization into a fresh async VOSpaceFileSystem;
3. source mapping and composition of App(sources).typer_app at canfar data.

CANFAR does not own a VOSpace backend, transfer engine, command implementations, public storage Python API, FUSE layer, signed-URL extension, or backend-specific native-move path.

Public configuration

Add a VOSpace service model:

class VOSpaceService(BaseModel):
    uri: AnyUrl
    url: AnyHttpUrl  # base service URL, without /capabilities

Add Server.storage: dict[str, VOSpaceService] with an empty default. Configuration schema version remains 1.

• The mapping key is the globally unique Storage Name exposed to fsspec-cli.
• local is reserved.
• Names must satisfy the upstream mapped-source constraints: non-empty; no colon, NUL, or newline; not starting with -.
• Duplicate Storage Names across Science Platform Servers are invalid.
• Every storage entry inherits its parent server's IDP credential.
• The model supports multiple entries per server, such as canSRC and canSRCs3.

This release automatically discovers one primary service per server and gives it the Server Name:

• CADC server → same-registry-namespace arc resource.
• SRCNet server → same-registry-namespace cavern resource.
• The endpoint comes from the registry record; never derive it by replacing /skaha in a URL.
• Additional services remain explicitly configured for now. Existing manual entries are preserved.

CLI contract

canfar data ls -lh canSRC:/path
canfar data cp -R local:/absolute/folder canSRC:/folder
canfar data cp canSRC:/file canfar:/file
canfar data rm canSRC:/file

The following destructive recursive surface is conditional on resolving the host-qualification gate below:

canfar data rm -R canSRC:/folder

• Mount the upstream core fsspec-cli application at canfar data.
• Flatten every configured Storage Name into the source mapping and always add local through fsspec's AsyncFileSystemWrapper.
• Suppress the CANFAR active-server banner for the entire group; stdout belongs exclusively to the embedded command.
• Keep explicit name:/absolute/path operands.
• Do not add :/path, bare local operands, an active alias, canfar storage, JSON/YAML modes, progress/confirmation wrappers, or the sign extension.
• Human-readable long listings use ls -lh or ll -h; standalone ls -h is not required.
• Cross-source mv remains an upstream source-free rejection. Cross-server movement is an explicit verified cp, followed by a separate user-requested rm only after the copy succeeds.
• Do not claim same-source mv for CANFAR VOSpace sources; the current native vosfs source form does not expose the exact async _mv seam required by fsspec-cli.
• If one-command relocation is ever justified, design it as a separately named, opt-in orchestration module with an explicit residual-state result; do not reinterpret portable mv or add it to this release.
• Users import vosfs directly for Python use; CANFAR exposes no public storage API.

Credential and lifecycle contract

Each source is a private async context-manager factory created from its parent server:

• reload configuration for each invocation;
• preserve existing runtime token/certificate precedence;
• refresh and persist an expired saved OIDC access token using existing CANFAR auth logic;
• validate saved or runtime X.509 material;
• pass only a current literal token or certificate path into VOSpaceFileSystem(endpoint_url, ..., asynchronous=True, skip_instance_cache=True);
• implement the upstream AsyncFilesystemSource contract as a callable returning a fresh async context manager;
• yield the fresh filesystem once and always await filesystem.aclose() on success, error, and cancellation;
• fail cleanly with an authentication hint, without a traceback or secret disclosure.

No CANFAR HTTPClient, refresh callback, or Authentication Record is passed into vosfs.

Upstream capability audit

Capability	Current upstream	Required disposition
Async VOSpace filesystem, list/info/get/put/mkdir/remove/copy hooks	available in v0.6.0	reuse
Embedded core commands and explicit mapped sources	available in fsspec-cli-v0.5.0	reuse
Long human listing	ls -lh / ll -h available	document; do not add standalone ls -h
Cross-source file copy through local staging	available	reuse
Recursive copy	implemented by #286 in PR #294; defaults enabled	reuse after fsspec-cli-v0.5.0
Recursive removal	implemented by #288 in PR #294; application capability defaults disabled	resolve the CANFAR host-qualification gate before enabling
Cross-source file move	locked source-free rejection by #285; #287 closed not planned	accepted omission; use explicit cp, then rm
Same-source file move	core command requires a source-declared async _mv; current Local wrapper and native vosfs forms do not provide it	do not advertise as a CANFAR workflow
Directory move	unavailable	explicit cp -R, then qualified rm -R
Native server-side move, FUSE, JSON/YAML, Rich progress, prompts, signed URLs	outside portable core	out of scope

Upstream completion record:

• Recursive copy research #283 and implementation #286: completed.
• Recursive removal research #284 and implementation #288: completed behind explicit application policy.
• Cross-source move research #285: completed with a locked rejection; implementation #287 closed as not planned.
• Final integration PR: shinybrar/vosfs #294, squash commit 9e5314d. Upstream recorded 574 core tests, 1,600 fsspec-cli tests, 984 installed-wheel tests, 31 native-vosfs wheel tests, strict docs, both builds, and Python 3.10-3.14/macOS CI.
• Tagged releases: v0.6.0 for vosfs and fsspec-cli-v0.5.0 for the embedded command library.

Recursive-removal integration decision

Decision for this release: omit recursive removal. Keep capabilities.recursion.remove=False and document the upstream application-disabled diagnostic. CANFAR always includes Local, and the current application-wide capability cannot safely qualify every configured target. A future source-scoped policy seam may reopen this without changing the portable command contract.

Transfer contract

• cp -R supports every admitted mapped pair: same-source, local↔remote, and remote↔remote.
• Recursive copy uses a bounded 10,000-entry manifest, one-file host-local staging, link/special-entry rejection, and complete source/destination metadata verification; it is not atomic or a snapshot.
• If enabled after the host-qualification gate, rm -R guards roots and dot paths, rejects links/special entries, removes leaves first, verifies final absence, and reports possible residue on partial failure.
• Cross-source mv remains unsupported because no admitted source supplies deletion-safe generation identity. File movement is explicit cp, verify success, then a separate rm.
• A future relocate-style workflow would require source-generation identity, destination verification, conditional deletion, and truthful residual-state reporting. Current VOSpace deletion cannot satisfy that interface, so it is not planned here.
• Directory movement is explicit cp -R, verify success, then rm -R only when recursive removal is enabled and qualified.
• Compute-near-data orchestration through canfar create may optimize future transfers, but is not part of generic fsspec-cli.

Packaging

vosfs and fsspec-cli are normal CANFAR dependencies. A standard installation includes data support:

pip install canfar

Published dependency metadata pins the tagged upstream sources directly:

• vosfs @ git+https://github.com/shinybrar/vosfs@v0.6.0
• fsspec-cli @ git+https://github.com/shinybrar/vosfs@fsspec-cli-v0.5.0#subdirectory=src/fsspec-cli

Do not add a data extra or document pip install canfar[data].

CANFAR delivery tickets

1. Configure named VOSpace services for Science Platform Servers
2. Discover primary VOSpace services from registry records
3. Materialize authenticated vosfs sources from CANFAR credentials
4. Embed fsspec-cli as the canfar data command group
5. Validate and document the data integration

Dependency order: configuration → discovery and authentication in parallel → upstream release and CLI embedding → validation/documentation.

Legacy branch audit and retirement record

Audited branch: feat/vospace at 08454ca8. It has no pull request and must never merge.

Useful requirements retained:

• named active and non-active server access;
• each server uses its own IDP credential;
• local↔remote and cross-server staged data movement;
• recursive copy, plus recursive removal as an explicit host-qualification decision;
• clean missing-credential behavior.

Implementation and UX deliberately discarded:

• CANFAR-owned VOSpace backend, XML models, transfer engine, and FUSE;
• /skaha → /arc or /cavern URL substitution;
• canfar storage, empty :/, and bare local operands;
• native /transfers move, beta flags, Rich progress, confirmation, dry-run, and CANFAR JSON/YAML wrappers.

Retirement completed on 2026-07-21: both the local and remote feat/vospace refs were deleted after this specification and its ticket graph were published. The audited implementation remains identified by full commit SHA 08454ca8dfe95a48fb73553ed8720bd16ab8ba00. feature/add-vospace was left untouched.

Test seam

CANFAR tests only its highest useful seams:

• configuration serialization, backward compatibility, global name validation, and reserved local;
• CADC ARC selection despite Vault, SRCNet Cavern pairing, preservation of manual entries, and malformed/unreachable storage handling;
• runtime credential precedence, OIDC refresh persistence, X.509 validation, missing/expired errors, and source cleanup;
• root CliRunner composition with fake sources, including source mapping, banner-free stdout, capability policy, rejected deprecated grammar, and the stable cross-source mv rejection.

The exhaustive command/backend matrices remain upstream. CANFAR may add one optional slow authenticated VOSpace smoke; deterministic CI does not require live credentials.

Acceptance criteria

• All five CANFAR child tickets close in dependency order.
• Required upstream research and implementation work is complete in PR #294; cross-source mv is explicitly not planned.
• vosfs v0.6.0 and fsspec-cli-v0.5.0 are tagged.
• Both tagged sources are pinned as normal CANFAR dependencies; no data extra exists.
• Recursive removal is explicitly omitted from this release; capabilities.recursion.remove remains disabled.
• canfar data exposes every configured Storage Name plus local.
• canfar data cp -R delegates to the released recursive-copy contract across admitted local/remote pairs.
• Cross-source and current VOSpace same-source mv are not advertised as supported CANFAR workflows.
• Data stdout never contains the CANFAR active-server banner.
• Runtime and saved credentials work without leaking CANFAR auth internals into vosfs.
• No CANFAR VOSpace backend, transfer engine, command implementation, FUSE layer, or public storage API is introduced.
• Deterministic lint, type, test, and docs gates pass.
• The authenticated smoke is isolated and skips before live I/O when valid credentials are unavailable; a live pass remains credential-dependent.
• CONTEXT.md defines VOSpace Service and Storage Name.

Implementation branch

• feat/vosfs-support

Epic Notes

• All five child tickets and all review-fix PRs are merged into feat/vosfs-support; no data extra exists.
• Recursive removal is disabled for this release; keep capabilities.recursion.remove=False.
• Final exact-head validation: pre-commit, Ruff, format, 811 deterministic tests, docs, and package build passed. The optional live smoke skipped before I/O because credentials were unavailable. The docs contributor lookup still logs a non-fatal GitHub API 403 warning.

---

Amended 2026-07-25

The acceptance criteria above are preserved as originally written. The items
below changed after this issue was filed, at the maintainer's request, and
shipped in #217.

• A public storage Python API now ships. This supersedes the non-goal
  "CANFAR exposes no public storage API" and the checked criterion
  "No ... public storage API is introduced". canfar.storage exposes
  identifiers(), filesystem(), sources(), and attribute access
  (from canfar.storage import arc, vault, local). CANFAR still owns no
  VOSpace backend and no transfer engine.
• Dependencies are vosfs@v0.8.0 and fsspec-cli-v0.7.0, not the v0.6.0 /
  v0.5.0 pins named above. See docs/agents/upstream-provenance.md.
• Storage Name became Storage Identifier in CONTEXT.md and throughout the
  code, so the criterion naming "Storage Name" reads as "Storage Identifier".