Skip to content

fix: scrape.sh incremental auto-save + chmod permission fix + branch threading#74

Open
tamara-builds wants to merge 1 commit into
mainfrom
fix/scrape-incremental-autosave
Open

fix: scrape.sh incremental auto-save + chmod permission fix + branch threading#74
tamara-builds wants to merge 1 commit into
mainfrom
fix/scrape-incremental-autosave

Conversation

@tamara-builds

Copy link
Copy Markdown
Collaborator

Summary

  • Adds a 30-minute background auto-save loop to scrape.sh (mirroring actions/extract's existing pattern) so a runner crash or lost connection mid-scrape only costs the last ~30 minutes of progress instead of the entire run. Prompted directly by FL's 21-hour self-hosted scrape that lost all progress to "runner lost communication with the server."
  • Fixes a chmod permission bug: the tarball-packaging step was running chmod -R 755 against the Docker-root-owned source directory instead of the already-copied runner-owned output. On Linux runners this fails with Operation not permitted and (under set -e) silently kills the rest of the script — surfacing as a false "📦 Nightly fallback" job summary with all-N/A stats, even when real data landed and got committed fine. Found live on WI and SD.
  • Adds a "Commit Content" section to the job summary showing insertions/deletions from the actual commit, so "re-scraped the same data with fresh UUIDs, no net new content" (e.g. an out-of-session state) is distinguishable at a glance from "genuinely new content" or "lost data" without having to dig through commit stats after the fact.
  • Threads branch through as a new argument to scrape.sh so the new auto-save loop pushes to the correct branch instead of assuming main.

Test plan

  • Dispatch a small, fast state (not FL) after merge to confirm the auto-save loop and chmod fix work end-to-end in a real run
  • Confirm the "Commit Content" summary renders correctly
  • Only then point FL (and its tinyproxy run) at these changes

🤖 Generated with Claude Code

…reading

Adds a 30-minute background auto-save loop to scrape.sh, mirroring the
existing pattern in actions/extract, so a runner crash or lost connection
mid-scrape (e.g. FL's 21-hour run that lost all progress) only costs the
last ~30 minutes instead of the entire run. Also fixes a chmod permission
bug where the tarball-packaging step operated on Docker-root-owned files
instead of the already-copied runner-owned output, silently killing the
script under set -e on Linux runners (surfacing as a false "Nightly
fallback" job summary despite real data landing) -- and threads the
branch input through to scrape.sh so the new auto-save pushes to the
right branch instead of assuming main.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant