Automattic · joeboydston · Apr 3, 2026 · simison · Apr 3, 2026 · simison
diff --git a/AGENTS.md b/AGENTS.md
@@ -11,7 +11,7 @@ This file is the entry point for any AI agent using this repository. Read it bef
 3. **User prompts** — non-technical users paste these into their AI to drive the whole migration
 4. **A living playbook** — this repo improves itself through AI-contributed discoveries
 
-Currently supports: **Wix**. Squarespace, Webflow, and Shopify are planned.
+Currently supports: **Wix** and **Substack**. Squarespace, Webflow, and Shopify are planned.
 
 ## If you're helping a user migrate from Wix
 
@@ -96,6 +96,57 @@ This approach works for any JavaScript-heavy platform, not just Wix.
 
 ---
 
+## If you're helping a user migrate from Substack
+
+### Step 1 — Understand the publication
+
+```bash
+node scripts/substack/discover.js <substack-url>
+```
+
+This queries Substack's public API to list all posts, categorize them (free/paid/podcast), and writes `output/inventory.json`. No browser needed.
+
+### Step 2 — Extract all content
+
+```bash
+node scripts/substack/extract.js <substack-url>
+```
+
+This fetches each post via the API. For **paid posts**, the API only returns the free preview. To get full paid content:
+
+1. Ask the user to export from Substack: Settings > Exports > Create new export
+2. They'll get a ZIP with a CSV file
+3. Unzip it and pass the CSV:
+
+```bash
+node scripts/substack/extract.js <substack-url> --csv-export posts.csv
+```
+
+The extractor merges API metadata (rich) with CSV content (complete), giving you the best of both.
+
+**If scripts aren't available**, you can extract directly:
+1. Fetch the archive: `<substack-url>/api/v1/archive?sort=new&limit=50&offset=0`
+2. For each post: `<substack-url>/api/v1/posts/<slug>` — returns full JSON with `body_html`
+3. Download images by unwrapping CDN URLs: `substackcdn.com/image/fetch/.../https://substack-post-media.s3.amazonaws.com/...` — the real URL is embedded after the transform params
+
+### Step 3 — Import to WordPress.com
+
+Same as Wix — the import script is platform-agnostic:
+
+```bash
+node scripts/import.js --site <wordpress-site> --token <app-password>
+```
+
+### Step 4 — Handle Substack-specific concerns
+
+After import, help the user with:
+- **Subscribers**: Export from Substack (Settings > Exports), import into their email service or WordPress.com newsletter
+- **Paid subscriptions**: Complex — involves Stripe migration. Discuss options before acting.
+- **Redirects**: If they have a custom domain, set up redirects from `/p/<slug>` to new WordPress paths. If on `*.substack.com`, they can't redirect — suggest a pinned farewell post.
+- **Podcast hosting**: Audio files need to be re-hosted if they have podcast episodes.
+
+---
+
 ## Platform-specific barriers
 
 ### Wix
@@ -109,6 +160,20 @@ This approach works for any JavaScript-heavy platform, not just Wix.
 | Anti-bot blocking | Use real browser via Playwright (not fetch/curl); add delays |
 | Dynamic pages (CMS collections) | Query Wix's `/_api/wix-data-server/` endpoints directly |
 
+### Substack
+
+| Problem | Solution |
+|---|---|
+| Paid content truncated in API | Use Substack's CSV export (`--csv-export`) for full body HTML |
+| No official API | Undocumented `/api/v1/posts` and `/api/v1/archive` endpoints are public |
+| Images behind CDN wrapper | Unwrap `substackcdn.com/image/fetch/.../` URLs to get original S3 paths |
+| URL structure (`/p/slug`) | Generate redirect map; set WordPress permalinks to match if possible |
+| Subtitles not in WordPress | Add as styled first paragraph or use a subtitle plugin |
+| Subscriber migration | Separate CSV export; requires email service setup |
+| Paid subscriber migration | Stripe account transfer needed; no automated path |
+| Comments not exportable | Not included in any export; consider a scraping approach if critical |
+| Notes/threads | Newer feature; may not export cleanly via API |
+
 ---
 
 ## How to contribute improvements back
@@ -149,11 +214,15 @@ data-liberation-agent/
 ├── DISCOVERIES.md         ← log of community-contributed findings
 ├── package.json
 ├── prompts/
-│   └── wix.md             ← what users paste into their AI for a Wix migration
+│   ├── wix.md             ← what users paste into their AI for a Wix migration
+│   └── substack.md        ← what users paste into their AI for a Substack migration
 ├── scripts/
 │   ├── wix/
 │   │   ├── discover.js    ← inventory the Wix site (sitemap + categorization)
 │   │   └── extract.js     ← extract all content via network interception
+│   ├── substack/
+│   │   ├── discover.js    ← inventory the Substack publication via API
+│   │   └── extract.js     ← extract content via API + CSV export
 │   └── import.js          ← publish to WordPress.com via REST API (platform-agnostic)
 ├── examples/
 │   ├── wix-api-blog-post.json    ← example of Wix internal API response

diff --git a/DISCOVERIES.md b/DISCOVERIES.md
@@ -6,6 +6,37 @@ AI agents: when you contribute an improvement, add an entry here. See [CONTRIBUT
 
 ---
 
+## 2026-04-02 — Substack public API for content extraction
+
+**Found by:** Claude + human contributor (Joe Boydston, Newspack team)
+**During:** Adding Substack as a new migration platform
+**Type:** API endpoint | new platform
+
+### What I found
+
+Substack has undocumented but publicly accessible API endpoints that return rich JSON for any publication without authentication:
+
+- `/api/v1/archive?sort=new&limit=50&offset=0` — paginated list of all posts with metadata
+- `/api/v1/posts/<slug>` — full post data including `body_html`, cover image, subtitle, audience tier, word count, reactions, and author bylines
+
+Images are served through a CDN wrapper at `substackcdn.com/image/fetch/w_XXXX,.../https://substack-post-media.s3.amazonaws.com/...`. The original full-resolution URL is embedded in the CDN path after the transform parameters.
+
+### How it works
+
+The discover script paginates through `/api/v1/archive` to build a complete inventory. The extract script then fetches each post individually via `/api/v1/posts/<slug>` for rich metadata and full HTML body. For paid posts (where the API only returns free preview content), the extractor can merge in full HTML from Substack's official CSV export via `--csv-export`.
+
+No browser or authentication needed — this runs entirely via `fetch()`.
+
+### Why it's better than the previous approach
+
+Unlike Wix (which requires Playwright to intercept internal API calls during page load), Substack's public API means:
+- No browser dependency — extraction is fast and lightweight
+- Clean, structured JSON with semantic fields
+- Dual-source strategy (API + CSV) ensures complete content including paid posts
+- Rate limiting is lenient with 500ms delays between requests
+
+---
+
 ## 2026-03-31 — Wix Dashboard API reverse engineering via CDP
 
 **Found by:** Claude + human contributor (live probing against Brave browser)

diff --git a/README.md b/README.md
@@ -17,10 +17,30 @@ This repo gives people a prompt they can paste into any AI assistant (Claude, Ch
 | Platform | Status | Prompt |
 |---|---|---|
 | **Wix** | Ready | [`prompts/wix.md`](./prompts/wix.md) |
+| **Substack** | Ready | [`prompts/substack.md`](./prompts/substack.md) |
 | Squarespace | Planned | — |
 | Webflow | Planned | — |
 | Shopify (blog/pages) | Planned | — |
 
+## Quick start (Substack)
+
+```bash
+# 1. Install dependencies
+npm install
+
+# 2. Discover all content on your Substack
+node scripts/substack/discover.js https://yourpub.substack.com
+
+# 3. Extract all content (uses Substack's public API)
+node scripts/substack/extract.js https://yourpub.substack.com
+
+# 4. For paid posts, export from Substack first, then:
+node scripts/substack/extract.js https://yourpub.substack.com --csv-export posts.csv
+
+# 5. Import to WordPress.com
+node scripts/import.js --site your-wp-site --token YOUR_APP_PASSWORD
+```
+
 ## Quick start (Wix)
 
 ```bash
@@ -58,6 +78,16 @@ This means the playbook gets smarter with every migration.
 - [ ] Wix Stores / WooCommerce migration
 - [ ] Wix Bookings migration
 
+### Substack
+- [x] Public API extraction (free content)
+- [x] CSV export support (paid content)
+- [x] Image CDN URL unwrapping
+- [x] WordPress.com REST API import
+- [x] Migration prompt for non-technical users
+- [ ] Podcast episode audio migration
+- [ ] Subscriber list import
+- [ ] Paid subscription migration (Stripe)
+
 ### General
 - [x] WordPress.com REST API import script
 - [ ] WordPress Studio local-first workflow

diff --git a/package.json b/package.json
@@ -1,7 +1,7 @@
 {
   "name": "data-liberation-agent",
   "version": "0.1.0",
-  "description": "AI-assisted migration from Wix to WordPress.com",
+  "description": "AI-assisted migration from closed platforms to WordPress.com",
   "type": "module",
   "scripts": {
     "discover": "node scripts/discover.js",

diff --git a/prompts/substack.md b/prompts/substack.md
@@ -0,0 +1,84 @@
+# Substack to WordPress.com Migration Prompt
+
+Copy everything below this line and paste it into your AI assistant (Claude, ChatGPT, Gemini, etc.).
+
+---
+
+I want to migrate my publication from Substack to WordPress.com. My Substack URL is: **[PASTE YOUR SUBSTACK URL HERE]**
+
+I have (or will create) a WordPress.com account. Please help me migrate using the playbook at https://github.com/Automattic/data-liberation-agent — read AGENTS.md first for full instructions.
+
+Here's what I need you to do:
+
+## Step 1: Inventory my publication
+
+- Use the Substack API to list all my posts: fetch `[MY SUBSTACK URL]/api/v1/archive?sort=new&limit=50&offset=0` (paginate through all results)
+- Categorize each post: free, paid, podcast, thread, page
+- Note my publication name, description, and about page
+- Count total posts, paid vs. free breakdown, and any podcast episodes
+- Show me the inventory and wait for my approval before proceeding
+
+## Step 2: Export and extract content
+
+**For free content:**
+- Fetch each post's full content via the API: `[MY SUBSTACK URL]/api/v1/posts/[SLUG]`
+- This gives you the full HTML body, metadata, cover images, and dates
+
+**For paid content (if I have any):**
+- Ask me to go to Substack Settings > Exports > Create new export
+- I'll download the ZIP and give you the CSV file
+- Use the CSV's `body_html` column for paid post content — the API only gives the free preview
+
+**For all content:**
+- Download every image — Substack wraps images through `substackcdn.com/image/fetch/...`. Extract the original URL from inside the CDN wrapper to get full-resolution images
+- Preserve for each post: title, subtitle, URL slug, publish date, categories/sections, cover image, audience (free/paid), word count
+
+## Step 3: Set up WordPress.com
+
+I need to create/have a WordPress.com site. Help me:
+- Recommend a theme that works well for newsletters/blogs
+- Create categories based on any Substack sections I have
+- Configure basic settings: site title, tagline, permalink structure matching `/p/[slug]` if possible (for easier redirects)
+
+For connecting to WordPress.com, I can either:
+- Enable MCP at wordpress.com/me/mcp and connect you directly
+- Generate an Application Password at wordpress.com/me/security/application-passwords
+
+Tell me which you need.
+
+## Step 4: Publish everything
+
+In this order:
+1. Upload all images to the WordPress media library (needed first to get new URLs)
+2. Create all posts with correct dates, featured images, and content
+3. Rewrite all internal links and image `src` attributes from Substack URLs to new WordPress URLs
+4. Set up navigation and homepage
+
+**Special handling needed:**
+- **Subtitles**: Substack has a subtitle field — add it as the first line of the post in `<em>` tags, or use a subtitle plugin if available
+- **Paid content**: If I want to gate content on WordPress, ask me which membership/paywall plugin to use (WooCommerce Memberships, Restrict Content Pro, etc.) and which posts should stay gated
+- **Podcast episodes**: If I have podcast content, check if the audio files are downloadable from Substack and re-host them. Set up a podcasting plugin if needed.
+
+## Step 5: Handle subscribers
+
+- Ask me to export my subscriber list from Substack (Settings > Exports)
+- Help me choose an email service (Mailchimp, ConvertKit, ActiveCampaign, etc.) or the built-in WordPress.com newsletter feature
+- Import the subscriber list
+
+**Note about paid subscribers**: If I have paid subscribers through Substack, this is complex — I may need to:
+- Set up a new payment system (Stripe on WordPress)
+- Communicate with subscribers about the move
+- Plan a transition period
+
+Tell me what my options are before doing anything.
+
+## Step 6: Verify and redirect
+
+When done:
+- Give me a URL mapping table: old Substack URL → new WordPress URL (for setting up redirects)
+- Check that no images still point to `substackcdn.com`
+- If I use a custom domain on Substack, help me set up 301 redirects from `/p/[slug]` to the new WordPress paths
+- If I'm on `[name].substack.com`, note that I can't set up redirects — but I can add a pinned post on Substack linking to my new site
+- List anything that needs manual attention
+
+Work methodically — do one step at a time, show me progress, and wait for my go-ahead before moving to the next step. If you hit something unexpected, tell me what you found rather than guessing.