Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

## Overview

`data-liberation-agent` extracts content from closed web platforms (Wix, Squarespace, Webflow, Shopify, Weebly, Hostinger, HubSpot, GoDaddy Websites & Marketing) and produces WordPress-compatible WXR files. All eight platform adapters are implemented.
`data-liberation-agent` extracts content from closed web platforms (Wix, Squarespace, Webflow, Shopify, Instagram, Weebly, Hostinger, HubSpot, GoDaddy Websites & Marketing) and produces WordPress-compatible WXR files. All nine platform adapters are implemented.

Three entry points — MCP server (11 tools), CLI (`src/cli.ts`), and Claude Code plugin (`claude plugin add .`) — all share `src/lib/` and `src/adapters/`. The plugin just wraps the MCP server.

Expand Down Expand Up @@ -44,6 +44,7 @@ When `adminToken` is present, `shopifyAdapter.extract` fetches products via the
- `classifyUrl` types: `homepage`, `post`, `product`, `gallery`, `event`, `page` (no `category`/`author`/`other`)
- Media filename collision handling uses numeric suffixes (`-2`, `-3`), not hashes
- `detect-platform` uses domain-level URL patterns and HTTP fingerprinting (headers + HTML markers) — no path-based detection
- Instagram adapter requires `cdpPort` for both `discover` and `extract` — Instagram has no public sitemap or unauthenticated API. Discover scrolls the profile and intercepts GraphQL responses; extract visits each post via `?img_index=N` for carousel slide capture.
- Shopify variant weights are normalized to kilograms regardless of source unit (`kg`, `g`, `lb`, `oz`) via `normalizeWeightToKg`
- Shopify simple-product sale price uses `compareAtPrice > price` semantics — when set, `compareAtPrice` becomes `regularPrice` and `price` becomes `salePrice`
- `WooProduct` has first-class `seoTitle`, `seoDescription`, `costOfGoods` fields plus an open `meta` record; the CSV builder emits `meta:_yoast_wpseo_title`, `meta:_yoast_wpseo_metadesc`, `meta:_wc_cog_cost` columns always, plus `meta:<key>` columns for any custom keys
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ A maintainer (human or AI with repo access) will review the PR. If the discovery

```
src/
adapters/ Platform-specific extraction (wix.ts, squarespace.ts, webflow.ts, shopify.ts, godaddy-wm.ts, shared.ts)
adapters/ Platform-specific extraction (wix.ts, squarespace.ts, webflow.ts, shopify.ts, instagram.ts, godaddy-wm.ts, shared.ts)
lib/
extraction/ WXR builder, sitemap, media, detection, extraction log
import/ WordPress REST API importer, WooCommerce CSV
Expand Down
28 changes: 28 additions & 0 deletions DISCOVERIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,34 @@ Changed `extractWixProduct()` to check both cases for offers (`obj.offers || obj

### Why it's better than the previous approach

---

## 2026-04-03 — Instagram data extraction via CDP and GraphQL interception

**Found by:** Claude + human contributor (live testing against a real 308-post Instagram profile)
**During:** Building Instagram support for the data-liberation-agent
**Type:** API endpoint | content type | architecture

### What I found

Instagram is a React app that communicates via GraphQL queries to `https://www.instagram.com/graphql/query/`. By connecting to an authenticated browser via CDP and intercepting responses during profile scroll, we capture structured JSON for every post. Key discoveries:

1. **Carousel slide direct access via `?img_index=N`**: Individual carousel slides can be loaded by appending `?img_index=1`, `?img_index=2`, etc. to the post URL. This is significantly more reliable than clicking through carousel arrows in the DOM.

2. **Carousel DOM has 3 `<li>` elements**: Instagram keeps previous, current, and next slides in the DOM simultaneously. Deduplication by Instagram media ID (the numeric prefix in CDN URLs like `/12345_67890.jpg`) is required to avoid capturing the same image from adjacent preloaded slides.

3. **Scroll-based pagination is more reliable than direct GraphQL**: Making direct `fetch()` calls to the GraphQL endpoint triggers rate limiting. Scrolling the profile with 2-3 second delays lets Instagram's own IntersectionObserver trigger pagination naturally.

4. **WordPress.com REST API doesn't support writes with app passwords**: Returns 401 for POST operations. XML-RPC (`wp.uploadFile`, `wp.newPost`) works correctly. The `post_date` must be sent as a `<string>` in `"YYYY-MM-DD HH:MM:SS"` format — WordPress ignores `<dateTime.iso8601>` typed values.

### How it works

Implemented in `src/adapters/instagram.ts`: `discover()` scrolls the profile and intercepts GraphQL responses to build a post inventory; `extract()` visits each post (walking `?img_index=N` for carousel slides, deduped by Instagram media ID), downloads media via the shared `downloadMedia` helper, and emits `wp:image` / `wp:video` / `wp:gallery` Gutenberg blocks into the WXR. Hashtags become WordPress tags; `@mentions` and `#hashtags` in captions are linkified. The WXR is then imported via the generic WP importer in `src/lib/import/`.

### Why it's better than the previous approach

Instagram's built-in data export takes days, provides lower-resolution images, and has no location data. The CDP approach captures everything in real-time at full resolution with complete metadata.

Tested against 2 live Wix Stores. Before: price="" and images=0 on every product. After: bestiehugs.com recovers price=200 (ILS) and 5 images; nue-modern.com recovers price=1295 (GBP) and 11 images. Stock status also corrected (bestiehugs hoodie correctly shows OutOfStock).

## 2026-04-16 — Wix /product-page/ URLs misclassified as pages
Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,12 @@ This tool extracts all content from closed platforms — posts, pages, media, na
| **Squarespace** | Ready | [`prompts/squarespace.md`](./prompts/squarespace.md) |
| **Webflow** | Ready | [`prompts/webflow.md`](./prompts/webflow.md) |
| **Shopify** (blog/pages/products) | Ready | [`prompts/shopify.md`](./prompts/shopify.md) |
| **Instagram** (posts, carousels, videos) | Ready | [`prompts/instagram.md`](./prompts/instagram.md) |
| **Weebly** (blog/pages/products) | Ready | — |
| **Hostinger Website Builder** (blog/pages/products) | Ready | — |
| **GoDaddy Websites & Marketing** (pages/blog) | Ready | [`prompts/godaddy-wm.md`](./prompts/godaddy-wm.md) |

All eight platforms have MCP adapters with full extraction support including products (exported as WooCommerce-compatible CSV). GoDaddy Websites & Marketing is pages + blog only in v1; GoDaddy Online Store (OLS) product support is planned for v1.1.
All nine platforms have MCP adapters with full extraction support. Wix/Squarespace/Webflow/Shopify/Weebly/Hostinger/HubSpot export products as WooCommerce-compatible CSV; Instagram exports posts as a WXR with `wp:image` / `wp:gallery` / `wp:video` blocks and hashtag-derived tags. GoDaddy Websites & Marketing is pages + blog only in v1; GoDaddy Online Store (OLS) product support is planned for v1.1.

## AI tool integration

Expand Down
2 changes: 1 addition & 1 deletion docs/mcp.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ Detect the platform of a website.
|-----------|----------|-------------|
| `url` | yes | The URL of the website to detect |

Returns: `platform` (wix, squarespace, webflow, shopify, godaddy-wm, or unknown), `confidence` (high/medium/low), `signals` (what was detected).
Returns: `platform` (wix, squarespace, webflow, shopify, instagram, weebly, hostinger, hubspot, godaddy-wm, or unknown), `confidence` (high/medium/low), `signals` (what was detected).

### liberate_discover

Expand Down
86 changes: 86 additions & 0 deletions prompts/instagram.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Instagram to WordPress Migration Prompt

Copy everything below this line and paste it into your AI assistant (Claude, ChatGPT, Gemini, etc.).

---

I want to migrate my Instagram posts to WordPress. My Instagram username (or profile URL) is: **[PASTE YOUR USERNAME OR PROFILE URL HERE]**

I have (or will create) a WordPress site. Please help me migrate using the playbook at https://github.com/Automattic/data-liberation-agent — read AGENTS.md first for full instructions.

**Important**: Instagram has no public sitemap and no unauthenticated API. The Instagram adapter requires an authenticated browser session connected via Chrome DevTools Protocol (CDP). I'll need Chrome (or another Chromium browser) launched with `--remote-debugging-port` and logged into Instagram **before** we run discovery or extraction.

## Step 1: Inspect

```bash
npm run inspect -- https://www.instagram.com/[MY-USERNAME]/
```

This confirms the platform is detected as Instagram and shows what the adapter expects.

## Step 2: Launch Chrome with remote debugging and log in to Instagram

```bash
# macOS:
/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome \
--remote-debugging-port=9222 \
--user-data-dir="$HOME/.data-liberation/cdp-profile/chrome" \
--restore-last-session

# Linux:
google-chrome --remote-debugging-port=9222

# Windows:
"C:\Program Files\Google\Chrome\Application\chrome.exe" --remote-debugging-port=9222
```

In that Chrome window, navigate to `https://www.instagram.com/` and log in. Confirm I'm logged in before continuing.

## Step 3: Discover all my posts

```bash
npm run liberate -- https://www.instagram.com/[MY-USERNAME]/ --output ./output --cdp-port 9222 --verbose --discover-only
```

The Instagram adapter will:
- Open a new tab in the connected Chrome window and navigate to my profile
- Scroll the profile, intercepting Instagram's GraphQL responses to capture post metadata (captions, dates, locations, hashtags, photo/video/carousel type, slide counts)
- Build an inventory at `output/inventory.json`

**If it stalls or gets rate limited**, add `--delay 3000` for a gentler scroll cadence.

Show me the inventory summary (counts by type, total post count, profile metadata) and wait for my approval before extracting.

## Step 4: Extract content and download media

```bash
npm run liberate -- https://www.instagram.com/[MY-USERNAME]/ --output ./output --cdp-port 9222 --verbose
```

For each post the adapter will:
- Visit the post URL in the connected browser
- Capture full-resolution images from the intercepted media API responses
- For carousels, walk `?img_index=N` for every slide and dedupe by Instagram media ID
- Download all images and videos locally (Instagram CDN URLs expire — this must happen promptly after discovery)
- Build a `wp:image` block for photos, `wp:video` for videos, and `wp:gallery` for carousels
- Convert hashtags to WordPress tags and `@mentions` / `#hashtags` in captions to links
- Append a "View on Instagram" source link to each post

If extraction is interrupted, resume with `--resume`.

## Step 5: Verify

```bash
npm run verify -- ./output
```

Show me the verification report and flag any posts with missing media or failed extractions.

## Step 6: Import to WordPress

```bash
npm run setup -- --site [MY-WORDPRESS-SITE] --username [MY-USERNAME] --token [APP-PASSWORD]
npm run import -- ./output --site [MY-WORDPRESS-SITE] --username [MY-USERNAME] --token [APP-PASSWORD]
```

Work methodically — do one step at a time, show me progress, and wait for my go-ahead before moving to the next step.
Loading