feat(adapters): add EmDash CMS platform adapter#50
Open
pmaiorana wants to merge 23 commits into
Open
Conversation
…rinting Adds the PathProbe signal type and an empty PATH_PROBES array. No consumers in this commit — the probe loop and EmDash entry land in follow-up commits / PR 2. PathProbe lets adapters detect a platform via HTTP HEAD against a stable admin or API path (e.g. /_emdash/admin returning 302), used as a fallback when URL/header/source-pattern detection returns 'unknown'. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the probe loop after SOURCE_SIGNALS in detectFromHttp(). For each PathProbe, issues a HEAD with redirect: 'manual' (so 302 status is preserved) and matches on response status. First match sets platform with confidence: 'high' and breaks. Probe loop only runs when URL/header/source-pattern detection returns 'unknown', so already-detected platforms pay no extra HTTP cost. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Move PATH_PROBES.length = 0 from the end of each test body into a shared afterEach hook so cleanup runs even when an assertion throws. Without this, a failed assertion in a probe test would leak the injected probe into subsequent tests, causing confusing cascade failures. Per code-review feedback on commit 7f7a120. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds an optional locationContains field to PathProbe. When present, the probe response's Location header must contain the substring for the probe to match. Tightens probes against false-positives where a non- platform site happens to return the same status code (e.g. a wildcard 302 redirect catching the probed path). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a test case where the probe response is 302 with NO Location
header. Verifies the `headers.get('location') || ''` fallback in the
probe loop correctly fails the locationContains check rather than
crashing on null.
Per code-review polish suggestion on commit 73842ad.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ers match Regression guard: the 'if (platform === "unknown")' gating in detectFromHttp ensures probes only fire when URL_PATTERNS, HTTP_SIGNALS, and SOURCE_SIGNALS all fail to identify the platform. Two tests assert fetch is called exactly once when source-pattern or header detection succeeds, proving the probe loop didn't fire. No production code change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three changes from final code review:
1. Same-origin guard on resolved probe URL. `new URL(probe.path, base)`
silently coerces protocol-relative ('//host/path') and absolute
('https://host/path') paths to a different origin. Without the guard,
a typo or hostile probe definition could send HEAD requests to
arbitrary hosts.
2. Wrap `response.text()` in its own try/catch. Without this, a
truncated/malformed body throw would jump to the outer catch and
skip the probe tier entirely — sites the probe could identify
would be misclassified as 'unknown'.
3. JSDoc warning on PATH_PROBES export documenting the test-injection
contract: exported for tests, production code must not mutate it.
Tests added for fix 1 (cross-origin probe rejected, fetchMock called
once) and fix 2 (probe tier runs and identifies platform when body
read throws).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Fixtures captured from: - yurulog.liberogic.jp (production EmDash, default theme, customized footer, ~20 Japanese posts with local media) - pmaioranatest.dashhost.cc (WP→EmDash migration, 173 posts, empty sitemap, external WP media URLs) Used by test/adapters/emdash.test.ts (added in follow-up commits). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds EmDash entries across all three signal tiers: - URL_PATTERNS: *.dashhost.cc (DashHost free-tier subdomains) - SOURCE_SIGNALS: <emdash-live-search>, /_emdash/api/ in attributes, class="emdash-*", class="ec-comment-form" - PATH_PROBES: /_emdash/admin returning 302 to /_emdash/admin/login The /_emdash/api/ pattern requires attribute context to avoid false-positives on tutorials or technical writing mentioning the path in code examples. Path probe is the universal fallback — verified on all 5 production EmDash sites regardless of theme. Updates the PR-1 'PATH_PROBES is empty' regression test to assert the EmDash entry is now present (the empty-array invariant was intentionally PR-1-only). Also updates PATH_PROBES infrastructure tests to account for the EmDash probe now being present at index 0: beforeEach/afterEach restores the array to its pre-test length (preserving the real EmDash entry), and each injected test-probe mock chain includes an extra non-matching response for the EmDash probe that runs first. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds src/adapters/emdash.ts with the PlatformAdapter contract (id, detect, discover, extract) and types (EmDashAdapterOpts, EmDashInventory). discover() and extract() stub with 'Not implemented' — filled in by follow-up commits. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…llback
Discover flow:
1. Fetch homepage (throw on non-2xx or fetch failure, bound to 5MB)
2. Extract site meta (title, tagline, language) from og:/html/meta tags
3. Primary: sitemap-index → posts/pages sitemaps via fetchSitemap helper
4. Fallback: crawl /posts archive page (default theme renders all posts
on one page) when sitemap is empty
5. Extract /pages/{slug} links from homepage nav for pages discovery
6. Merge + dedup, filter out taxonomy archives (/category, /tag),
listing pages (bare /posts), error pages (/404), and internal
routes (/_emdash/*)
7. Classify via existing classifyUrl() — /posts/{slug} → post,
/pages/{slug} → page
Also fixes yurulog-home.html fixture: lang="ja" → lang="en" to match
the test expectation that exercises html-lang extraction.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…havior Two regression tests: 1. Empty sitemap triggers /posts listing fallback (observed on DashHost production sites — sitemap regen lag means sitemap returns empty <urlset> despite content being live). 2. Taxonomy archives (/category, /tag), listing pages (bare /posts), error pages (/404), and internal routes (/_emdash/*) are filtered out of the inventory before extraction. No production code change. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ascade extractEmDashContent() tries containers in order: 1. .article-content (EmDash default theme — clean body) 2. .post-body / .entry-content / .post-content (WP-influenced themes) 3. <article> 4. <main> chrome-stripped 5. <body> chrome-stripped Widget stripping (aside, more-posts, ec-comments, comment form, widget-area, emdash-live-search, playground toolbar) applies to every tier. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
extractEmDashMetadata() extracts: - Title: h1.article-title → og:title → first h1 → <title> - Excerpt: p.article-excerpt → og:description → meta[name=description] - Date: article:published_time → JSON-LD datePublished → <time datetime> - Modified date: article:modified_time → JSON-LD dateModified JSON-LD parsing handles @graph wrappers and CDATA-wrapped content per the HubSpot adapter pattern. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
extractEmDashAuthors() returns an array of display names, supporting multi-author posts. Priority: 1. JSON-LD BlogPosting author (handles single object or array) 2. .byline-name / .post-byline-name / .featured-byline-name elements (default theme — supports multi-author via repeated .byline-name) 3. <meta name="author"> Returns [] when no author found (caller skips addAuthor registration). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
extractEmDashTaxonomy() scans all <a href> on the page for
/category/{slug} and /tag/{slug} patterns, using the link text as the
display name. Deduplicates case-insensitively by slug so widget and
inline references don't produce duplicates.
Per default theme: categories typically appear in sidebar widgets
(widget-categories__link) and article meta; tags in article meta and
post cards (meta-tag, post-tag, card-tag).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three helpers:
- extractEmDashMediaUrls(): recognizes /_emdash/api/media/file/{ULID}
as images regardless of file extension (some URLs have .png suffix,
some don't), passes through external CDN URLs with image extensions,
includes og:image and twitter:image.
- resolveRelativeUrls(): rewrites relative src/href/data-src to
absolute so WordPress media import can match attachment URLs. Strips
srcset/sizes attributes (default theme doesn't emit them, but custom
themes might — orphan-linking is worse than missing responsive
variants; full srcset support deferred per spec).
- stripDuplicateTitle(): removes the first <h1> when it matches the
post title (case-insensitive, whitespace-normalized), avoiding the
post_title + embedded h1 double-render.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
extract() composes the helpers (extractEmDashContent, extractEmDashMetadata, extractEmDashAuthors, extractEmDashTaxonomy, extractEmDashMediaUrls, resolveRelativeUrls, stripDuplicateTitle) into an extractPage callback and passes it to the shared runExtractionLoop. Per-page flow: 1. Fetch with 15s timeout + UA + 5MB cap, throw on non-2xx 2. Extract metadata, taxonomy, authors 3. Extract content → strip duplicate h1 → resolve relative URLs 4. Extract media URLs (local + external) 5. Score quality by text length 6. Return ExtractedPage for WxrBuilder Multi-author posts: first author becomes the primary WP author; additional authors are currently lost (WxrBuilder supports one author per post). Acceptable trade-off for v1. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds emdashAdapter to the static adapter list in src/mcp-server.ts and the adapter registries in src/ui/discover.tsx and src/ui/inspect.tsx, placed alphabetically between godaddy-wm and hostinger. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- README.md: add EmDash row to supported platforms table (alphabetical, between GoDaddy and Hostinger), update count from eight to nine - DISCOVERIES.md: document three findings from reverse-engineering EmDash for the adapter — /_emdash/admin as universal detection signal, empty-sitemap quirk on DashHost, /posts archive non-pagination - AGENTS.md (CLAUDE.md): update 'Non-obvious Details' to acknowledge path-based detection now exists alongside URL/header/source markers - skills/adapt/SKILL.md: add PATH_PROBES bullet so future adapter authors know the option exists Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The fixture is a faithful capture of yurulog.liberogic.jp, which is a Japanese-language blog (<html lang="ja">). Commit 29c3763 accidentally left the fixture at lang="en" from an intermediate test-driven edit. Test was already updated to expect 'ja'; this just brings the fixture in sync with reality. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three fixes surfaced during live validation: 1. detect-platform.ts: add User-Agent to path probe fetch. Some Cloudflare configs (DashHost-hosted EmDash sites observed) time out HEAD requests that arrive with Node.js / undici's default bot-ish UA. The homepage fetch in the same function already sets UA; the probe fetch was missing it. Without UA the probe can't detect themed sites where /_emdash/admin is the only signal. 2. ui/discover.tsx: two hardcoded "Supported: ..." strings (lines 383, 390) listed 8 platforms and missed EmDash. Updated both to include EmDash alphabetically. 3. CLAUDE.md: Overview sentence said 8 platforms; updated to 9 with EmDash in the alphabetical list. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four small fixes flagged in the combined PR1+PR2 review: 1. Remove unused imports (extractHeading, IMAGE_EXTENSIONS) in src/adapters/emdash.ts — leftover from the scaffold task. 2. Document the whitelist→blacklist deviation in extractEmDashMediaUrls. The implementation accepts external <img src> URLs unless they match known non-image extensions, rather than requiring a known image extension as the spec originally called for. Needed for Unsplash/Cloudflare-Images-style extensionless CDN URLs. 3. Centralize USER_AGENT constant in detect-platform.ts and apply to the homepage fetch in detectFromHttp. The path probe added in b4574a7 needed the UA against Cloudflare; the homepage fetch has the same fragility but was masked by detectFromHttp using redirect: 'follow' (which fires a GET, more permissive than HEAD). 4. Add TODO in both hubspot.ts and emdash.ts pointing at the duplicated parseJsonLdBlocks/isArticleLd helpers — extract to shared.ts when a third caller appears. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds support for Cloudflare's EmDash CMS.
Stacked on #49 — consumes the
PathProbeinfrastructure from PR 1. When #49 merges, the diff here reduces to just the EmDash-specific changes.Why
EmDash launched 2026-04-01 as an Astro-based CMS positioned by Cloudflare as a "spiritual successor to WordPress."
EmDash is unusually difficult to detect: no
<meta name="generator">, no reliable response header (Server-Timingis dev-only), and themes can strip every HTML source marker. The one universal signal —/_emdash/adminreturning 302 to/_emdash/admin/login— is what #49PathProbeinfrastructure exists to catch.Caveats
This is a weird one - no hard feelings if we decide to just close for now. EmDash is three weeks old and adoption is small, we only have five production sites to test, and two of them have no blog content to extract. This adapter handles what those sites expose; expect rough edges as the platform matures and more themes/plugins ship. Treat it as a starting point.
Tested against
/postslisting crawl). Full extraction → import verified./_emdash/adminas detection signal. Verifies the path-probe fallback works on themed sites.npx vitest run→ 435 passing (9 pre-existing failures intest/legacy-scripts.test.tsunchanged frommain).npx tsc --noEmitclean.Out of scope
Comments, products, custom plugin collections, and pagination on
/posts— documented inDISCOVERIES.mdand the adapter file header. None observed in shipped EmDash templates yet.🤖 Generated with Claude Code