feat: Firecrawl cloud browser driver + dedicated search/scrape tools#1
Open
developersdigest wants to merge 23 commits intomainfrom
Open
feat: Firecrawl cloud browser driver + dedicated search/scrape tools#1developersdigest wants to merge 23 commits intomainfrom
developersdigest wants to merge 23 commits intomainfrom
Conversation
…e contexts Three bugs fixed: 1. applyResolvedConfig() in resolved-config-refresh.ts unconditionally overwrote runtime.profile with a freshly resolved profile on every request, resetting cdpUrl back to "" and destroying the dynamic WSS URL that ensureBrowserAvailable() set from the firecrawl session. 2. ensureBrowserAvailable() early return path (existing session still alive) didn't re-apply cdpUrl, so if config refresh cleared it between requests, it stayed empty. 3. Route handlers (snapshot, tab context) read cdpUrl from the static profileCtx.profile instead of the runtime profile state. Added getCdpUrl() getter to ProfileContext that reads from runtime state. Also fixes API request body field names (ttlTotal/ttlWithoutActivity → ttl/activityTtl) to match the actual Firecrawl v2 browser API, and removes debug console.error left from prior debugging session.
The Firecrawl v2 search API returns { data: { web: [...] } } instead
of the v1 flat { data: [...] }. Handle both formats gracefully.
Capture both URLs from the Firecrawl v2 browser API response. Interactive allows human click/type; read-only is watch-only.
…sions focusTab and closeTab in selection ops used the static profile.cdpUrl (always empty for firecrawl) instead of the runtime cdpUrl set by ensureBrowserAvailable. Also fix listProfiles to detect active firecrawl sessions (no RunningChrome process) and report runtime cdpUrl.
Add FIRECRAWL_TOOL_NAMES constant and update applyFirecrawlKey to merge Firecrawl tool names into tools.alsoAllow (deduplicating against existing entries). Also append source=openclaw to the Firecrawl auth URL during browser auth flow to include the source parameter in the authorization request.
Trim verbose mocking, add assertions for alsoAllow tool merge, deduplication, and source=openclaw auth URL param.
Apply the same fc- prefix check to browser auth results, not just manual entry. Drop unnecessary async from generateCodeChallenge.
Prevents a stalled request from blocking the spinner indefinitely. The outer catch already handles network errors so this just aborts cleanly.
- ensureBrowserAvailable now re-reads the API key from disk instead of using the stale value captured at startup, so runtime config updates take effect without a restart. - Profile creation route and service now forward driver="firecrawl" instead of silently dropping it to undefined.
Importing resolveFirecrawlApiKey from web-fetch.ts pulled in external-content.js which broke the firecrawl availability test mock. Inline the trivial config+env lookup instead. Also fix formatting.
- Clean up stale firecrawl sessions before creating replacements to avoid leaking cloud resources - Add abort timeouts to firecrawl-browser.ts fetch calls (30s create, 10s delete) so stalled requests don't block indefinitely - Preserve base path prefix in resolveSearchEndpoint so reverse-proxied deployments resolve correctly - Skip CDP port allocation for firecrawl profiles since they use cloud sessions, not local ports
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Firecrawl as a first-class browser driver alongside
openclawandextension, plus dedicatedfirecrawl_searchandfirecrawl_scrapeagent tools.Browser driver (
driver: "firecrawl"):POST /v2/browser,DELETE /v2/browser/:id) — no SDK, rawfetch()firecrawlprofile when API key is present (matches search/scrape auto-enablement)cdpUrlmanagement:ensureBrowserAvailable()creates a cloud session, injects the WSS URL into profile state, and Playwright connects viaconnectOverCDP()liveViewUrlandinteractiveLiveViewUrlin browser status for human-in-the-loop/json/version)Search & scrape tools:
firecrawl_search: v2 search withdata.webresponse format handlingfirecrawl_scrape: scrape with markdown outputConfig & onboarding:
driver: "firecrawl"added to browser profile typeBug fixes included:
cdpUrlacross config hot-reload (was being clobbered byapplyResolvedConfig())getCdpUrl()in all route handlers (snapshot, agent actions, status)data.webresponse formatTest plan
pnpm test -- --run src/browser/firecrawl-browser.test.ts— session create/delete/reachabilitypnpm test -- --run src/browser/server-context.firecrawl-availability.test.ts— availability, reachability, stop, config-refresh clobber fixpnpm test -- --run src/browser/config.test.ts— firecrawl profile resolutionpnpm test -- --run src/agents/tools/firecrawl-tools.test.ts— search/scrape tools + v2 formatpnpm test -- --run src/commands/onboard-firecrawl.test.ts— onboarding flowbrowser start→browser open→browser snapshot→browser stopagainst live Firecrawl API