Skip to content

DEPLOYMENT.md

Latest commit

 

History

History
599 lines (422 loc) · 18.7 KB

DEPLOYMENT.md

File metadata and controls

599 lines (422 loc) · 18.7 KB
Raw

Deployment Playbook

This is the single deploy guide for MetaCortex.

Use it for:

  • local verification before release
  • the first production deployment
  • the first production smoke test
  • the first weeks of rollout after launch

Current release contract

The deploy path in this repo currently assumes:

  • Firebase Cloud Functions 2nd Gen in us-central1
  • Firestore in Native mode
  • Firebase Blaze plan for production deploys
  • Firestore collection memory_vectors
  • Firestore collection memory_events for audit and observability
  • embedding output dimensions aligned with the Firestore vector indexes
  • runtime model choices supplied by the deploy dotenv files, using functions/.env.example as the tracked template
  • total MCP surface of 5 tools
  • public/browser toolset of 3 tools: remember_context, search_context, fetch_context
  • admin-only maintenance tools: deprecate_context, consolidate_context
  • WIP consolidation is available only through the admin maintenance surface

For the first production release, if memory_vectors is empty, no embedding migration is required.

Before you start

Install and verify:

node -v
npm -v
firebase --version
java -version

You need:

  • Firebase CLI authenticated with firebase login
  • a Firebase project with Blaze enabled
  • Firestore created in Native mode
  • access to the correct Firebase project alias
  • a valid GEMINI_API_KEY
  • a production MCP_ADMIN_TOKEN

If the repo is not bound to the right Firebase project yet:

cd /Users/nick/git/metacortex
firebase use --add

Runtime config

Firebase Functions loads dotenv files from functions/.

Recommended layout:

  • functions/.env: local development values
  • functions/.env.dev: development project values
  • functions/.env.prod: production project values

functions/.env.prod is local deployment config and should stay out of Git.

Start from the template:

cp /Users/nick/git/metacortex/functions/.env.example /Users/nick/git/metacortex/functions/.env

Minimum required production values:

GEMINI_API_KEY=...
MCP_ADMIN_TOKEN=...
GEMINI_EMBEDDING_MODEL=...
GEMINI_MULTIMODAL_MODEL=...
GEMINI_MERGE_MODEL=...
GEMINI_GENERATION_VERTEX_LOCATION=...
GEMINI_EMBEDDING_DIMENSIONS=...
MEMORY_COLLECTION=...

Use functions/.env.example for the current tracked defaults. Keep exact model ids there and in the deployed dotenv file, not duplicated in this playbook.

Recommended admin endpoint defaults for the first release:

MCP_ALLOWED_TOOLS=remember_context,search_context,fetch_context,deprecate_context
MCP_ALLOWED_ORIGINS=
MCP_ALLOWED_FILTER_STATES=active,merged,deprecated,wip
SEARCH_RESULT_LIMIT=5
DEFAULT_FILTER_STATE=active

Origin config split:

  • MCP_ALLOWED_ORIGINS applies only to the default admin /mcp endpoint
  • browser-hosted clients should use MCP_CLIENT_PROFILES_JSON[].allowedOrigins
  • leave MCP_ALLOWED_ORIGINS empty unless you intentionally want browser access to the admin endpoint

Important constraints:

  • GEMINI_EMBEDDING_DIMENSIONS must match the vector index dimension in firestore.indexes.json
  • if you change embedding models or dimensions after seeding data, do not mix vector spaces in the same collection
  • this codebase embeds text; image-backed memories are normalized into text before embedding

Use the default /mcp endpoint as the admin surface only. For ChatGPT web and Claude web, deploy separate scoped client profiles from day one:

  • admin: <FUNCTION_BASE_URL>/mcp
  • ChatGPT web: <FUNCTION_BASE_URL>/clients/chatgpt-web/mcp
  • Claude web: <FUNCTION_BASE_URL>/clients/claude-web/mcp

Recommended browser read/write toolset:

  • remember_context
  • search_context
  • fetch_context

Do not add deprecate_context to browser-hosted client profiles. Keep it on the admin endpoint only.

Recommended web client profile shape:

MCP_CLIENT_PROFILES_JSON=[{"id":"chatgpt-web","token":"replace-chatgpt-token","allowedTools":["remember_context","search_context","fetch_context"],"allowedFilterStates":["active"],"allowedOrigins":["https://chatgpt.com"]},{"id":"claude-web","token":"replace-claude-token","allowedTools":["remember_context","search_context","fetch_context"],"allowedFilterStates":["active"],"allowedOrigins":["https://claude.ai"]}]

For non-browser agent clients such as OpenClaw, add a separate scoped profile instead of reusing the admin token. Recommended shape:

MCP_CLIENT_PROFILES_JSON=[{"id":"chatgpt-web","token":"replace-chatgpt-token","allowedTools":["remember_context","search_context","fetch_context"],"allowedFilterStates":["active"],"allowedOrigins":["https://chatgpt.com"]},{"id":"claude-web","token":"replace-claude-token","allowedTools":["remember_context","search_context","fetch_context"],"allowedFilterStates":["active"],"allowedOrigins":["https://claude.ai"]},{"id":"openclaw","token":"replace-openclaw-token","allowedTools":["remember_context","search_context","fetch_context"],"allowedFilterStates":["active"],"allowedOrigins":[]}]

Use allowedOrigins: [] only when the OpenClaw runtime is a headless or non-browser client that does not send an Origin header. If the runtime sends Origin because it runs inside Electron, a WebView, or another browser-like environment, replace the empty list with the exact origin value or values emitted by that client.

Keep each scoped-client token distinct from MCP_ADMIN_TOKEN. The admin token should stay reserved for maintenance and ops-only clients.

Local verification

Run the preflight first:

cd /Users/nick/git/metacortex
./scripts/deploy-session-preflight.sh

That script checks:

  • git status
  • expected env file presence
  • effective production embedding config versus Firestore index dimensions
  • current Firebase project selection
  • full test suite
  • TypeScript build

Validate the live Gemini model configuration before production deploy:

cd /Users/nick/git/metacortex
npm --prefix functions run validate:models

If you want a manual local round-trip before production:

cd /Users/nick/git/metacortex
npm --prefix functions run serve

Then in another shell:

curl -i "http://127.0.0.1:5001/demo-open-brain/us-central1/metaCortexMcp/healthz"
cd /Users/nick/git/metacortex/functions
MCP_BASE_URL="http://127.0.0.1:5001/demo-open-brain/us-central1/metaCortexMcp/mcp" \
MCP_ADMIN_TOKEN="replace-me" \
npm run smoke -- --mode admin-read-write

Browser-client flow:

cd /Users/nick/git/metacortex/functions
MCP_BASE_URL="http://127.0.0.1:5001/demo-open-brain/us-central1/metaCortexMcp/clients/chatgpt-web/mcp" \
MCP_ADMIN_TOKEN="replace-chatgpt-token" \
MCP_SMOKE_MODE="browser-read-write" \
npm run smoke

Repeat with /clients/claude-web/mcp and the Claude token to validate Claude separately.

The automated tests and build can also be run directly:

cd /Users/nick/git/metacortex
npm --prefix functions test
npm --prefix functions run build
npm --prefix functions run validate:models

Deploy

1. Confirm the target project

cd /Users/nick/git/metacortex
firebase use
firebase projects:list

Do not deploy while unsure which alias is active.

2. Confirm production env values

Verify that functions/.env.prod or the dotenv file you plan to deploy with includes the intended values, especially:

  • GEMINI_API_KEY
  • MCP_ADMIN_TOKEN
  • MCP_ALLOWED_ORIGINS only if you intentionally want browser access to the admin endpoint
  • MCP_CLIENT_PROFILES_JSON with both chatgpt-web and claude-web profiles
  • model and generation settings copied from the current functions/.env.example: GEMINI_EMBEDDING_MODEL, GEMINI_MULTIMODAL_MODEL, GEMINI_MERGE_MODEL, and GEMINI_GENERATION_VERTEX_LOCATION
  • GEMINI_EMBEDDING_DIMENSIONS matching firestore.indexes.json
  • MEMORY_COLLECTION

For the first release, an empty production collection means there is no migration work to do.

If you later switch embedding models or dimensions and want to keep old memories, re-embed them or start with a fresh collection.

Also confirm the actual web-client registration values you will use:

  • ChatGPT URL: <FUNCTION_BASE_URL>/clients/chatgpt-web/mcp
  • Claude URL: <FUNCTION_BASE_URL>/clients/claude-web/mcp
  • each bearer token comes from the matching client profile, not MCP_ADMIN_TOKEN
  • each web origin must match the profile's allowedOrigins

3. Backfill TTL fields

The hardening release uses Firestore TTL policies for unbounded operational collections:

  • memory_vectors_write_fingerprints.expires_at: 30-day retention
  • memory_events.expires_at: 90-day retention

Run a dry run first:

cd /Users/nick/git/metacortex
npm --prefix functions run backfill:ttl

If the counts look correct, apply the backfill:

cd /Users/nick/git/metacortex
npm --prefix functions run backfill:ttl -- --write --project my-brain-88870

The backfill preserves numeric event timestamp, copies legacy numeric fingerprint expires_at into dedupe_expires_at when needed, and writes Date-valued expires_at fields for Firestore TTL.

4. Deploy Firestore indexes

cd /Users/nick/git/metacortex
firebase deploy --only firestore:indexes

Required vector indexes:

  • metadata.module_name ASC + embedding VECTOR
  • metadata.branch_state ASC + embedding VECTOR
  • metadata.branch_state ASC + metadata.module_name ASC + embedding VECTOR

Wait until those indexes are fully built before trusting search results.

5. Deploy the function

cd /Users/nick/git/metacortex
firebase deploy --only functions

Or deploy both together:

cd /Users/nick/git/metacortex
firebase deploy --only firestore:indexes,functions

Important: always use firebase deploy with the prod alias active (confirmed in step 1), not firebase deploy --project my-brain-88870. Firebase loads functions/.env.prod based on the alias name, not the project ID. Passing the raw project ID silently skips functions/.env.prod and omits MCP_CLIENT_PROFILES_JSON from the deployment, causing all client endpoints to 404. Use firebase deploy --project prod if you need to pass the project explicitly.

Capture the deployed base URL for metaCortexMcp.

The useful production routes are:

  • <FUNCTION_BASE_URL>/healthz
  • <FUNCTION_BASE_URL>/mcp
  • <FUNCTION_BASE_URL>/clients/<CLIENT_ID>/mcp

6. Enable Firestore TTL policies

Enable TTL policies after the expires_at fields exist:

cd /Users/nick/git/metacortex
./scripts/deploy-firestore-ttl.sh --project my-brain-88870

Verify the policies:

gcloud firestore fields ttls list --project=my-brain-88870

Post-deploy verification

1. Health check

curl -i "<FUNCTION_BASE_URL>/healthz"

Expected:

  • HTTP 200
  • response includes ok: true

2. Unauthorized request check

curl -i \
  -X POST "<FUNCTION_BASE_URL>/mcp" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"ping"}'

Expected:

  • HTTP 401

3. Client profile deployment check

Verify that MCP_CLIENT_PROFILES_JSON was bundled into the deployed function by checking that client endpoints resolve (not 404). A 404 here means the env file was not picked up — the most common cause is deploying with the raw project ID instead of the prod alias.

curl -s -o /dev/null -w "%{http_code}\n" \
  -X OPTIONS "<FUNCTION_BASE_URL>/clients/chatgpt-web/mcp" \
  -H "Origin: https://chatgpt.com"
# Expected: 204
curl -s -o /dev/null -w "%{http_code}\n" \
  -X OPTIONS "<FUNCTION_BASE_URL>/clients/claude-web/mcp" \
  -H "Origin: https://claude.ai"
# Expected: 204

If either returns 404, the client profiles were not deployed. Fix:

firebase use prod
firebase deploy --only functions

4. Browser CORS preflight details

Confirm the correct origin headers are returned:

curl -i \
  -X OPTIONS "<FUNCTION_BASE_URL>/clients/chatgpt-web/mcp" \
  -H "Origin: https://chatgpt.com"

Expected:

  • HTTP 204
  • Access-Control-Allow-Origin: https://chatgpt.com

Repeat with:

curl -i \
  -X OPTIONS "<FUNCTION_BASE_URL>/clients/claude-web/mcp" \
  -H "Origin: https://claude.ai"

5. Authenticated admin MCP smoke test

cd /Users/nick/git/metacortex/functions
MCP_BASE_URL="<FUNCTION_BASE_URL>/mcp" \
MCP_ADMIN_TOKEN="<ADMIN_MCP_TOKEN>" \
MCP_SMOKE_MODE="admin-read-write" \
npm run smoke

Expected:

  • tool listing succeeds
  • remember_context succeeds
  • search_context returns the stored sample

This is the first proof that:

  • auth works
  • the live Gemini call works
  • Firestore writes work
  • Firestore vector search works

6. Authenticated browser MCP smoke test

cd /Users/nick/git/metacortex/functions
MCP_BASE_URL="<FUNCTION_BASE_URL>/clients/chatgpt-web/mcp" \
MCP_ADMIN_TOKEN="<CHATGPT_WEB_TOKEN>" \
MCP_SMOKE_MODE="browser-read-write" \
npm run smoke -- --content "Remember that we use Ktor for shared Android and iOS networking." --query "shared networking for android and ios"

Expected:

  • remember_context succeeds
  • search_context returns a result with id=...
  • fetch_context accepts that same id and returns the full stored record content and metadata

Repeat the same smoke test against /clients/claude-web/mcp with <CLAUDE_WEB_TOKEN>.

This is the first proof that the 3-tool client-facing browser contract is usable end to end.

7. Verify observability events

Open Firestore and inspect memory_events.

Confirm:

  • at least one event exists for each successful smoke-test tool call
  • admin calls are recorded with client_id=default
  • ChatGPT web calls are recorded with client_id=chatgpt-web
  • Claude web calls are recorded with client_id=claude-web
  • tool events include tool_name, status, timestamp, latency_ms, and a compact request / response or error
  • request rejections use event_type=request with a reason such as unauthorized or origin_not_allowed

Cloud Logging should also contain structured metaCortexMcp tool event and metaCortexMcp request event entries for the same calls.

8. Verify the written document

Open Firestore and inspect memory_vectors.

Confirm:

  • one document was written
  • metadata.branch_state is active
  • the record stores both canonical content and internal retrieval_text
  • metadata.created_at and metadata.updated_at are present
  • the stored content is searchable through search_context

9. Optional multimodal browser smoke test

cd /Users/nick/git/metacortex/functions
MCP_BASE_URL="<FUNCTION_BASE_URL>/clients/chatgpt-web/mcp" \
MCP_ADMIN_TOKEN="<CHATGPT_WEB_TOKEN>" \
MCP_SMOKE_MODE="browser-read-write" \
MCP_IMAGE_BASE64="$(base64 < path/to/image.png | tr -d '\n')" \
MCP_IMAGE_MIME_TYPE="image/png" \
MCP_ARTIFACT_REF="gs://your-bucket/path/to/image.png" \
npm run smoke -- --content "Settings screen screenshot for the Compose UI" --query "compose settings screenshot"

Repeat with /clients/claude-web/mcp and <CLAUDE_WEB_TOKEN> if Claude web will ingest images.

Expected:

  • remember_context accepts the image-backed memory
  • returned JSON metadata includes modality=mixed when both text and image are present
  • search_context returns a summary-only result with the same id=...
  • fetch_context accepts that same id and returns the same artifact_refs

Token Management

Use separate tokens for separate trust boundaries:

  • MCP_ADMIN_TOKEN is the admin token for /mcp
  • MCP_CLIENT_PROFILES_JSON[].token is the scoped token for each client endpoint

Rotation and revocation rules:

  • rotate a web client token by changing that profile's token and redeploying functions
  • revoke a client by removing the profile or replacing its token and redeploying functions
  • do not reuse MCP_ADMIN_TOKEN for browser-hosted clients
  • if ChatGPT web and Claude web should be revoked independently, give them separate client profiles
  • if OpenClaw should be revoked independently from browser clients, give it its own scoped openclaw profile

Observability

After deployment, use these views together:

  • memory_vectors for the current corpus
  • memory_events for client-attributed usage and audit history
  • Cloud Logging for request failures and structured tool-event logs

memory_events is populated automatically by successful and failed tool calls plus ingress-level auth/CORS rejections. It is the easiest way to answer:

  • which client is writing memories
  • which client is searching or fetching most often
  • which memory ids are being returned or fetched repeatedly
  • how many searches return zero results
  • whether a specific client is generating repeated tool errors
  • whether a specific client is hitting repeated 401 or 403 failures

The event payload is intentionally compact. It records ids, filters, counts, states, reasons, and latency rather than duplicating full memory bodies.

First-release rollout

Do not bulk-seed the corpus before launch.

For the first release:

  • deploy the hosted MCP server
  • prove the hosted round trip works
  • let the first memories come from real work
  • watch retrieval quality before expanding automation

Early target:

  • 5 to 20 durable memories

Good early memories:

  • stable architecture decisions
  • durable project constraints
  • reusable workflows
  • canonical requirements
  • meaningful screenshots with lasting retrieval value

Recommended rollout order:

  1. Admin endpoint reserved for maintenance and smoke tests
  2. Browser client rollout on remember_context, search_context, and fetch_context
  3. Controlled writes only for clearly durable events
  4. Search-only downstream clients such as Nanobot
  5. Later use of deprecate_context plus internal WIP curation workflows

Failure checks

If deploy succeeds but search fails:

  • confirm vector indexes finished building
  • confirm GEMINI_EMBEDDING_MODEL and GEMINI_EMBEDDING_DIMENSIONS match what you deployed
  • confirm Firestore is in Native mode
  • confirm the production collection does not mix vectors from different models or dimensions

If requests return 401:

  • verify Authorization: Bearer <TOKEN>
  • verify the token belongs to the endpoint you are calling
  • verify the deployed dotenv alias loaded the values you expect

If the function deploys but cannot store documents:

  • verify the runtime service account has Firestore access
  • verify the Firestore API is enabled in the backing Google Cloud project

If browser clients get 403 Origin not allowed:

  • verify the request is using a client-scoped endpoint
  • verify that client profile has the expected allowedOrigins
  • verify the browser token matches the scoped client endpoint
  • do not use the admin endpoint for browser-hosted clients

Debugging

Useful commands:

cd /Users/nick/git/metacortex
firebase functions:list

Use Firebase console logs or Cloud Logging for failed production requests.