👁️👂 Echo: The Ambient, Intent-First Cognitive Guardian

Echo is a stateful, low-latency multi-agent architecture engineered to support elderly longevity and cognitive safety through proactive intent-anticipation.

Instead of building another reactive chatbot that relies on complex, precisely engineered text prompts—the exact kind of digital interaction that a declining memory or motor skill cannot provide—Echo introduces an Ambient Prefrontal Cortex. It continuously processes unstructured live audio and video streams to autonomously identify hazards, validate medication schedules, and block financial scams before they escalate.

---

🚀 Why We Open-Sourced Echo

We built Echo during the SuperAI 2026 NEXT Hackathon to address the massive "prompting crisis" in elder care tech. While the hackathon has ended, our commitment to dignified longevity hasn't.

We believe the underlying framework—translating ambient multi-modal streams into discrete, proactive agentic actions—is a powerful paradigm shift. We have open-sourced the entire codebase to encourage developers, roboticists, and healthcare advocates to fork it, pillage the architecture, adapt it to new environments (like CCTVs or social robots), and help scale safety solutions for an aging global population.

---

🛠️ Core Technology Stack

• Frontend / Capture: Next.js 15 (App Router) + WebRTC getUserMedia; the capture and dashboard pages are served directly by the Lambda at the API Gateway origin (no separate host, no CORS). A browser Web Audio → 16 kHz PCM encoder (client/streaming/pcm-encoder.ts) prepares audio for streaming speech-to-text.
• Agent Compute: AWS Lambda (echo-agent) fronted by API Gateway HTTP API — the three-agent loop actually runs on AWS, not just as external API calls.
• Vision & Reasoning: Amazon Bedrock multimodal streaming (Claude Haiku 4.5 for fast per-frame assessment), invoked through the Vercel AI SDK (@ai-sdk/amazon-bedrock).
• Executive Classification: Vercel AI SDK generateObject, routed through Vercel AI Gateway when AI_GATEWAY_API_KEY is set (verifiable in the Vercel dashboard), with a Bedrock-direct fallback.
• Speech-to-Text (opt-in): Amazon Transcribe (streaming) folds per-chunk transcripts into the prompt so the model reasons over what is heard as well as seen. Off by default for latency (ECHO_TRANSCRIBE=1).
• Durable State: Amazon DynamoDB — EchoAccounts (subscription gating), EchoAlerts (delivered feed) and EchoUndeliveredAlerts (TTL ≥ 30 days).
• External Neural Memory: Exa for fast, real-time, non-blocking neural/semantic web search (scam verification, local context).
• Agentic Commerce & Gatekeeping: Stripe (test mode) for the "Peace of Mind" subscription tier — Checkout sessions + signed webhook reduction that flips the alert-network gate.
• Language & Quality: strict TypeScript end-to-end, modular pure-core/IO-adapter split, and fast-check property tests (≥100 cases each) over all safety-critical logic.

---

🤖 The Multi-Agent Orchestration Architecture

Echo moves past standard single-prompt LLM wrappers by implementing a highly decoupled, stateful, three-agent loop:

1. The Executive Agent: Maintains a continuous, stateful connection to the user's live audio/video feed. It tracks a sliding "Daily History Log" to contextualize fragmented thoughts and autonomously spot environmental changes.
2. The Research Agent: Fired off asynchronously by the Executive Agent when real-world validation is needed. It queries Exa out-of-band to pull real-time scam patterns, local updates, or medical documentation without lagging the main stream.
3. The Action & Safety Agent: Executes physical or digital overrides. It handles smart-home triggers (e.g., stove shutoffs), processes autonomous care delivery orders via Stripe, and trips the Human-in-the-Loop (HITL) telemetry dashboard to alert family members during severe anomalies.

---

🤝 How You Can Contribute (Forks & PRs Welcome!)

We designed Echo to be modular so that you can easily rip out parts of the stack or build heavily on top of it. We are actively seeking contributors to help expand the project in these areas:

• 🔌 Hardware Connectors: Porting the stream client from web browsers to dedicated IoT cameras, smart glasses, or Raspberry Pi/CCTV feeds.
• 🧠 Reasoning Models: Testing and adding native support for other high-speed streaming audio/video LLMs.
• 🔒 Local-First & Privacy: Implementing fully local execution layers (e.g., using local vector DBs or on-prem models) to ensure 100% data privacy within the home.
• 🚨 Dashboard Expansions: Building out the Human-in-the-Loop Twilio/SMS alert pipeline for caregivers.

Getting Started

1. Fork the repository.
2. Review the /docs or the setup variables inside .env.example.
3. Submit a Pull Request with your feature or optimization—let's build a safer, more common-sense world for our seniors together.

---

Explore our original submission on DoraHacks BUIDL #44529.

Technology Breakdown

Autonomous, ambient, multi-agent real-time video/audio monitoring for elderly longevity and cognitive safety.

Echo captures live webcam + microphone media in a Next.js 15 client, streams it over a stateful WebSocket to an edge media pipeline, forwards multimodal chunks to Amazon Bedrock streaming reasoning models, and runs a three-agent loop (Executive, Research, Action/Safety) over the output. Severe safety exceptions trigger a Human-in-the-Loop circuit that pushes telemetry alerts to a family dashboard. Alert access is gated by a Stripe "Peace of Mind" subscription.

Architecture at a glance

Capture (WebRTC) → WebSocket → Chunk Assembler → Bounded Buffer (max 30)
   → Bedrock Forwarder → Executive Agent
        ├─ Research Agent → Exa (async, non-blocking)
        └─ Action/Safety Agent → Webhook → Family Dashboard  (gated by Stripe)

The codebase is deliberately split into two layers:

• Pure cores (*.ts logic with no I/O): buffering, chunk ordering, connection state machine, agent routing, research eligibility/capping, alert formatting, gating, degradation classification, audio fallback, subscription reduction. These are deterministic and property-tested with fast-check — no external services required.
• I/O adapters (thin edges): WebRTC/WebSocket, Bedrock, Exa, Stripe, webhooks. These need accounts/keys and are exercised by integration tests.

Prerequisites

• Node.js 20+ and npm (install from https://nodejs.org or via nvm/winget install OpenJS.NodeJS.LTS).
• A modern browser with camera/microphone access for the capture client.

Setup

npm install

Copy the environment template and fill in values (only needed for the live system, not for unit/property tests):

cp .env.example .env.local

Testing

Echo uses a dual testing strategy.

Property-based + unit tests — NO API keys required

These cover all the safety-critical pure logic and run entirely offline.

npm test          # run all tests once (vitest run)
npm run test:watch # watch mode
npm run typecheck # strict TypeScript, no emit

Property tests use fast-check at a minimum of 100 generated cases each, tagged with // Feature: echo, Property {N}: {property_text}. The shared iteration config lives in shared/testing/fast-check-config.ts.

What's covered with zero credentials:

• Connection state machine closure + bounded retries
• Chunk assembly duration, buffer ordering + overflow
• Degradation classification + audio fallback timeline
• Agent routing + delegation retry bounds
• Research eligibility threshold + result capping
• Telemetry alert formatting, dispatch gating + retry bounds
• Dashboard ordering, status projection, acknowledgment
• Subscription status reducer

Run npm install && npm test for a green run (266 passing, 2 live-gated probes skipped by default).

Integration tests — require credentials

Integration tests (timing budgets, transport, external services) are wired in tasks 12–18 and need the accounts below. They validate latency budgets, webhook delivery, Stripe sessions, and persistence with 1–3 representative cases rather than property iteration.

Accounts & API keys

You only need these to run the live end-to-end system and the integration tests. The table maps each external dependency to where it's used.

Service	Used by	What you need	Where to get it
AWS (Amazon Bedrock)	Media pipeline → reasoning	AWS account with Bedrock model access enabled (e.g. a multimodal/streaming model), IAM credentials	https://console.aws.amazon.com/bedrock — request model access, then create an IAM user/role with bedrock:InvokeModelWithResponseStream
Exa	Research Agent (neural search)	Exa API key	https://exa.ai → dashboard → API keys
Stripe	Subscription Service + webhooks	Secret key, publishable key, webhook signing secret, a Price ID for the "Peace of Mind" tier	https://dashboard.stripe.com → Developers → API keys; create a Product/Price; set up a webhook endpoint
Vercel	Executive Agent classification via Vercel AI SDK + AI Gateway	Vercel account + AI Gateway API key (optional; falls back to Bedrock-direct)	https://vercel.com → AI Gateway → API Keys

Environment variables

See .env.example. Summary:

# AWS Bedrock
AWS_REGION=us-east-1
AWS_ACCESS_KEY_ID=...
AWS_SECRET_ACCESS_KEY=...
BEDROCK_MODEL_ID=...

# Exa
EXA_API_KEY=...

# Stripe
STRIPE_SECRET_KEY=sk_test_...
NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY=pk_test_...
STRIPE_WEBHOOK_SECRET=whsec_...
STRIPE_PEACE_OF_MIND_PRICE_ID=price_...

# App
ALERT_WEBHOOK_URL=https://your-dashboard/webhooks/alerts

Keep real secrets in .env.local (gitignored). Never commit keys. Use Stripe test mode keys (sk_test_/pk_test_) for development.

Local Stripe webhooks

For local webhook delivery, use the Stripe CLI:

stripe login
stripe listen --forward-to localhost:3000/api/webhooks/stripe

The CLI prints a whsec_... signing secret to put in STRIPE_WEBHOOK_SECRET.

Running the app

npm run dev   # Next.js dev server on http://localhost:3000

The capture client requires camera/microphone permission and a secure context (localhost or HTTPS). Browsers block getUserMedia on insecure origins.

Project layout

shared/              Shared types + fast-check test config (no I/O)
client/streaming     Connection state machine, capture controller, PCM encoder (WebRTC/WS)
edge/pipeline        Chunk assembler, bounded buffer, degradation/audio cores, transcription, Bedrock forwarder
agents/orchestrator  Executive Agent: task routing + delegation
agents/research      Research Agent: eligibility, result capping, Exa adapter
agents/safety        Action/Safety Agent: alert formatting, gating, dispatch + webhook adapter
dashboard            Alert ordering, status projection, acknowledgment, store
subscription         Subscription reducer + Stripe checkout/webhook handler
app/                 Composed pipeline, AI evaluator, AWS runtime + DynamoDB stores
app/aws              DynamoDB stores, Bedrock/Transcribe runtime adapters
lambda/handler.ts    AWS Lambda entrypoint (serves UI + agent loop over HTTP)
scripts/             AWS deploy + probe scripts (Bedrock, DynamoDB, API Gateway, Transcribe)

Implementation status

Echo is fully wired and deployed on AWS. The pure cores and their fast-check property tests, the I/O adapters (Bedrock, Exa, Stripe, DynamoDB, Transcribe, WebRTC/WebSocket), and the end-to-end composition (app/aws/runtime.ts → lambda/handler.ts) are all in place and verified live (npm test → 266 passing, 2 live-gated probes skipped by default). The spec history lives in .kiro/specs/echo/tasks.md; judge-facing verification notes are in JUDGING.md.

Manual end-to-end testing (local harness)

A standalone Node harness (server.mts) lets you exercise the real composed pipeline from a browser webcam. It is a local test harness, not the production Vercel/Next deployment.

npm run harness

Cloud mode (browser → deployed AWS agent)

Set ECHO_CLOUD_API to the deployed API Gateway URL (see .env.example) and the harness proxies the browser demo to the AWS agent instead of running the pipeline in-process:

• capture frames → POST {cloud}/process (Lambda → Bedrock vision)
• Force alert / scam research / Subscribe / Cancel buttons → the cloud Lambda (DynamoDB gating, Vercel AI Gateway classification, Exa research)
• the dashboard shows cloud-produced alerts and a mode: cloud badge

The startup log prints the active mode. Leave ECHO_CLOUD_API empty for the in-process local pipeline.

Then open:

• http://localhost:3000/ — capture page: grant camera/mic, click Start. A frame is sent every 2s over WebSocket; each triggers one live Bedrock multimodal call and the model's assessment is logged.
• http://localhost:3000/dashboard — family dashboard: live stream status + alerts via SSE, plus Subscribe / Cancel buttons to demonstrate subscription gating (Req 8.4 — alerts are withheld while the account is inactive).
• http://localhost:3000/selftest — injects a synthetic frame through the full pipeline and returns the reasoning JSON (one live Bedrock call; handy for a no-camera check).

How it flows: capture → WebSocket → EchoPipeline.forwarder.forward (Anthropic body → live Bedrock) → processReasoning (executive → orchestrator → research/safety agents) → ActionSafetyAgent (subscription-gated) → DashboardStore → SSE to the dashboard.

Dashboard test controls (no webcam / no Bedrock cost)

The dashboard page has buttons to exercise the alert + degradation paths without staging a real hazard:

• Force HIGH / MEDIUM alert — injects a synthetic reasoning output through the real evaluator → orchestrator → safety agent → gating → dashboard (no Bedrock call). With the account active an alert appears; after Cancel it is withheld (demonstrates Req 8.4 gating).
• Trigger scam research (Exa) — injects a scam-pattern classification (confidence ≥ 0.70) so the Executive Agent emits a research task and the Research Agent fires a live Exa neural search; the ranked results (capped at 10) render in the dashboard's Research panel.
• Simulate degradation — drives the real classifyDegradation and evaluateAudioFallback cores with a low-SNR / high-frame-loss window and shows the dashboard's degraded indicator; Recover clears it.

Equivalent HTTP endpoints (handy for scripted checks):

curl -X POST "http://localhost:3000/force-alert?severity=high"
curl -X POST "http://localhost:3000/force-alert?severity=medium"
curl -X POST http://localhost:3000/trigger-research
curl -X POST http://localhost:3000/degrade
curl -X POST http://localhost:3000/recover
curl -X POST http://localhost:3000/subscribe
curl -X POST http://localhost:3000/unsubscribe

Note: Trigger scam research makes a live (billable/quota'd) Exa API call; the alert and degradation buttons do not call any external service.

Notes:

• Each captured frame is a billable Bedrock call. The default cadence is one frame per 2s; stop the stream when done.

Real Stripe Checkout (test mode)

The dashboard Subscribe (Stripe Checkout) button redirects to a live Stripe test-mode Checkout session created by the real SubscriptionService.startCheckout (GET /checkout → 302 → checkout.stripe.com). Use a Stripe test card (e.g. 4242 4242 4242 4242, any future expiry/CVC).

To complete the loop so the account flips to active after payment, forward Stripe webhooks to the harness with the Stripe CLI:

stripe listen --forward-to localhost:3000/api/stripe/webhook

The harness verifies the webhook signature with STRIPE_WEBHOOK_SECRET, maps the event, and reduces it through reduceAccountStatus (e.g. checkout.session.completed → active). Without the CLI running, use Simulate active to toggle the account without Stripe.

• Simulate active (POST /subscribe) and Cancel (POST /unsubscribe) toggle the account status without Stripe, for quick gating tests.
• The harness uses a risk-parsing evaluator: the model's reasoning text is scanned for risk keywords/levels, and a medium/high risk produces a Telemetry_Alert. A normal scene produces no alert.
• The demo account starts active; use Cancel on the dashboard to see alerts gated off, Subscribe to re-activate.
• This harness sends one image per frame straight to the forwarder for responsiveness, bypassing the chunk assembler/buffer (which are covered by the automated tests).
• Requires .env with valid AWS credentials and an invocable BEDROCK_MODEL_ID (cross-region inference profile; defaults to the fast vision model us.anthropic.claude-haiku-4-5-20251001-v1:0 — swap to an Opus profile for deeper assessments).
• A demo clip in public/ (e.g. public/demo.mp4) enables a one-click Play sample video button on the capture page — drive the whole demo with no webcam. Audio speech-to-text (Amazon Transcribe) is opt-in via ECHO_TRANSCRIBE=1 (off by default for latency).

AWS deployment (hackathon: agents running on AWS)

Echo's agents run on AWS compute against AWS data + AI/ML services — not just external API calls:

Concern	AWS service	Where
Agent compute	AWS Lambda (echo-agent) fronted by API Gateway HTTP API	lambda/handler.ts
Reasoning model	Amazon Bedrock (Claude Haiku 4.5 vision profile; swap BEDROCK_MODEL_ID to Opus for deeper assessments)	invoked via the Vercel AI SDK (@ai-sdk/amazon-bedrock)
Agent intelligence	Vercel AI SDK generateObject (Executive Agent classification)	app/ai-evaluator.ts
Executive classifier routing	Vercel AI Gateway (when AI_GATEWAY_API_KEY set)	app/ai-evaluator.ts (resolveExecutiveModel)
Database	Amazon DynamoDB — EchoAccounts (gating) + EchoUndeliveredAlerts (TTL ≥30d)	app/aws/dynamo-stores.ts
Real-time research	Exa neural search (Research Agent)	agents/research

Live composition: app/aws/runtime.ts (initializeAwsRuntime) wires DynamoDB stores + the Bedrock-backed AI evaluator + the rest of the pipeline; the Lambda handler exposes it over HTTP.

Deploy (from .env credentials, region us-east-1)

npm run aws:check           # verify credentials + identity
npm run aws:deploy:dynamo   # create DynamoDB tables (+ TTL)
npm run aws:deploy:lambda   # esbuild-bundle + zip + create/update the Lambda + role
npm run aws:deploy:apigw    # public API Gateway HTTP API in front of the Lambda
npm run aws:data            # scan the DynamoDB tables (proof of writes)

Note: public Lambda Function URLs are blocked by the sandbox org SCP, so the agent is exposed via API Gateway HTTP API instead.

Live endpoints (verified)

• GET / → service/wiring banner (compute=aws-lambda, db=dynamodb, model=bedrock, ai_sdk=Vercel AI SDK, research=exa)
• GET /ui → capture page (webcam → POST /process every 2s), served by the Lambda
• GET /dashboard → family dashboard (polls /state, with Subscribe/Cancel/Force-alert/Scam-research buttons)
• GET /state → JSON { account, alerts[] } read live from DynamoDB (EchoAccounts + EchoAlerts)
• POST /process { "raw": "<scene text>" } or { "frameBase64": "<png>" } → runs the agent loop on Lambda
• POST /subscribe / POST /unsubscribe → flip the account's alert-network gating in DynamoDB

Example:

curl https://<api-id>.execute-api.us-east-1.amazonaws.com/
curl -X POST https://<api-id>.execute-api.us-east-1.amazonaws.com/subscribe
curl -X POST https://<api-id>.execute-api.us-east-1.amazonaws.com/process \
  -H "content-type: application/json" \
  -d '{"raw":"stove on fire, smoke filling the kitchen. Risk Level: HIGH."}'

Verified end-to-end on AWS: a scam scenario triggers a live Exa search (10 ranked results); a fire scenario produces a Bedrock-classified stove_fire_hazard alert that is gated by DynamoDB account status and delivered. npm run aws:data shows the persisted EchoAccounts record.

Test the deployed version (no install, just a browser)

The whole demo is served from the deployed AWS agent — the frontend UI is served by the Lambda itself at the API Gateway origin (no separate hosting, no CORS, no S3). Replace <API> with the deployed base URL printed by npm run aws:deploy:apigw (currently https://mwjz9qme01.execute-api.us-east-1.amazonaws.com).

1. Open the dashboard: <API>/dashboard
   • Click Subscribe (activate alerts) — flips the EchoAccounts record to active in DynamoDB.
   • Click Force HIGH alert — sends a scene to POST /process; the Vercel AI Gateway → Amazon Bedrock Executive Agent classifies it, the Action/Safety agent gates on the DynamoDB subscription, and the delivered alert is persisted to the EchoAlerts table and shows up in the feed (the page polls /state every 3s).
   • Click Cancel (gate alerts) then Force HIGH alert again — the alert is now withheld (subscription gating, Req 8.4).
   • Click Trigger scam research (Exa) — fires a live Exa neural search; the ranked results render in the Research panel.
2. Open the capture page: <API>/ui
   • Easiest: click ▶ Play sample video — frames are sampled every 2s from a bundled demo clip and assessed live, no webcam needed.
   • Or click ▶ Start camera to use your webcam. Every 2s a JPEG frame is posted to POST /process → Amazon Bedrock (Claude Haiku 4.5) multimodal reasoning, and the model's assessment is logged. (Each frame is a billable Bedrock call; click ■ Stop when done.)

Scripted checks (curl)

API=https://mwjz9qme01.execute-api.us-east-1.amazonaws.com

curl $API/                       # wiring banner
curl $API/state                  # account + persisted alerts (from DynamoDB)
curl -X POST $API/subscribe      # activate alert network
curl -X POST $API/process \
  -H "content-type: application/json" \
  -d '{"raw":"Risk Level: HIGH - a stove fire hazard is observed in the kitchen."}'
curl $API/state                  # the delivered alert is now persisted + listed
curl -X POST $API/process \
  -H "content-type: application/json" \
  -d '{"raw":"caller claims to be from the IRS demanding gift card payment from grandma"}'
                                 # → live Exa research in the response "research" field

The local harness can also drive this same cloud agent from a browser webcam — set ECHO_CLOUD_API=<API> and run npm run harness (see Cloud mode above).

Vercel AI Gateway

The Executive Agent's classification step routes through Vercel AI Gateway when AI_GATEWAY_API_KEY is set (see .env.example), giving a verifiable Vercel-side integration with request logs in the Vercel dashboard. When the key is absent it falls back to calling Bedrock directly. The main vision/reasoning forwarder always uses Amazon Bedrock on AWS, so:

• main multimodal reasoning → Amazon Bedrock (AWS), and
• Executive Agent classification → Vercel AI Gateway (Vercel product, with API key).

Setup: create a key at Vercel → AI Gateway → API Keys, set AI_GATEWAY_API_KEY (and optionally AI_GATEWAY_MODEL, default anthropic/claude-3.5-haiku) in .env, then:

npm test -- app/gateway-probe.test.ts   # gated live probe (runs only when the key is set)
npm run aws:deploy:lambda               # redeploy so the Lambda picks up the key

The deployed GET / banner reports executive_classifier: "vercel-ai-gateway" once the key is present.

---

🙏 Thank You — SuperAI 2026 NEXT Hackathon Sponsors

Echo was built at the SuperAI 2026 NEXT Hackathon, and it simply would not exist without the platforms our sponsors put in our hands. Echo's whole thesis — turning an ambient multimodal stream into proactive, agentic safety actions — leans directly on their tooling. A heartfelt thank you to:

• Amazon Web Services (AWS) — the backbone of the deployed system. Bedrock powers the multimodal vision/reasoning on every frame, the three-agent loop runs on Lambda behind API Gateway, durable state lives in DynamoDB, and Transcribe provides opt-in streaming speech-to-text. Echo's agents genuinely run on AWS, not just call out to it.
• Vercel — the Vercel AI SDK (generateObject) drives the Executive Agent's structured classification, and Vercel AI Gateway gives us verifiable, observable model routing with first-class logs.
• Anthropic — Claude (Haiku 4.5 via Bedrock) is the perception and reasoning engine that lets Echo tell a normal scene from a genuine emergency.
• Exa — the Exa neural/semantic search API is the Research Agent's real-time knowledge tool for verifying scams and surfacing current safety guidance, fast and non-blocking.
• Stripe — Stripe Checkout + webhooks power the "Peace of Mind" subscription tier and the human-in-the-loop gate that decides when Echo is allowed to contact a family.

And thank you to the SuperAI 2026 NEXT Hackathon organizers, mentors, and judges for the venue, the prize tracks, and the push to build something that matters for elderly longevity and cognitive safety. 🧡