Daydream Cloud: Remote GPU Processing in Scope by emranemran · Pull Request #397 · daydreamlive/scope

emranemran · 2026-02-04T06:40:20Z

Daydream Cloud: Remote GPU Processing in Scope

Summary

This PR adds a Cloud Mode toggle to Scope, enabling users to run video generation pipelines on remote cloud GPUs (H100s for now) instead of requiring a local GPU. Aside from Plugins and LoRAs, Local and Cloud mode are at feature parity.

Architecture Overview

Local Mode (Current)

┌─────────────────────────────────────────────────────────────┐
│                      User's Machine                          │
│  ┌─────────────┐     WebRTC      ┌────────────────────────┐ │
│  │   Browser   │ ◄─────────────► │    Scope Backend       │ │
│  │  (Frontend) │                 │  (Local GPU - RTX/etc) │ │
│  └─────────────┘                 └────────────────────────┘ │
│                                            │                 │
│                                     ┌──────┴──────┐          │
│                                     │    Spout    │          │
│                                     │  (Windows)  │          │
│                                     └─────────────┘          │
└─────────────────────────────────────────────────────────────┘

Cloud Mode

When to use: Desktop app where Spout I/O or local frame processing is needed.

How to enable: Start Scope, sign in, click "Connect to Cloud" in UI.

┌───────────────────────────────────────────────────────────────────────┐
│                           User's Machine                               │
│                                                                        │
│  ┌───────────┐        ┌─────────────────────────┐        ┌─────────┐  │
│  │  Browser  │ WebRTC │    Scope Backend        │ Spout  │ TouchD  │  │
│  │ (Frontend)│◄──────►│    (Client Mode)        │◄──────►│ Resolume│  │
│  └───────────┘        │                         │        │ etc.    │  │
│                       │  ┌───────────────────┐  │        └─────────┘  │
│                       │  │ CloudConnection   │  │                     │
│                       │  │ Manager           │  │                     │
│                       │  └────────┬──────────┘  │                     │
│                       └───────────┼─────────────┘                     │
└───────────────────────────────────┼───────────────────────────────────┘
                                    │
                          WebSocket │ + WebRTC
                          (signaling│ + video)
                                    │
                                    ▼
                       ┌─────────────────────────────────┐
                       │         Daydream Cloud            │
                       │  ┌───────────────────────────┐  │
                       │  │       fal_app.py          │  │
                       │  │  ┌─────────────────────┐  │  │
                       │  │  │   Scope Backend     │  │  │
                       │  │  │   (H100 GPU)        │  │  │
                       │  │  └─────────────────────┘  │  │
                       │  └───────────────────────────┘  │
                       └─────────────────────────────────┘

Flow:
1. User clicks "Connect to Cloud" in Settings panel
2. Local backend connects to Cloud backend via WebSocket
3. Local backend acts as WebRTC CLIENT to Cloud backend (role reversal!)
4. Frames from browser/Spout → local backend → Cloud → processed → back
5. Spout I/O works transparently (local backend handles it)

Key Components

Backend (scope-server)

File	Purpose
`cloud_connection.py`	WebSocket connection manager with request/response correlation
`cloud_webrtc_client.py`	WebRTC peer connection (client role) to fal.ai
`cloud_track.py`	MediaStreamTrack for bidirectional video relay
`frame_processor.py`	Modified to route frames to cloud when connected
`webrtc.py`	Extended to handle cloud mode offers

Cloud (fal.ai)

File	Purpose
`fal_app.py`	Serverless wrapper running Scope on fal.ai H100
`FAL_DEPLOYMENT.md`	Deployment guide for fal.ai

Frontend

File	Purpose
`cloudAdapter.ts`	WebSocket-based API client for fal.ai
`cloudContext.tsx`	React context for cloud mode state
`useUnifiedWebRTC.ts`	Unified hook supporting both local and cloud modes
`useApi.ts`	API hook that routes to local or cloud
`CloudModeToggle.tsx`	UI component for enabling cloud mode

API Endpoints

New endpoints for cloud connection management:

POST /api/v1/cloud/connect    - Connect to fal.ai cloud
POST /api/v1/cloud/disconnect - Disconnect from cloud
GET  /api/v1/cloud/status     - Get connection status and stats

When connected to cloud, existing endpoints are automatically proxied:

POST /api/v1/pipeline/load → proxied to cloud
GET /api/v1/pipeline/status → proxied to cloud
GET /api/v1/pipelines/schemas → proxied to cloud
POST /api/v1/webrtc/offer → handled locally (relay mode)

Features

Core Functionality

Connect/disconnect to fal.ai cloud
WebRTC video relay through local backend
API call proxying via WebSocket
Spout input/output in cloud mode
Reference image upload to cloud
Recording download from cloud

Authentication

Cloud mode requires Daydream authentication:

User signs in via OAuth flow
Frontend passes user ID to backend
Backend includes user ID in cloud connection
Events are tracked with user context in Kafka (only on Cloud backend - Local backend does not track anything)

Add fal.ai serverless wrapper (fal_app.py) that runs Scope server on fal.ai GPU infrastructure (H100): - Start Scope server as subprocess on fal runner - WebSocket endpoint for WebRTC signaling and API proxying - Session data cleanup on disconnect (prevents data leakage) - Connection ID for correlating logs across client and server - Base64 encoding for binary file uploads/downloads - Assets directory configuration for fal environment Signed-off-by: emranemran <emran.mah@gmail.com> Signed-off-by: Max Holland <max@livepeer.org>

Add modules enabling local Scope server to act as WebRTC client to cloud for GPU processing: - CloudConnectionManager: WebSocket connection with request/response correlation - CloudWebRTCClient: Peer connection from local server to cloud - CloudTrack: MediaStreamTrack for bidirectional video through cloud - Request keyframe (PLI) on track received for VP8 decoder stability - Cloud-related Pydantic schemas for API requests/responses Signed-off-by: emranemran <emran.mah@gmail.com> Signed-off-by: Max Holland <max@livepeer.org>

Modify FrameProcessor to support cloud mode (cloud processing): - Route frames to cloud via WebRTC when connected - Receive processed frames from cloud callback - Spout sender/receiver work transparently in both modes - WebRTC manager handles cloud mode offers - Fix Spout integration in cloud mode Signed-off-by: emranemran <emran.mah@gmail.com> Signed-off-by: Max Holland <max@livepeer.org>

Add REST API endpoints for cloud connection management: - POST /api/v1/cloud/connect - Connect to cloud - POST /api/v1/cloud/disconnect - Disconnect from cloud - GET /api/v1/cloud/status - Get connection status - Proxy model download requests to cloud in cloud mode - Proxy hardware API calls to cloud - Reference image upload handling in cloud mode - Recording download support in cloud mode Signed-off-by: emranemran <emran.mah@gmail.com> Signed-off-by: Max Holland <max@livepeer.org>

Add UI components for managing cloud connection: - CloudModeToggle: Connection status, connect/disconnect buttons - Connection ID display with copy-to-clipboard - Disable controls during cloud connection attempt - Disable LoRA selection in cloud mode (not supported) - Show connecting/disconnecting state with spinner Signed-off-by: emranemran <emran.mah@gmail.com> Signed-off-by: Max Holland <max@livepeer.org>

Add React hooks and utilities for cloud mode WebRTC: - useUnifiedWebRTC: Unified hook supporting both local and cloud modes - useApi: API request hook with cloud mode support - usePipelines: Refresh pipeline list on cloud mode toggle - usePipeline: Remove duplicate status polling - cloudAdapter/cloudContext: Cloud SDK integration Signed-off-by: emranemran <emran.mah@gmail.com> Signed-off-by: Max Holland <max@livepeer.org>

Pipeline changes for text-to-video mode: - Skip video input for text-to-video pipelines - Input mode detection in wan2_1 pipeline Signed-off-by: emranemran <emran.mah@gmail.com> Signed-off-by: Max Holland <max@livepeer.org>

Add GitHub Actions workflow to build Docker images for feature branches. Signed-off-by: emranemran <emran.mah@gmail.com> Signed-off-by: Max Holland <max@livepeer.org>

…connections Implements a maximum connection duration limit to control cloud GPU costs. Connections are automatically closed after 60 minutes regardless of activity. Changes: - Add MAX_CONNECTION_DURATION_SECONDS (3600s) and TIMEOUT_CHECK_INTERVAL_SECONDS (60s) constants - Track connection_start_time when WebSocket connects - Add check_max_duration_exceeded() helper that sends a graceful "closing" message with reason "max_duration" before disconnect - Use asyncio.wait_for() with timeout on ws.receive_text() to enable periodic duration checks during idle periods - Also check duration after each received message to catch high-activity scenarios The client receives a JSON message {"type": "closing", "reason": "max_duration", "elapsed_seconds": ...} before disconnection, allowing graceful handling. Signed-off-by: emranemran <emran.mah@gmail.com> Signed-off-by: Max Holland <max@livepeer.org>

Integrates Daydream authentication and adds comprehensive event tracking via Kafka for both local and cloud modes. Frontend: - Add OAuth sign-in flow with Daydream authentication - Show cloud mode toggle only when authenticated - Pass user ID to backend for log correlation Backend: - Add kafka_publisher module for async Kafka event publishing - Publish stream lifecycle events (stream_started, stream_stopped, stream_error) - Track user_id through cloud connections for log correlation Cloud (fal_app.py): - Add KafkaPublisher class for WebSocket event tracking - Publish websocket_connected/disconnected events with user context - Support set_user_id message for log correlation Signed-off-by: emranemran <emran.mah@gmail.com> Signed-off-by: Max Holland <max@livepeer.org>

- Add PipelinesProvider wrapper in App.tsx - Fix ReportBugDialog prop (onClose vs onOpenChange) - Add vaeTypes to PipelineInfo type - Add refreshPipelines to PipelinesContext - Install @radix-ui/react-tabs dependency

- Add DaydreamAccountSection component with auth + cloud mode toggle - Move cloud mode from StreamPage right panel to Settings General tab - Add Switch UI component for toggle - Remove sign in/sign out buttons from Header (now in Settings) - Delete unused CloudModeToggle component

load_params in PipelineLoadRequest is already dict[str, Any], not a Pydantic model, so calling model_dump() on it fails.

The method is handle_offer_with_relay, not handle_offer_with_cloud.

Method was renamed in cloud_connection.py but frame_processor.py was still calling the old name, causing frames not to be sent in video mode.

Signed-off-by: Yondon Fu <yondon.fu@gmail.com>

yondonfu · 2026-02-06T16:17:31Z

src/scope/core/pipelines/wan2_1/blocks/setup_caches.py

            components.config.vae_spatial_downsample_factor
            * components.config.patch_embedding_spatial_downsample_factor
        )
+


Stray can be removed.

yondonfu · 2026-02-06T16:18:54Z

src/scope/server/build.py

    dist_dir = frontend_dir / "dist"
    if not dist_dir.exists():
-        print("❌ Error: Frontend build failed - dist directory not found")
+        print("Error: Frontend build failed - dist directory not found")


If these changes are env specific eg for CI they shouldn't be necessary as the emojis can be left if proper env vars are set. See how the current GH actions do this

scope/.github/workflows/test.yml

Line 43 in a1ddc9d

env:

I'll remove this. I think this was added when I tried to use WSL to compile and I forgot to remove.

yondonfu · 2026-02-06T16:20:23Z

frontend/vite.config.ts

    },
  },
  server: {
+    host: true,


Shouldn't need this as a default because can just run npm run dev -- --host if network access is needed.

yondonfu · 2026-02-06T16:23:31Z

.github/workflows/docker-build-branch.yml

This looks pretty similar to the existing Docker build action. Can that one be used with whatever modifications are needed? In practice, I think once dev on this branch is done only need pushes of images when merging to main?

yondonfu · 2026-02-06T16:25:46Z

frontend/src/components/settings/DaydreamAccountSection.tsx

+ * DaydreamAccountSection - Auth and Cloud Mode UI for Settings
+ *
+ * Displays:
+ * - Not logged in: Sign in/Sign up buttons


Clean up comment needed no sign up button

yondonfu · 2026-02-06T16:44:13Z

src/scope/server/frame_processor.py

        max_parameter_queue_size: int = 8,
        initial_parameters: dict = None,
        notification_callback: callable = None,
+        cloud_manager: "Any | None" = None,  # CloudConnectionManager for cloud mode


Consider using TYPE_CHECKING import for CloudConnectionManager as is done elsewhere. Example

scope/src/scope/core/plugins/manager.py

Line 20 in a1ddc9d

if TYPE_CHECKING:

yondonfu · 2026-02-06T16:45:07Z

src/scope/server/pipeline_processor.py

+                # Check if pipeline actually supports VACE before routing to vace_input_frames
+                from scope.core.pipelines.wan2_1.vace import VACEEnabledPipeline
+
+                pipeline_supports_vace = isinstance(self.pipeline, VACEEnabledPipeline)


Can we keep what was previously here as that avoided doing a check per chunk. This looks like a regression

yondonfu · 2026-02-06T16:45:35Z

src/scope/server/pipeline_processor.py


    def _calculate_output_fps(self):
-        """Calculate FPS from the average inter-frame delta."""
+        """Calculate FPS based on how fast frames are produced into the output queue."""


What is the reason for this change? Looks like a regression

yondonfu · 2026-02-06T16:46:01Z

src/scope/server/pipeline_processor.py

            output_dict = self.pipeline(**call_params)

            # Extract video from the returned dictionary
-            output = output_dict.get("video")


Why is this being removed this seems like a regression

yondonfu · 2026-02-06T16:47:58Z

frontend/src/lib/cloudAdapter.ts

+/**
+ * React hook for using the CloudAdapter
+ */
+export function useCloudAdapter(wsUrl: string | null, apiKey?: string) {


Please check given the existence of cloudContext whether this file can be cleaned up as there is a bunch of code here that seems like dup and not used.

- setup_caches.py: Remove stray blank line - app.py: Fix duplicate env var name in error message (SCOPE_CLOUD_APP_ID repeated twice → SCOPE_CLOUD_API_KEY) - cloud_track.py: Remove black frame fallback that caused flash at stream start; wait indefinitely for first real frame from cloud, matching local track behavior. Remove unused numpy import. - frame_processor.py: Use TYPE_CHECKING import for CloudConnectionManager instead of typing cloud_manager param as Any - build.py: Revert to main (restore emoji in print messages) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: emranemran <emran.mah@gmail.com>

Remove dead-code Kafka publish_event call and session_id/user_id params that were added for cloud mode but are unreachable (cloud mode bypasses PipelineProcessor entirely). Restores VACE routing guard, dual-output forwarding, and inter-frame delta FPS tracking. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: emranemran <emran.mah@gmail.com>

- setup_caches.py: Remove stray blank line - app.py: Fix duplicate env var name in error message (SCOPE_CLOUD_APP_ID repeated twice → SCOPE_CLOUD_API_KEY) - cloud_track.py: Remove black frame fallback that caused flash at stream start; wait indefinitely for first real frame from cloud, matching local track behavior. Remove unused numpy import. - frame_processor.py: Use TYPE_CHECKING import for CloudConnectionManager instead of typing cloud_manager param as Any - build.py: Revert to main (restore emoji in print messages) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: emranemran <emran.mah@gmail.com>

- frame_processor.py: Remove Spout pause check in _spout_receiver_loop so frames continue flowing to the pipeline while paused, preserving original local-mode behavior - pipeline_manager.py: Remove stale pipeline status cleanup in load_pipelines() to preserve original status dict behavior Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: emranemran <emran.mah@gmail.com>

emranemran and others added 16 commits February 3, 2026 16:00

[scope-core] Add text-to-video input mode support

9401eb5

Pipeline changes for text-to-video mode: - Skip video input for text-to-video pipelines - Input mode detection in wan2_1 pipeline Signed-off-by: emranemran <emran.mah@gmail.com> Signed-off-by: Max Holland <max@livepeer.org>

[infra] Add Docker build workflow for branches

5e709f2

Add GitHub Actions workflow to build Docker images for feature branches. Signed-off-by: emranemran <emran.mah@gmail.com> Signed-off-by: Max Holland <max@livepeer.org>

fix: Resolve merge conflicts and fix TypeScript errors

51a7c16

- Add PipelinesProvider wrapper in App.tsx - Fix ReportBugDialog prop (onClose vs onOpenChange) - Add vaeTypes to PipelineInfo type - Add refreshPipelines to PipelinesContext - Install @radix-ui/react-tabs dependency

fix: Remove incorrect model_dump() call on dict

0051f75

load_params in PipelineLoadRequest is already dict[str, Any], not a Pydantic model, so calling model_dump() on it fails.

fix: Correct method name for cloud WebRTC offer handling

3963438

The method is handle_offer_with_relay, not handle_offer_with_cloud.

fix: Update send_frame_to_cloud to send_frame_to_fal

60c762a

Method was renamed in cloud_connection.py but frame_processor.py was still calling the old name, causing frames not to be sent in video mode.

Set daydream userID based on auth callback

0f70c5d

mjh1 force-pushed the backend-fal-v6 branch from 91c267d to 0f70c5d Compare February 4, 2026 13:28

Fix recording download and tidy up

6837b94

mjh1 force-pushed the backend-fal-v6 branch 2 times, most recently from 5857021 to 73dac11 Compare February 4, 2026 16:59

fix linting errors

2fba0d2

mjh1 force-pushed the backend-fal-v6 branch from 73dac11 to 2fba0d2 Compare February 4, 2026 17:04

remove todo, this is done

67dda45

emranemran requested review from leszko, ryanontheinside and yondonfu February 4, 2026 22:43

emranemran marked this pull request as ready for review February 4, 2026 22:43

emranemran changed the title ~~Backend fal v6~~ Daydream Cloud: Remote GPU Processing in Scope Feb 4, 2026

Update to prod daydream url

ffb4eb4

mjh1 force-pushed the backend-fal-v6 branch from 631c524 to ffb4eb4 Compare February 5, 2026 11:40

Merge branch 'main' into backend-fal-v6

9eaff78

mjh1 force-pushed the backend-fal-v6 branch from c2b8be2 to fbfbc30 Compare February 5, 2026 12:17

Fix after merge

fdf9d23

mjh1 force-pushed the backend-fal-v6 branch from fbfbc30 to fdf9d23 Compare February 5, 2026 12:31

Implement auth UI to match latest wireframe design

d244356

mjh1 force-pushed the backend-fal-v6 branch from e0276a2 to d244356 Compare February 5, 2026 14:39

mjh1 added 2 commits February 5, 2026 16:56

Fix desktop auth flow

ad5cb26

Fix websocket timeout issue

a0fa62e

mjh1 force-pushed the backend-fal-v6 branch from 35aa9a8 to a0fa62e Compare February 6, 2026 12:13

mjh1 and others added 3 commits February 6, 2026 15:32

Point at prod fal env

c2a3525

Auth flow display name + layout + redirect to Account tab

e7c7b46

Signed-off-by: Yondon Fu <yondon.fu@gmail.com>

Connect to cloud in background

8fb3536

Signed-off-by: Yondon Fu <yondon.fu@gmail.com>

yondonfu requested changes Feb 6, 2026

View reviewed changes

emranemran and others added 3 commits February 6, 2026 12:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Daydream Cloud: Remote GPU Processing in Scope#397

Daydream Cloud: Remote GPU Processing in Scope#397
emranemran wants to merge 31 commits intomainfrom
backend-fal-v6

emranemran commented Feb 4, 2026 •

edited

Loading

Uh oh!

yondonfu Feb 6, 2026

Uh oh!

yondonfu Feb 6, 2026

Uh oh!

emranemran Feb 6, 2026

Uh oh!

yondonfu Feb 6, 2026

Uh oh!

yondonfu Feb 6, 2026

Uh oh!

yondonfu Feb 6, 2026

Uh oh!

yondonfu Feb 6, 2026

Uh oh!

yondonfu Feb 6, 2026

Uh oh!

yondonfu Feb 6, 2026

Uh oh!

yondonfu Feb 6, 2026

Uh oh!

yondonfu Feb 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

emranemran commented Feb 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Daydream Cloud: Remote GPU Processing in Scope

Summary

Architecture Overview

Local Mode (Current)

Cloud Mode

Key Components

Backend (scope-server)

Cloud (fal.ai)

Frontend

API Endpoints

Features

Core Functionality

Authentication

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

emranemran commented Feb 4, 2026 •

edited

Loading