[RFC] v2: Per-user rule-based filtering, Domain List system, proxy hardening, and UI overhaul

[jbarwick]

PR Title

[RFC] v2: Per-user rule-based filtering, Domain List system, proxy hardening, and UI overhaul

PR Description

Hi @fifthsegment — I've been working on a substantial set of improvements in my fork and wanted to share them for feedback. This is not a "please merge now" PR — it's more of an RFC / showcase. The branch is v2 and represents about 40 commits / ~44K insertions across 169 files.

Architecture Shift: Global → Per-User Rule-Based Filtering

The most significant change in v2 is moving away from global filter lists toward a fully rule-based filtering system.

In v1, proxy rules could already be defined for a user, and the rule system provided per-user targeting — that foundation was solid. However, the primary filtering mechanisms — blocked sites, exception lists, keyword filters, and DNS blocklists — operated as global pipelines that applied to everyone on the network identically. The per-user rules and the global filters were essentially two separate systems. Additionally, rules in v1 were strictly block rules — they could only deny access, not explicitly grant it.

In v2, those global filtering pipelines have been consolidated into the per-user rule system. There are no longer separate global blocked-site lists or global keyword filters running outside of rules. Instead, every filtering decision flows through the rule pipeline:

• Everything is a rule now — Blocked domains, URL patterns, content-type filtering, and keyword scanning are all configured as properties of individual rules rather than global lists. What was previously a global "blocked sites" list becomes a rule referencing a Domain List with action "block."
• Rules are match + action, not just block lists — Rules are no longer strictly "block" rules. Each rule is a matching rule with a configurable action: Allow or Block. Combined with priority ordering (lower number = higher priority, first match wins), this enables patterns that weren't possible before. For example, a high-priority "Allow" rule can grant a specific user access to educational-site.com even when a lower-priority "Block" rule blocks the entire category for everyone else. Exception lists become simply higher-priority Allow rules.
• Per-user filtering becomes natural — Since all filtering is rule-scoped, targeting specific users is just a field on the rule. A "kids" rule can block adult content while an "adults" rule allows it. An empty user list means "all users."
• DNS filtering is now optional — Domain blocking no longer requires the DNS server. All filtering can be handled entirely by proxy rules. DNS blocking is still available for users who want network-wide domain-level filtering, but it's no longer the only mechanism.
• Reusable Domain Lists — Both DNS filtering and proxy rules share the same DomainListManager and in-memory index. A single curated list (StevenBlack, Hagezi, etc.) can be referenced by multiple rules and by DNS simultaneously.

8-Step Rule Pipeline

Each proxy request is evaluated through a well-defined pipeline, rule by rule in priority order. The first rule that fully matches is applied — subsequent rules are skipped:

1. Rule status — Enabled/disabled, active hours schedule
2. User match — Does the requesting user match the rule's user list?
3. Domain match — Hostname checked against domain patterns (glob wildcards) and domain lists (by list ID)
4. MITM resolution — Per-rule MITM setting: enable, disable, or default (fall back to global setting). Determines whether steps 5–7 can inspect HTTPS traffic.
5. URL regex — Match against the full request URL (requires MITM for HTTPS)
6. Content-type — Match against response Content-Type header (requires MITM for HTTPS)
7. Keyword filter — Scan response body for blocked keywords with score watermark (requires MITM for HTTPS)
8. Action — Allow or Block

If no rule matches after evaluating all rules, the request is allowed (default-allow). Because Allow rules participate in the same priority-ordered pipeline as Block rules, administrators can build layered policies: a broad Block rule at priority 100 blocking a category, with targeted Allow exceptions at priority 10 for specific users or domains.

Domain List System

A new DomainListManager provides CRUD operations, an O(1) in-memory index (map[domain] → set of list IDs), and support for both local lists and URL-sourced blocklists (StevenBlack, Hagezi, AdGuard, Firebog, etc.). DNS and proxy filtering share the same index. Includes automatic migration from the old blocklist format. Full API endpoints and a management UI at /domainlists.

Device Discovery (Foundation)

A new device discovery system using mDNS/Bonjour browsing and passive DNS observation lays the groundwork for per-device filtering in a future release. Devices are identified, tracked, and exposed via API. The data model and record store are in place (with 30+ tests), and the DNS handler already performs passive discovery. The next step would be allowing rules to target devices in addition to users.

Server-Side Rule Tester

Two new API endpoints (POST /api/test/rule-match, POST /api/test/domain-lookup) simulate the full 8-step pipeline against a rule definition without affecting live traffic. The rule editor UI has a built-in tester panel with optional live fetch to verify that a rule behaves as expected before saving.

Real-Time Stats and Logs via SSE

The stats page now uses Server-Sent Events for near-real-time updates, replacing the previous 5-second polling approach. DNS request events are emitted from the handler and streamed to the browser as they happen.

Proxy Hardening

• Streaming 3-path content router (small buffer / scan / stream-through) with configurable GS_MAX_SCAN_SIZE_MB
• Graceful shutdown with in-flight request draining
• Proxy loop detection (Via header + X-GateSentry-Loop + XFF depth limit)
• SSRF protection blocking proxy requests to the admin UI
• HTTPS block page delivery (MITM-signed error page for blocked HTTPS requests)
• WebSocket tunnel support
• TRACE method blocking, response header sanitization

DNS Improvements

• Sharded response cache with configurable TTL and negative caching
• TCP query support (large responses > 512 bytes)
• RFC 2136 dynamic DNS update handler
• WPAD/PAC auto-configuration endpoints
• IPv6 listener support
• All settings configurable via environment variables

UI Overhaul

• Complete Svelte/Carbon rewrite of the rule form matching the 8-step pipeline layout
• New Domain Lists management page (/domainlists)
• DNS page with allow/block domain list assignment sections
• Configurable base path support for reverse proxy deployments
• Docker deployment support with docker-compose.yml and publish script

Test Coverage

• 19 rule-tester endpoint unit tests, 19 domain list unit tests
• Comprehensive proxy deep test script (scripts/proxy_deep_tests.sh) covering MITM, content filtering, keyword scanning, loop detection, and more

Breaking Changes from v1

• Admin UI default port: 10786 → 8080 (configurable via GS_ADMIN_PORT)
• Rule struct expanded: domain_patterns, domain_lists, mitm_action, url_regex_patterns, blocked_content_types, keyword_filter_enabled, active_hours, users
• Settings keys added: dns_domain_lists, dns_whitelist_domain_lists
• Old global blockedDomains map replaced by shared DomainListIndex

How to Try It

git remote add jbarwick https://github.com/jbarwick/Gatesentry.git
git fetch jbarwick v2
git checkout jbarwick/v2
./build.sh && cd bin && ../run.sh
# Admin UI at http://localhost:8080

Happy to answer questions, break this into smaller PRs, or adjust anything based on your feedback. I'm continuing to evaluate and test the software and plan to add parental controls in the near future — per-user and per-device content filtering — so additional changes will be coming.

Also — AI filtering is an interesting idea you have, and I'd like to help out on that front. I have some ideas that may or may not work, but I'd be happy to discuss them with you.

Great project — I've really enjoyed working on it.

[Copilot]

Pull request overview

This RFC PR introduces a v2 architecture with per-user rule-based filtering, a reusable Domain List system shared across DNS/proxy, and several hardening/operability improvements (WPAD, cert generation/reload, SSE, device discovery, base-path support).

Changes:

• Added Domain List CRUD + shared in-memory index and migrated DNS filtering to consume it.
• Added new admin endpoints for WPAD/PAC, DNS cache stats/history/events (SSE), device inventory, and CA generation/reload.
• Refactored routing to support a configurable base path and removed legacy frontend/proxy assets.

Reviewed changes

Copilot reviewed 75 out of 169 changed files in this pull request and generated 18 comments.

Show a summary per file

File	Description
application/webserver/frontend/files/material-mini.css	Removed legacy Material CSS asset
application/webserver/frontend/files/index.html	Removed legacy static index.html (likely replaced by new build pipeline)
application/webserver/frontend/files/bundle.js.LICENSE.txt	Removed legacy bundle license artifact
application/webserver/endpoints/handler_wpad.go	Added WPAD/PAC generation + info endpoints
application/webserver/endpoints/handler_status.go	Expanded status response (DNS/proxy URLs) + LAN IP detection
application/webserver/endpoints/handler_stats.go	Added query parsing and time-bucketed stats support
application/webserver/endpoints/handler_settings.go	Whitelisted new DNS/WPAD settings + hot-reload behaviors
application/webserver/endpoints/handler_rules.go	Relaxed legacy “Domain required” validation to support new rule fields
application/webserver/endpoints/handler_domainlists.go	Added Domain List CRUD + lookup endpoints
application/webserver/endpoints/handler_dns_cache.go	Added DNS cache admin APIs + SSE event stream
application/webserver/endpoints/handler_devices.go	Added device inventory endpoints backed by discovery store
application/webserver/endpoints/handler_certificate.go	Added CA generation + proxy cert reload helpers
application/webserver/endpoints/handler_blocked.go	Added DNS-level blocked page handler
application/webserver/api.go	Added basePath-aware router/subrouter + global CORS middleware
application/webserver.go	Wired rule manager + domain list manager + base path into server registration
application/types/rule.go	Expanded rule schema for domain patterns/lists and keyword flag
application/start.go	Initialized shared DomainListManager + migration + async list load
application/rules_test.go	Added tests for domain globbing, domain list matching, block types
application/rules.go	Implemented glob matching + domain-list-aware rule matching + MITM “default” behavior
application/responder/responder.go	Updated proxy block page HTML template/layout
application/proxy/structures.go	Removed legacy proxy wrapper types
application/proxy/session.go	Removed legacy goproxy session glue
application/proxy/ext/html.go	Removed legacy goproxy HTML handler extension
application/proxy/certs.go	Removed hardcoded CA cert/key constants
application/logger/logger.go	Added SSE-style subscribers + log grouping-by-format + DB maintenance
application/go.mod	Removed old goproxy dependencies
application/filters/loader.go	Removed blocked MIME filter registration (migrating to rules)
application/filters/filter-blocked-mimes.go	Removed blocked MIME filter implementation
application/filters.go	Removed goproxy ConditionalMitm hook
application/domainlist/types.go	Added DomainList and summary types
application/domainlist/migrate.go	Added migration from legacy DNS list config to Domain Lists
application/domainlist/manager.go	Added DomainListManager CRUD + index refresh/load
application/domainlist/loader.go	Added URL/local domain parsing + blocklist downloader
application/domainlist/index.go	Added O(1) DomainListIndex with thread-safe ops
application/domainlist/domainlist_test.go	Added tests for index/manager/loader/migration helpers
application/dns/utils/network.go	Improved local IP selection logic
application/dns/server/ddns.go	Added RFC 2136 DDNS update handling
application/dns/scheduler/scheduler.go	Scheduler now refreshes domain lists via DomainListManager
application/dns/http/http-server.go	Removed legacy DNS HTTP/HTTPS server
application/dns/filter/internal-records.go	Switched filter mutex to RWMutex
application/dns/filter/exception-records.go	Switched filter mutex to RWMutex
application/dns/filter/domains.go	Removed legacy blocklist download path; now initializes internal/exception only
application/dns/discovery/types.go	Added device + record model for discovery
application/dns/discovery/passive_test.go	Added passive discovery unit tests
application/dns/discovery/passive.go	Added passive discovery via ARP correlation
application/dns/discovery/mdns.go	Added Bonjour/mDNS browser feeding device store
application/dns/cache/recorder_test.go	Added cache recorder tests
application/dns/cache/recorder.go	Added per-minute cache stats recorder in BuntDB
application/dns/cache/events.go	Added cache EventBus for SSE
application/dns.go	DNS server thread now accepts DomainListManager
application/consumptionupdater.go	Removed unused ticker goroutine stub
application/bonjour.go	Advertised admin UI + proxy via Bonjour with base path/port
README.md	Updated documented ports and access URL
Makefile	Updated integration test harness to use port 8080 and new health URL
Dockerfile	Added runtime-only container packaging (prebuilt binary)
DOCKERHUB_README.md	Added Docker Hub-focused documentation
DNS_CACHE_FEATURE.md	Added DNS cache feature documentation
AGENTS.md	Added AI-agent/project operational notes
.dockerignore	Restricted Docker build context to prebuilt binary + Dockerfile

Comments suppressed due to low confidence (1)

application/domainlist/manager.go:1

• Deduplication uses the raw stored dl.Domains values as map keys, but incoming domains are normalized (lowercased, trailing dot stripped). If existing entries are not normalized (e.g., legacy data), duplicates can slip through (Example.com vs example.com). Normalize existing domains when building the existing map so dedup is consistent.

package domainlist

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

This returns the address of the range loop variable (dl), which is a copy that gets reused each iteration. Callers can receive a pointer to a value that is not the actual slice element. Fix by iterating by index and returning &lists[i], or by assigning to a new variable and returning a pointer that doesn’t escape the loop variable.

Suggested change

	for _, dl := range lists {
	if dl.ID == id {
	return &dl, nil
	for i := range lists {
	if lists[i].ID == id {
	return &lists[i], nil

[Copilot]

GSApiDomainListCheck only checks membership for Source == \"local\", so it will always report found=false for URL-sourced lists (even though those are the main large lists). Since the manager already exposes IsDomainInList, use that (with the same normalization) so both local and URL lists are supported and the check uses the shared in-memory index.

Suggested change

	// We need access to the index — get it from the concrete manager.
	// The interface doesn't expose the index directly, but we can check
	// by trying to get the list and checking membership.
	dl, err := domainListManager.GetList(id)
	if err != nil {
	http.Error(w, err.Error(), http.StatusInternalServerError)
	return
	}
	if dl == nil {
	http.Error(w, "Domain list not found", http.StatusNotFound)
	return
	}
	// For local lists, check the domains array directly
	found := false
	if dl.Source == "local" {
	domain = normalizeForCheck(domain)
	for _, d := range dl.Domains {
	if normalizeForCheck(d) == domain {
	found = true
	break
	}
	}
	}
	w.Header().Set("Content-Type", "application/json")
	json.NewEncoder(w).Encode(map[string]interface{}{
	"domain": domain,
	// Normalize the domain before checking membership to match how domains are stored.
	normalizedDomain := normalizeForCheck(domain)
	// Use the manager's shared index so both local and URL-sourced lists are supported.
	found := domainListManager.IsDomainInList(normalizedDomain, id)
	w.Header().Set("Content-Type", "application/json")
	json.NewEncoder(w).Encode(map[string]interface{}{
	"domain": normalizedDomain,

[Copilot]

The block page interpolates host (derived from r.Host) directly into HTML without escaping, which can allow HTML injection/XSS if a client supplies a crafted Host header. Escape the host with html.EscapeString(host) (from html or html/template) before inserting it into the response.

[Copilot]

The block page interpolates host (derived from r.Host) directly into HTML without escaping, which can allow HTML injection/XSS if a client supplies a crafted Host header. Escape the host with html.EscapeString(host) (from html or html/template) before inserting it into the response.

[Copilot]

The block page interpolates host (derived from r.Host) directly into HTML without escaping, which can allow HTML injection/XSS if a client supplies a crafted Host header. Escape the host with html.EscapeString(host) (from html or html/template) before inserting it into the response.

[Copilot]

If basePath is configured with a trailing slash (e.g., "/gatesentry/"), basePath+\"/\" produces a double-slash redirect target. Normalize basePath (ensure it starts with a single leading '/', and trim any trailing '/') before building the redirect and PathPrefix.

[Copilot]

The README now states the admin UI runs on port 80, but other repo docs in this PR (e.g., AGENTS.md and Makefile) reference 8080 as the default admin port. Please reconcile the documented defaults across files to avoid confusing users, and also fix the missing space after Note: (Note: Ensure ...).

[Copilot]

The README now states the admin UI runs on port 80, but other repo docs in this PR (e.g., AGENTS.md and Makefile) reference 8080 as the default admin port. Please reconcile the documented defaults across files to avoid confusing users, and also fix the missing space after Note: (Note: Ensure ...).

[Copilot]

The README now states the admin UI runs on port 80, but other repo docs in this PR (e.g., AGENTS.md and Makefile) reference 8080 as the default admin port. Please reconcile the documented defaults across files to avoid confusing users, and also fix the missing space after Note: (Note: Ensure ...).

[Copilot]

http.Error will set Content-Type: text/plain; charset=utf-8, but the body is JSON. For API consistency, write a JSON response with Content-Type: application/json (and ideally a struct) so clients don’t need to special-case error parsing.