feat(notifier): add process_writes WAL trigger entry point

[caterryan]

Summary

• Add process_writes WAL trigger entry point to the notifier plugin, enabling pub-sub notification dispatch when rows are written to a table
• Add new mode to process_request for env-var-based credential resolution via trigger args (*_env pattern)
• Batch async senders (Slack, Discord, HTTP) concurrently via asyncio.gather() in a single event loop
• Existing process_request legacy code path and all sender functions are completely untouched — no breaking changes

Design

Add a WAL trigger entry point (process_writes) to the notifier plugin so it can send notifications both when it receives an HTTP request and when lines are written to a table. The table acts as a pub-sub notification channel: alert plugins publish rows, notifier triggers subscribe and dispatch. Secrets never touch the table — each notifier resolves its own credentials from environment variables, referenced by name through trigger args.

Goals

• Support WAL triggers alongside the existing HTTP trigger
• Define a clear boundary (contract) between how data is received and how notifications are dispatched
• Keep secrets out of tables and trigger args — credentials live in env vars only
• Support N:M fan-out: multiple alert plugins write to a table, multiple notifier triggers fire on the same table
• No breaking changes to the existing process_request legacy code path

Non-Goals

• Refactoring SMS/WhatsApp sender internals (DRY consolidation is out of scope)
• Adding new sender types
• Multi-file module split (plugin must remain a single deployable file)

Architecture

The table is a notification channel (pub-sub message bus):

              PUBLISHERS                          SUBSCRIBERS
              (alert plugins write rows)          (notifier triggers fire on writes)

 ┌──────────────┐  ┌──────────────┐       ┌──────────────┐  ┌──────────────┐
 │  threshold   │  │   deadman    │       │    slack     │  │     sms      │
 │  alert       │  │   alert      │       │  notifier    │  │  notifier    │
 │  plugin      │  │   plugin     │       │  (trigger)   │  │  (trigger)   │
 └──────┬───────┘  └──────┬───────┘       └──────┬───────┘  └──────┬───────┘
        │                 │                      │                 │
        └────────┬────────┘                      └────────┬────────┘
                 │                                        │
                 ▼                                        ▼
        ┌────────────────────────────────────────────────────────┐
        │              notifications table                       │
        │              (the contract)                             │
        └────────────────────────────────────────────────────────┘

• Publishers (alert plugins) write rows containing the notification message and metadata
• Subscribers (notifier triggers) fire on every write and dispatch via their configured sender
• Fan-out is structural: attach N triggers to the same table → every row triggers N notifications
• Fan-in is natural: M alert plugins write to the same table → all messages reach all subscribers
• Secrets never touch the table. Each notifier owns its credentials via trigger args (env var names) + env vars (secret values)

[Aliaksei-Kharlap]

we need to add a section with possible arguments for onwrite so that the UI can display

[Aliaksei-Kharlap]

Since we now also require an args setting for request, we need to add a section for it as well. I honestly don't remember what it should be called for the HTTP trigger type; I'll check with the team

[Aliaksei-Kharlap]

must be http_args_config

[Aliaksei-Kharlap]

I checked with the team, the UI doesn't support this parameter yet, but they'll fix it

[Aliaksei-Kharlap]

I think sender_type should be in the contract table

[Aliaksei-Kharlap]

In this case, we'll be able to use a single trigger for several possible sending paths at once. Currently, the configuration is limited to one channel per trigger. To configure several channels at once, we need to pass not just one channel in the webhook_url_env arguments, but split them into separate ones (slack_webhook_url_env, sms_webhook_url_env, whatsapp_webhook_url_env, http_webhook_url_env). This will give us a single trigger that supports multiple channels simultaneously.

[Aliaksei-Kharlap]

It's the same here, I think it's better to leave sender_type in the request body

[Aliaksei-Kharlap]

I think we need to specify the ID somewhere else in the logs, not just in the case of a skip. So that we can track which specific message is being processed