Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,4 +48,4 @@ jobs:
uses: codecov/codecov-action@v5
with:
token: ${{ secrets.CODECOV_TOKEN }}
slug: flowdacity/flowdacity-queue-server
slug: flowdacity/queue-server
86 changes: 45 additions & 41 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,27 +1,23 @@
[![Run tests and upload coverage](https://github.com/flowdacity/flowdacity-queue-server/actions/workflows/test.yml/badge.svg)](https://github.com/flowdacity/flowdacity-queue-server/actions/workflows/test.yml)
[![codecov](https://codecov.io/github/flowdacity/flowdacity-queue-server/graph/badge.svg?token=9AK3GR856C)](https://codecov.io/github/flowdacity/flowdacity-queue-server)
[![Run tests and upload coverage](https://github.com/flowdacity/queue-server/actions/workflows/test.yml/badge.svg)](https://github.com/flowdacity/queue-server/actions/workflows/test.yml)
[![codecov](https://codecov.io/github/flowdacity/queue-server/graph/badge.svg?token=9AK3GR856C)](https://codecov.io/github/flowdacity/queue-server)

Flowdacity Queue Server
=======================

An async HTTP API for the [Flowdacity Queue (FQ)](https://github.com/flowdacity/flowdacity-queue) core, built with Starlette and Uvicorn. It keeps the original SHARQ behavior (leaky-bucket rate limiting and dynamic queues) while modernizing the stack.
An async HTTP API for [Flowdacity Queue (FQ)](https://github.com/flowdacity/queue-engine), built with Starlette and Uvicorn.

## Prerequisites

- Python 3.12+
- Redis 7+ reachable from the server
- A Flowdacity Queue config file (see `default.conf` for a starter)
- Redis 7+

## Installation

Clone the repo and install the package plus dev tools (uses [`uv`](https://github.com/astral-sh/uv) by default):

```bash
uv sync --group dev
# or: uv pip install --system .
```

If you prefer pip/venv without `uv`:
If you prefer a virtualenv without `uv`:

```bash
python -m venv .venv
Expand All @@ -32,29 +28,48 @@ pip install pytest pytest-cov

## Configuration

- Point the server at your FQ config via `FQ_CONFIG` (defaults to `./default.conf`).
- `default.conf` defines three sections:
- `[fq]` queue behavior (intervals, requeue limits).
- `[fq-server]` host/port for the HTTP server (used by Docker/local defaults).
- `[redis]` connection details for your Redis instance.
- Copy and tweak as needed:
The server reads all queue and Redis settings from environment variables. No config file is required.
These application settings are validated at startup by `QueueServerSettings` with `pydantic-settings`.

| Variable | Default | Description |
| --- | --- | --- |
| `JOB_EXPIRE_INTERVAL` | `1000` | Milliseconds before a dequeued job is considered expired. |
| `JOB_REQUEUE_INTERVAL` | `1000` | Milliseconds between expired-job requeue passes. |
| `DEFAULT_JOB_REQUEUE_LIMIT` | `-1` | Default retry limit. `-1` retries forever. |
| `ENABLE_REQUEUE_SCRIPT` | `true` | Enables the background requeue loop. |
| `LOG_LEVEL` | `INFO` | Application log level. |
| `REDIS_DB` | `0` | Redis database number. |
| `REDIS_KEY_PREFIX` | `fq_server` | Prefix used for Redis keys. |
| `REDIS_CONN_TYPE` | `tcp_sock` | Redis connection type: `tcp_sock` or `unix_sock`. |
| `REDIS_HOST` | `127.0.0.1` | Redis host for TCP connections. |
| `REDIS_PORT` | `6379` | Redis port for TCP connections. |
| `REDIS_PASSWORD` | empty | Redis password. |
| `REDIS_CLUSTERED` | `false` | Enables Redis Cluster mode. |
| `REDIS_UNIX_SOCKET_PATH` | `/tmp/redis.sock` | Redis socket path when `REDIS_CONN_TYPE=unix_sock`. |

Boolean env vars accept only `true` or `false`.

`PORT` is not part of `QueueServerSettings`. It is runtime launcher configuration used by the container entrypoint or by the `uvicorn` CLI, so pass it as a launcher environment variable or `--port` argument.

## Run locally

Start Redis:

```bash
cp default.conf local.conf
# edit local.conf to match your Redis host/port/password
make redis-up
```

## Run the server locally
Run the API:

```bash
# ensure Redis is running (make redis starts a container)
make redis

# start the ASGI server
FQ_CONFIG=./local.conf uv run uvicorn asgi:app --host 0.0.0.0 --port 8080
export PORT=8300
export REDIS_HOST=127.0.0.1
uv run uvicorn asgi:app --host 0.0.0.0 --port "${PORT}"
```

Docker Compose is also available:
## Docker

`docker-compose.yml` now passes the queue settings through env vars, so there is no mounted config file:

```bash
docker compose up --build
Expand All @@ -63,34 +78,23 @@ docker compose up --build
## API quick start

```bash
# health
curl http://127.0.0.1:8080/
curl http://127.0.0.1:8300/

# enqueue a job
curl -X POST http://127.0.0.1:8080/enqueue/sms/user42/ \
curl -X POST http://127.0.0.1:8300/enqueue/sms/user42/ \
-H "Content-Type: application/json" \
-d '{"job_id":"job-1","payload":{"message":"hi"},"interval":1000}'

# dequeue
curl http://127.0.0.1:8080/dequeue/sms/
curl http://127.0.0.1:8300/dequeue/sms/

# mark finished
curl -X POST http://127.0.0.1:8080/finish/sms/user42/job-1/
curl -X POST http://127.0.0.1:8300/finish/sms/user42/job-1/

# metrics
curl http://127.0.0.1:8080/metrics/
curl http://127.0.0.1:8080/metrics/sms/user42/
curl http://127.0.0.1:8300/metrics/
curl http://127.0.0.1:8300/metrics/sms/user42/
```

All endpoints return JSON; failures surface as HTTP 4xx/5xx with a `status` field in the body.

## Testing

Redis must be available. With dev deps installed:

```bash
uv run pytest
# or
make test
```

Expand Down
34 changes: 21 additions & 13 deletions asgi.py
Original file line number Diff line number Diff line change
@@ -1,19 +1,27 @@
# Copyright (c) 2025 Flowdacity Team. See LICENSE.txt for details.
# ASGI application entrypoint for Flowdacity Queue (FQ) Server

import os
from fq_server import setup_server

# read config path from env variable, use default if not set
fq_config_path = os.environ.get("FQ_CONFIG")
if fq_config_path is None:
print(
"Warning: FQ_CONFIG environment variable not set. Using default config path './default.conf'."
)
fq_config_path = "./default.conf"
fq_config_path = os.path.abspath(fq_config_path)

server = setup_server(fq_config_path)
import logging

from fq_server import QueueServerSettings, setup_server


def configure_logging(log_level: str) -> None:
level = getattr(logging, log_level)
root_logger = logging.getLogger()

if not root_logger.handlers:
logging.basicConfig(
level=level,
format="%(asctime)s %(levelname)s [%(name)s] %(message)s",
)

logging.getLogger("fq_server").setLevel(level)


settings = QueueServerSettings.from_env()
configure_logging(settings.log_level)
server = setup_server(settings.to_fq_config())

# ASGI app exposed for Uvicorn/Hypercorn
app = server.app
17 changes: 0 additions & 17 deletions default.conf

This file was deleted.

18 changes: 13 additions & 5 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,20 @@ services:
app:
build: .
environment:
- FQ_CONFIG=/app/default.conf
- PORT=8080
JOB_EXPIRE_INTERVAL: 1000
JOB_REQUEUE_INTERVAL: 1000
DEFAULT_JOB_REQUEUE_LIMIT: -1
ENABLE_REQUEUE_SCRIPT: "true"
LOG_LEVEL: INFO
REDIS_DB: 0
REDIS_KEY_PREFIX: fq_server
REDIS_CONN_TYPE: tcp_sock
REDIS_HOST: redis
REDIS_PORT: 6379
REDIS_PASSWORD: ""
REDIS_CLUSTERED: "false"
ports:
- "8080:8080"
volumes:
- ./default.conf:/app/default.conf:ro
- "8300:8300"
depends_on:
redis:
condition: service_healthy
Expand Down
8 changes: 4 additions & 4 deletions docs/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -85,17 +85,17 @@ qthelp:
@echo
@echo "Build finished; now you can run "qcollectiongenerator" with the" \
".qhcp project file in $(BUILDDIR)/qthelp, like this:"
@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/fqserver.qhcp"
@echo "# qcollectiongenerator $(BUILDDIR)/qthelp/queue-server.qhcp"
@echo "To view the help file:"
@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/fqserver.qhc"
@echo "# assistant -collectionFile $(BUILDDIR)/qthelp/queue-server.qhc"

devhelp:
$(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp
@echo
@echo "Build finished."
@echo "To view the help file:"
@echo "# mkdir -p $$HOME/.local/share/devhelp/fqserver"
@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/fqserver"
@echo "# mkdir -p $$HOME/.local/share/devhelp/queue-server"
@echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/queue-server"
@echo "# devhelp"

epub:
Expand Down
2 changes: 1 addition & 1 deletion docs/_templates/layout.html
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{% extends "!layout.html" %}

{%- block extrahead %}
<a href="https://github.com/plivo/fq-server"><img style="position: absolute; top: 0; right: 0; border: 0;" src="https://camo.githubusercontent.com/e7bbb0521b397edbd5fe43e7f760759336b5e05f/68747470733a2f2f73332e616d617a6f6e6177732e636f6d2f6769746875622f726962626f6e732f666f726b6d655f72696768745f677265656e5f3030373230302e706e67" alt="Fork me on GitHub" data-canonical-src="https://s3.amazonaws.com/github/ribbons/forkme_right_green_007200.png"></a>
<a href="https://github.com/flowdacity/queue-server"><img style="position: absolute; top: 0; right: 0; border: 0;" src="https://camo.githubusercontent.com/e7bbb0521b397edbd5fe43e7f760759336b5e05f/68747470733a2f2f73332e616d617a6f6e6177732e636f6d2f6769746875622f726962626f6e732f666f726b6d655f72696768745f677265656e5f3030373230302e706e67" alt="Fork me on GitHub" data-canonical-src="https://s3.amazonaws.com/github/ribbons/forkme_right_green_007200.png"></a>
{% endblock %}
Loading
Loading