onthebench.ai benchmarks GoModel: a review of our setup for fairness

[MattJackson]

Hi, I run onthebench.ai, an open benchmark that measures LLM-gateway overhead (latency, throughput, memory, streaming, protocol translation) on neutral hardware. GoModel is one of the gateways on the board, and I want to make sure I'm testing it fairly.

How it works, so there are no surprises:

• Every gateway runs on the same rig, same mock upstream, same load, same CPU pinning, no per-gateway special-casing.
• Each gateway is defined by a single file, gateways/gomodel/gateway.sh, which declares how to build, launch, and probe it. That file is the whole story of how I configured yours.
• Every number regenerates from committed JSON; the method is documented here and the whole thing is open source and re-runnable.

My guiding rule is that a failure is my bug until proven the gateway's. If a cell shows red or a number looks off, I'd rather find out I configured GoModel wrong than publish something unfair. So two asks, both optional:

1. Look at your results (your page) and your gateway.sh. If I've mis-set a flag, a version, an endpoint, or declared a capability you don't claim, tell me or open a PR, you're welcome to own your own gateway.sh.
2. If the setup looks fair, a thumbs-up is genuinely useful too.

Full disclosure: this is built and operated by the Busbar team, and busbar is also one of the entrants. Same harness for everyone, fully open, so you can verify exactly what I'm running.

Thanks for building GoModel.

[SantiagoDePolonia]

Hello Matt,

Thank you for creating this benchmark. As I wrote when launching GoModel on Hacker News, I would genuinely love to see more and more independent benchmarks.

I am also the author of what I believe is currently the most reliable and reproducible comparison of the most influential AI gateways. It runs entirely on AWS and documents the infrastructure, methodology, payloads, and raw results: Benchmarking AI Gateways: LiteLLM, Bifrost, GoModel and Portkey.

That said, in its current state, your benchmark looks heavily vibe-coded and does not appear to have received enough human review. (The screenshot below - it's an AI slop there).

I hope you will be more open to technical feedback than OpenAI or OpenRouter. :)

Regarding the GoModel configuration

Considering your methodology and your stated goal-measuring the overhead added by each gateway-the current GoModel configuration is not fair yet.

The main problems are:

1. The benchmark configures OpenAI, Anthropic, Gemini, and Bedrock simultaneously, even though the primary latency and memory lanes test only OpenAI. This unnecessarily initializes additional providers and affects idle memory.

2. The "Passthrough" benchmark calls /v1/chat/completions. In GoModel, this is the OpenAI-compatible translated route, not the native provider passthrough route. GoModel's real passthrough endpoint is:

/p/openai/chat/completions

3. LOGGING_ENABLED=false disables audit logging, but it does not disable GoModel's informational HTTP request logs. Without LOG_LEVEL=error, the benchmark may generate request-log output for every request.

4. Usage tracking, budgets, rate limits, failover, retries, the circuit breaker, admin endpoints, live logs, MCP, and realtime support remain enabled by default, despite not being exercised by the plain proxy benchmark.

5. The governed benchmark reports GoModel as not supporting native governance. GoModel does support managed API keys, rate limits, token limits, and budgets; the corresponding benchmark hooks have simply not been configured.

For a plain, ungoverned proxy-overhead lane, I propose the following configuration:

PORT=8080

OPENAI_BASE_URL=http://127.0.0.1:${MOCK_PORT}/v1
OPENAI_API_KEY=dummy
OPENAI_MODELS=gpt-4o-mini

CONFIGURED_PROVIDER_MODELS_MODE=allowlist
MODELS_ENABLED_BY_DEFAULT=true
PROVIDER_RECHECK_INTERVAL=0

LOG_LEVEL=error

LOGGING_ENABLED=false
USAGE_ENABLED=false
ENFORCE_RETURNING_USAGE_DATA=false
USAGE_PRICING_RECALCULATION_ENABLED=false

BUDGETS_ENABLED=false
RATE_LIMITS_ENABLED=false

FAILOVER_ENABLED=false
RETRY_MAX_RETRIES=0
CIRCUIT_BREAKER_FAILURE_THRESHOLD=0

METRICS_ENABLED=false

ADMIN_ENDPOINTS_ENABLED=false
ADMIN_UI_ENABLED=false
DASHBOARD_LIVE_LOGS_ENABLED=false

MCP_ENABLED=false
REALTIME_ENABLED=false
GUARDRAILS_ENABLED=false

RESPONSE_CACHE_SIMPLE_ENABLED=false
SEMANTIC_CACHE_ENABLED=false

ENABLE_PASSTHROUGH_ROUTES=true
ENABLED_PASSTHROUGH_PROVIDERS=openai
ALLOW_PASSTHROUGH_V1_ALIAS=false

STORAGE_TYPE=sqlite

The plain passthrough lane should use:

GW_PATH=/p/openai/chat/completions
GW_DIRECT_PATH=/v1/chat/completions
GW_MODEL=gpt-4o-mini

GW_DIRECT_PATH should be implemented as a generic optional manifest setting, not as GoModel-specific logic:

GW_DIRECT_PATH="${GW_DIRECT_PATH:-$GW_PATH}"

This allows the load generator to call GoModel's real passthrough route while sending the direct baseline to the mock's canonical OpenAI endpoint.

For the translation lane, GoModel should continue using its translated API:

gw_xlate_env() {
  GW_PATH=/v1/chat/completions
  GW_MODEL=gpt-4o-mini
}

For the protocol matrix:

GW_MATRIX_PATH_OPENAI=/v1/chat/completions

Each matrix egress column should configure only the provider being tested. For example:

• OpenAI column: configure only OpenAI.
• Anthropic column: configure only Anthropic.
• Gemini column: configure only Gemini.
• Bedrock column: configure only Bedrock.

The matrix already relaunches the gateway for each egress, so there is no reason to keep four provider clients configured simultaneously. This is especially relevant to the idle-memory measurement.

I would also disable GoModel's external model metadata catalog for this benchmark and use only the explicit model allowlist:

cache:
  model:
    model_list:
      url: ""

This file can be mounted as:

-v "$GW_DIR/config.yaml:/app/config/config.yaml:ro"

I would deliberately not set GOMEMLIMIT, GOGC, custom HTTP connection limits, or an artificially small request-body limit. I would also keep all response caching disabled. Those changes could optimize specifically for the benchmark or invalidate the comparison by serving repeated requests without contacting the mock.

Finally, the governed lane should be configured separately with GoModel authentication, managed keys, rate limits, usage tracking, and budgets enabled. Those features should not remain active in the plain proxy benchmark, but GoModel should not be reported as lacking them simply because the manifest has not wired the governed hooks.

With these corrections, the benchmark would measure three clear and honest things:

• Native passthrough overhead.
• Protocol translation overhead.
• Governance overhead.

I would be happy to review the updated gateway.sh or provide a PR once we agree on the configuration and lane separation.

[MattJackson]

Thanks for responding, this is the collaboration I wanted and I think true benchmarks needs.

I have to disagree though on using a highly tuned config purely for benchmarking purposes. Our aim is for real world configs that users would use and this the number represent what a user can expect. If you genuinely feel logging, failover, admin, budgets, etc are disabled by default and not core features then we can discuss adjusting.

As for the endpoints: You are an interesting case in that you have passthrough optimized endpoints and translate endpoints. Most others have 1 endpoint and do the routing. I also believe this is how most users would use a gateway. They don't want (or shouldn't have) to think about what endpoint to hit for performance. Different philosophies here i suppose.

You state "The benchmark configures OpenAI, Anthropic, Gemini, and Bedrock simultaneously, even though the primary latency and memory lanes test only OpenAI." - This goes to having 1 config for all tests. We again do not believe that for test X we use config X that is optimized for it. It is not how a real world deployment would be. If you want the matrix tested then we configure as such and then your memory value is realistic.

I am working this keep this fair and consistent for all entrants and some agree and some will disagree, the joys of benchmarking...

Keep the feedback coming, I will continue to review your message as I work on this. We will be showing on the website the config used for the benchmark to help make it easier for end users to see the config used for all tests. This should alleviate any unknowns for users about why gateway x <> gateway y, they can confirm the configs are setup the same and each maintainer can also review easier everybody else's config.

[SantiagoDePolonia]

"Most others have 1 endpoint and do the routing (...)" - this statement is totally false.

I'm not sure who "most others" refers to, but:

• docs.litellm.ai/docs/pass_through/intro
• docs.getbifrost.ai/integrations/passthrough

"(...)They don't want (or shouldn't have) to think about what endpoint to hit for performance.(...)" - The presupposition that the only benefit of a passthrough API is performance is totally wrong.