Skip to content

fix(mcp): deny MCP access when a named entitlement cannot be read - #35160

Merged
yassin-berriai merged 1 commit into
litellm_internal_stagingfrom
litellm_mcp_entitlement_fault_semantics
Jul 31, 2026
Merged

fix(mcp): deny MCP access when a named entitlement cannot be read#35160
yassin-berriai merged 1 commit into
litellm_internal_stagingfrom
litellm_mcp_entitlement_fault_semantics

Conversation

@yassin-berriai

@yassin-berriai yassin-berriai commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

TLDR

Problem this solves:

  • An unreadable entitlement silently removed an MCP ceiling
  • Key auth and keyless auth disagreed on the same fault

How it solves it:

  • A named but unreadable permission now denies
  • A lookup that cannot tell still places no ceiling

Relevant issues

Linear ticket

Resolves LIT-4960

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

  • I have added meaningful tests
  • My PR passes all CI/CD checks (e.g., lint, format, unit tests)
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Live proxy on port 4960 against its own postgres, with a real upstream MCP server (deepwiki, no mocks). Two virtual keys, each in a team that grants the server; the first key's organization carries an MCP entitlement, the second key's organization carries none. The entitlement is then made unreadable, which is the fault the fix is about, by corrupting the stored permission row so it no longer deserializes; the link from the organization to it stays intact, so the gateway still knows the entitlement exists

Setup (captured at 551e5d0, both builds share it)

curl -s -X POST http://localhost:4960/organization/new -H "Authorization: Bearer $M" -H 'Content-Type: application/json' \
  -d '{"organization_alias":"org-entitled","object_permission":{"mcp_servers":["deepwiki"]}}'
curl -s -X POST http://localhost:4960/organization/new -H "Authorization: Bearer $M" -H 'Content-Type: application/json' \
  -d '{"organization_alias":"org-plain"}'
# a team per org, each granting deepwiki, then a key per team also granting deepwiki
docker exec litswarm-4960-pg psql -U litellm -d litellm -c \
  "UPDATE \"LiteLLM_ObjectPermissionTable\" SET mcp_tool_permissions = '\"corrupted-not-an-object\"'::jsonb
   WHERE object_permission_id = '8a8ac833-21cf-44ba-8c20-8953e9772140';"

Before, at 551e5d0

curl -s -X POST http://localhost:4960/mcp-rest/tools/call -H "x-litellm-api-key: $K1" -H 'Content-Type: application/json' \
  -d '{"name":"deepwiki-read_wiki_structure","server_id":"266024044f9612bf481c78f6cfef1ff0","arguments":{"repoName":"BerriAI/litellm"}}'

{"_meta":null,"content":[{"type":"text","text":"Available pages for BerriAI/litellm:\n\n- 1 Overview\n  - 1.1 Getting Started and Repo Orientation\n  - 1.2 System Architecture at a Glance\n ...

and the proxy log says what happened

WARNING: user_api_key_auth_mcp.py:2311 - Failed to get allowed MCP servers for org: org '9184c8dc-3e59-42f0-be25-04b21968a6b7' names object_permission_id '8a8ac833-21cf-44ba-8c20-8953e9772140' which could not be loaded
WARNING: user_api_key_auth_mcp.py:1445 - MCP org ceiling unresolved for org_id='9184c8dc-3e59-42f0-be25-04b21968a6b7'; leaving uncapped (key auth)

After, at 705af58, same database, same request

curl -s -X POST http://localhost:4960/mcp-rest/tools/call -H "x-litellm-api-key: $K1" -H 'Content-Type: application/json' \
  -d '{"name":"deepwiki-read_wiki_structure","server_id":"266024044f9612bf481c78f6cfef1ff0","arguments":{"repoName":"BerriAI/litellm"}}'

{"detail":{"error":"access_denied","message":"The key is not allowed to access server 266024044f9612bf481c78f6cfef1ff0"}}
WARNING: user_api_key_auth_mcp.py:1460 - Denying MCP access, entitlement unreadable: org '9184c8dc-3e59-42f0-be25-04b21968a6b7' names object_permission_id '8a8ac833-21cf-44ba-8c20-8953e9772140' which could not be loaded

Negative control, at 705af58, the key whose organization has no entitlement at all is untouched

curl -s -X POST http://localhost:4960/mcp-rest/tools/call -H "x-litellm-api-key: $K2" -H 'Content-Type: application/json' \
  -d '{"name":"deepwiki-read_wiki_structure","server_id":"266024044f9612bf481c78f6cfef1ff0","arguments":{"repoName":"BerriAI/litellm"}}'

{"_meta":null,"content":[{"type":"text","text":"Available pages for BerriAI/litellm:\n\n- 1 Overview\n  - 1.1 Getting Started and Repo Orientation\n ...

Second negative control, at 705af58, restoring the entitlement to a readable state serves the first key again, so this is not a blanket deny

docker exec litswarm-4960-pg psql -U litellm -d litellm -c \
  "UPDATE \"LiteLLM_ObjectPermissionTable\" SET mcp_tool_permissions = NULL WHERE object_permission_id = '8a8ac833-21cf-44ba-8c20-8953e9772140';"
curl -s -X POST http://localhost:4960/mcp-rest/tools/call -H "x-litellm-api-key: $K1" -H 'Content-Type: application/json' \
  -d '{"name":"deepwiki-read_wiki_structure","server_id":"266024044f9612bf481c78f6cfef1ff0","arguments":{"repoName":"BerriAI/litellm"}}'

{"_meta":null,"content":[{"type":"text","text":"Available pages for BerriAI/litellm:\n\n- 1 Overview\n  - 1.1 Getting Started and Repo Orientation\n ...

Type

🐛 Bug Fix

Changes

An MCP permission level answers which servers and tools it permits, and a level that answers nothing places no restriction. For a key-authenticated caller the end user, agent and organization levels were reading a lookup fault as that same answer, so a ceiling an operator had configured quietly disappeared for as long as the fault lasted and the caller kept the wider scope its key grants. The keyless gateway-admitted path already failed closed at all of the same points, so the two caller shapes gave opposite answers to one fault

Those three levels now separate the two fault classes, which is what the per-user level added in #35146 already does; that level therefore needs no change here. A principal row that names an object_permission_id whose contents cannot be read is a known entitlement with unknown contents, so it raises UnloadableEntitlementError and the resolver denies, on the servers axis and the tools axis alike. A lookup that fails before we can tell whether the principal is entitled at all still places no ceiling, because that is the state that existed before the level did, and denying there would refuse MCP to the majority of callers, who have no such entitlement configured, during any cold-cache or database fault. Blanket fail-closed was considered and rejected on exactly that ground: it trades a global availability regression for a partial fix

One helper, _load_named_object_permission, is the single place that fault is minted, so the three levels cannot drift on what counts as evidence of an entitlement. The contract, including why key auth and the keyless path diverge, is written into the MCPRequestHandler class docstring and restated per level, since it was undocumented product behavior until now

The keyless path is unchanged; it denied on any fault before and still does. No existing test changed behavior: rebased onto 3c2264cfac, the mapped directory reports 2802 passing against that base's 2792, with the same 23 pre-existing failures on both, so the 10 added tests are the whole delta. Each new deny is mutation-checked: reverting this file to the base fails the five deny tests and leaves the five no-ceiling controls green, and dropping either the tool-axis deny or the org re-raise fails a test on its own

The two new deny paths reuse the resolvers' existing except blocks rather than adding clauses of their own, so the type-discipline budget sees no new mutable-collection construction; the resolved answers and the log lines are the same either way

Final Attestation

  • The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

@yassin-berriai

Copy link
Copy Markdown
Contributor Author

@greptileai

@greptile-apps

greptile-apps Bot commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

The PR makes MCP authorization deny access when a principal names an entitlement that cannot be loaded.

  • Adds a shared exception and loader for named-but-unreadable object permissions.
  • Applies the fail-closed behavior to organization, end-user, and agent server/tool ceilings while preserving existing handling for indeterminate lookups.
  • Adds focused tests covering key-authenticated and keyless callers across both fault classes.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename Overview
litellm/proxy/_experimental/mcp_server/auth/user_api_key_auth_mcp.py Separates named-but-unreadable entitlements from indeterminate lookups and propagates the former to the central deny paths.
tests/test_litellm/proxy/_experimental/mcp_server/auth/test_user_api_key_auth_mcp.py Adds regression coverage for server and tool authorization across organization, end-user, agent, key-authenticated, and keyless cases.

Reviews (2): Last reviewed commit: "fix(mcp): deny MCP access when a named e..." | Re-trigger Greptile

@codecov

codecov Bot commented Jul 30, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 91.46341% with 7 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...erimental/mcp_server/auth/user_api_key_auth_mcp.py 91.46% 7 Missing ⚠️

📢 Thoughts on this report? Let us know!

@yassin-berriai

Copy link
Copy Markdown
Contributor Author

The misc / Run tests red on this PR is base breakage, not this diff.

The sole failure is tests/test_litellm/interactions/test_openapi_compliance.py::TestRequestCompliance::test_content_schema_uses_discriminator, which asserts a discriminator on the interactions Content union. Nothing in this PR touches interactions.

Evidence it is base:

  • it reproduces locally on a clean tree branched from 551e5d097c with no interactions changes (1 failed, 12 passed in that file, no network or credentials involved)
  • the identical single failure is on four unrelated PRs of mine touching different subsystems, and on an open PR from a different author

Tracked as LIT-4975. Leaving it red rather than pushing a no-op to re-roll, since a re-run will fail the same way until the base is fixed.

@yassin-berriai

Copy link
Copy Markdown
Contributor Author

Root cause found for the misc / Run tests red on this PR, and it is not this diff. Fix is up as #35161.

tests/test_litellm/interactions/test_openapi_compliance.py::TestRequestCompliance::test_content_schema_uses_discriminator fetches Google's Interactions OpenAPI document over the network at run time and required an OpenAPI discriminator on the Content union. Google removed that keyword and now pins type with a const on each variant, so the assertion fails against the live spec on every open PR and will keep failing until the test is changed. Tracked as LIT-4975.

@codspeed-hq

codspeed-hq Bot commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 31 untouched benchmarks


Comparing litellm_mcp_entitlement_fault_semantics (e00bca4) with litellm_internal_staging (05c9815)1

Open in CodSpeed

Footnotes

  1. No successful run was found on litellm_internal_staging (3c2264c) during the generation of this report, so 05c9815 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report.

@yassin-berriai
yassin-berriai enabled auto-merge (squash) July 31, 2026 15:25
An MCP permission level answers which servers and tools it permits, and a
level that answers nothing places no restriction. Key auth was reading a
lookup FAULT as that same answer, so the end user, agent and org ceilings
quietly disappeared for as long as one lasted, while the keyless
gateway-admitted path failed closed on the very same fault.

Those levels now separate the two fault classes the user level already
did. A principal row that NAMES an object_permission_id whose contents
cannot be read is a known entitlement with unknown contents, so it denies.
A lookup that fails before we can tell whether the principal is entitled
at all still places no ceiling, that being the state which existed before
the level did; denying there would refuse MCP to the majority of callers,
who have no such entitlement configured. The keyless path is unchanged.

Resolves LIT-4960
@yassin-berriai
yassin-berriai force-pushed the litellm_mcp_entitlement_fault_semantics branch from 705af58 to e00bca4 Compare July 31, 2026 15:36
@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@yassin-berriai
yassin-berriai merged commit 473f43d into litellm_internal_staging Jul 31, 2026
78 of 79 checks passed
@yassin-berriai
yassin-berriai deleted the litellm_mcp_entitlement_fault_semantics branch July 31, 2026 15:49
@yassin-berriai

Copy link
Copy Markdown
Contributor Author

Rebased onto current staging and squashed to one commit; head is now e00bca4d6b. The conflicts were both additive: the Sequence/_as_list helper from the key/team grant predicate landed next to the new UnloadableEntitlementError, and the per-user entitlement tests from #35146 landed next to this branch's, so both sides are kept whole

One behavior-preserving change on top of the approved diff: the two deny paths now branch inside the resolvers' existing except blocks instead of adding except UnloadableEntitlementError clauses of their own. Same answers, same log lines, and it keeps the type-discipline budget flat, which the extra return [] sites would not have

Mapped directory is 2802 passing against the new base's 2792, with the same 23 pre-existing failures on both. misc stays red on the live-spec test tracked as LIT-4975, unrelated to this diff

@greptileai please review the current head e00bca4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants