Repository navigation
Conversation
litellm 1.106.0-1.108.3 import mcp on every completion(tools=...), so each skill-agent arm answer from a locked install failed with No module named 'mcp' (BerriAI/litellm#46045). Raise the floor to the release with the fix and lock 1.109.1, the first one conda-forge also ships. Signed-off-by: Alexander Rybkin <alexander.rybkin@intel.com>
xaleryb
force-pushed
the
fix/litellm-1.109.1
branch
from
November 17, 2026 23:22
c56109d to
961ef71
Compare
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #897 +/- ##
=======================================
Coverage 95.79% 95.79%
=======================================
Files 432 432
Lines 57204 57204
Branches 9858 9858
=======================================
Hits 54796 54796
Misses 1497 1497
Partials 911 911 🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft until 2026-11-19: the locks pin a litellm release younger than their 7-day cooldown. See "Quarantine" below.
Summary
From a locked install, every
skill-agent:arm answer fails withagent_failed: No module named 'mcp'. The run still prints[1/1] Q1 OKandarms runexits 0, so the batch looks healthy and the arm scores as if the agent had answered nothing.Cause: litellm 1.106.0 through 1.108.3 import
mcp.typeson anycompletion(tools=...).main.pytakes theif not skip_mcp_handler and tools:branch, which importsresponses/mcp/litellm_proxy_mcp_handler, which importsproxy/_experimental/mcp_server/catalog.py, which doesfrom mcp.types import CacheableResult.mcpis only in litellm'sproxy/mcpextras, so a base install does not have it. It is fixed upstream in 1.108.4 and later (BerriAI/litellm#45961, backported in BerriAI/litellm#50605, merged 2026-11-09; reports: BerriAI/litellm#46045, BerriAI/litellm#50781). Our lock moved onto 1.106.0 on 2026-10-26 (#224). Since then, the agent arms have not worked fromrequirements.lock.txtorci/conda-lock/.The fix raises the floor to the first fixed release and locks 1.109.1. Rejected alternatives:
mcpas a runtime dependency: it brings 13 packages (starlette, uvicorn, sse-starlette, python-multipart among them) into the base install, which goes against fix(deps): take ragas, langchain and diskcache out of the CI/dev lock #245 (ragas moved out of the base install to reduce the CVE surface)._skip_mcp_handler: a private argument that can change in any release.litellm<1.106: this stays on an old release and blocks the upstream fix.Changes
litellm>=1.108.4(was>=1.105.0) inrequirements.txt,requirements-test.txt,pyproject.tomlandconda-recipe/meta.yaml. No other file lists the spec.requirements.lock.txt:litellm==1.109.1(was 1.106.6). The only other change is thepython_full_version < '3.15'marker that 1.109.1 adds to itsfastuuidrequirement, plus a# via litellmline. Every other pin is the same as on main.ci/conda-lock/linux-64-py3.{11,12,13,14}.txt: litellm1.106.0becomes conda-forge1.109.1.async-timeoutdrops out because 1.109.1 no longer requires it on Python 3.11 and later. The# specs:header carries the new floor. Every other record is the same build as on main.tests/test_agent_arm_dependencies.py: runs askill_agent:arm through the reallitellm.completionwithmock_response. This sends nothing over the network, but it imports what a call with tools imports. The test fails on main's lock and passes with this one.Fixed).Quarantine
Why 1.109.1 and not 1.108.4: conda-forge skipped 1.108.x (1.106.0 on 2026-10-21, then 1.109.0 on 2026-11-11 and 1.109.1 on 2026-11-12). 1.109.1 is the oldest fixed release that both PyPI and conda-forge ship, so the pip lock and the conda locks pin the same version.
Why a draft: both locks admit only releases at least 7 days old (
--exclude-newer=P7Din the uv header,--min-age-days 7inci/lock.py). The 1.109.1 wheels reached PyPI on 2026-11-12 02:00 UTC and the conda-forge builds on 2026-11-12 10:47 UTC, so both clear the cooldown on 2026-11-19 at 10:47 UTC (02:47 PST). Before that, this PR relaxes the cooldown for litellm only:requirements.lock.txt: the header's command plus--upgrade-package litellm --exclude-newer-package litellm=false. The header line keeps the canonical command.ci/conda-lock/:ci/lock.py --min-age-days 5, with every record except litellm held to main's build so that only litellm and what it requires could move. The header keeps# min-age-days: 7.So both headers already describe the state they will have on 2026-11-19. If you run the canonical uv command today, it does not keep 1.109.1: it falls back to 1.108.5, which is fixed but is not on conda-forge.
On 2026-11-19, after 10:47 UTC:
requirements.lock.txtwith the exact command in its header. It must come out identical. A run of that command withP7Dreplaced by its 2026-11-19 03:00 UTC cutoff already does.python ci/lock.py --if-stale. It must report all four conda locks current. Then re-run the litellm-only solve at--min-age-days 7and expect the same four files byte for byte.Risk
This changes litellm only, from 1.106.x to 1.109.1, for every LLM call Dokimi makes. The full suite passes on the new lock, and an agent arm answered end to end on Bedrock.
pip install dokimiusers with an older litellm get the new floor on their next upgrade. The changelog entry tells them to re-run affectedarms runbatches.1.109.1 declares
filelock<5, so thefilelock>=4.2.1override and the--no-depsinstalls that SECURITY.md describes are no longer needed (pip checkis clean on the new lock). This PR leaves both alone. They are follow-up 1 below.Testing
On Python 3.13.18 (Linux x86_64), in a venv built from the new
requirements.lock.txtwithpip install --no-deps --require-hashes -r requirements.lock.txt -r requirements-ci.lock.txt:pytest -n auto --outcomes ...(the CI command): 22842 passed, 38 skipped, rc 0.tests/test_agent_arm_dependencies.py: fails in a venv from main's lock (litellm 1.106.6) withModuleNotFoundError: No module named 'mcp'fromlitellm/proxy/_experimental/mcp_server/catalog.py. Passes in the new venv, and passes in a conda env created from the newlinux-64-py3.13.txt.ruff check .,mypy(438 files),scripts/check_growth.py --report,scripts/check_comment_blocks.py,scripts/check_changelog_fragment.py,scripts/check_test_narrowing.py,scripts/check_duplicates.py,python -m dokimi.schema_compat --base origin/main,deptry .,scripts/check_docs_review_triggers.py: all clean.pip-audit.ymlruns it over all six locks: no known vulnerabilities. osv-scanner 2.7.1 withosv-scanner.tomlonrequirements.lock.txt: no issues.micromamba create --filesucceeds for all fourci/conda-lockfiles, and each env imports litellm 1.109.1.cli.py arms run --product demo --questions q.json --arms skill-agent:<demo skill> --model us.anthropic.claude-haiku-5-5 --provider amazon-bedrock --max-iterations 2. On main the answer isagent_failed: No module named 'mcp'withQ1 OKand rc 0. On this branch the answer is "The demo widget uses port 4242." after 2 model calls, with noagent_failed.Open questions
python ci/lock.pyre-solve would also move about 15 unrelated packages per file (boto3, datasets, filelock 3.32.6 to 4.2.9, regex, ...). I left that to Monday'sconda-lock.ymlrefresh. Say so if you would rather this PR carry the full re-solve.Revert
Revert the commit. The floors and both locks return to litellm 1.105.0/1.106.x, and the agent arms fail again as described above. No stored records or schema versions are involved.
Follow-ups (separate PRs)
requirements-overrides.txt(filelock>=4.2.1), the--overrideflag in both lock headers and the--no-depsinstalls, and update the filelock paragraph of SECURITY.md: litellm 1.109.1 allows filelock 4.arms runreportsOKand exits 0 when an arm answer isagent_failed. That is why this breakage went unnoticed for three weeks.Checklist
pytestpassesruff,mypy, and schema checks passAGENTS.mdraise/exit and test names the doc, contract or issue it traces to ([Bug]: "ModuleNotFoundError: No module named 'mcp'" BerriAI/litellm#46045).github/.licenserc.yaml)