Skip to content

Commit 67fc38e

Browse files
committed
pip: run 51 ledger
1 parent 22b00b2 commit 67fc38e

2 files changed

Lines changed: 106 additions & 4 deletions

File tree

‎entries/pip/20261029T021000Z.md‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
[agent] 2026-10-29 (02:10 UTC): pip / requirements.txt bug-hunt run
2+
3+
**Tested:** main `ecd4924` (11 commits since `b4b6d98`, none touching PyPI code), latest release v5.0.1. Linux only: real pip 26.2.2 / CPython 3.11 and pip 20.3.4 / CPython 3.8.20 (uv). Release binary built from main.
4+
5+
**Probe branches:** none (branch deletion is still blocked).
6+
7+
## Re-triage
8+
9+
- #1794: **still reproduces** (2/2 on `ecd4924`): `idna==3.7` + `six[foo]==1.16.0` + pylock, hosted then `vendor` → requirements.txt un-hosted to `six[foo]==1.16.0`, only pylock wired, exit 0, `vendor --check` exit 1. No new information, so no comment.
10+
- #977 / #638: not re-run (no related code changed).
11+
12+
## Cells (Linux)
13+
14+
| Cell | Result |
15+
| --- | --- |
16+
| #1623: two-package `pip lock` pylock (idna, six), `scan --mode vendored` then `vendor --revert` (pip 26.2.2) | **pass** (byte-identical; fixed by #1625) |
17+
| #1623: same, with comments above `[[packages.wheels]]` and inside a `[[packages]]` table | pass (byte-identical, comments kept) |
18+
| #1623: `remove pkg:pypi/idna@3.7` (first vendored) | pass (idna's wheels table restored in place; six stays vendored; `tomllib` parses) |
19+
| Fresh `pip install -r pylock.toml` of the two-package vendored pylock | pass (both patched) |
20+
| #1602: pylock vendored + requirements.txt `six===1.16.0` → `vex` | **pass** (`patched_ref_unattributable`, nothing attested) |
21+
| #1602: `==1.16.0.0`, `==v1.16.0`, `==1.16`, `SIX == 1.16.0` | pass (wired as siblings by the vendored run) |
22+
| #1602: `~=1.16.0`, `==1.16.*`, `>=1.16.0,<=1.16.0`, bare `six` | `not_affected`: documented in CLI_CONTRACT.md "Contested locks" (ranges don't contest). `>=X,<=X` still waits on the maintainer question in backlog 1m |
23+
| #1587: `./wheels/…`, `wheels/…`, `wheels//…`, `wheels/./…`, `wheels\…`, `file:wheels/…`, `six @ file:///…` | **pass** (all contest) |
24+
| `-e ./src/six#egg=six`, `six @ ./src/six` | contest (pass) |
25+
| `./src/six`, `-e ./src/six` (no name on the line) | don't contest: deliberate, see false positives |
26+
| Transitive-only six (requirements.txt has `python-dateutil`, pylock locks six), hosted then eject | pass (pylock wired, `installs driven by requirements.txt retain their existing sources` warning, both modes) |
27+
| Symlinked root requirements.txt + pylock, hosted | refused up front (`redirect_symlinked_file_unsupported`, nothing written): pass |
28+
| Symlinked `-r` include (target in / outside root), vendored | refused for that package, exit 1, other package wired: pass |
29+
| Two-package vendored → hosted takeover through a CRLF `-r` include (uppercase `IDNA`, marker, hash continuation) | kept vendored with `redirect_requirements_takeover_unreachable` (documented); fresh install patched; `vendor --revert` byte-identical, CRLF kept: pass |
30+
| pip 20.3.4 / py3.8: pip-compile CRLF hashed `idna` + `six`, hosted → `--require-hashes` install → `vendor` eject → install → `vendor --check` | pass (patched both times, check 0) |
31+
| Same, `vendor --revert` after the eject | restores upstream pins re-hashed from PyPI (documented pip-equivalent restore) |
32+
| Sole-package pylock (six only) + `six[foo]` requirements.txt, hosted → eject | refused: "no sibling registry package shows the registry and artifact fields" (#804 / #410 class) |
33+
| Lexer: `\` at the end of a line carrying a trailing `# comment` | matches pip (`COMMENT_RE.match` anchors at the line start, so only whole-line comments stop a continuation) |
34+
35+
## Issues
36+
37+
- Filed: none. Commented: none. Closed: none.
38+
39+
## False positives ruled out
40+
41+
- Name-less source directories (`./src/six`, `-e ./src/six`) not contesting a pylock wiring: `foreign_reference` in `vex/discover/pypi_other.rs` documents that a requirements line names its package only through a filename or `#egg=`, so source directories are left out on purpose. Recorded under Known non-bugs.
42+
- First `vex` attempts printed "No applied patches with vulnerability metadata": the mock's view had `vulnerabilities: {}`. Fixed in the harness below.
43+
- First #1794 re-run against a sole-package pylock hit the documented sole-package restore refusal, not #1794. With the two-package pylock from the issue, #1794 reproduces.
44+
45+
## Harness (rebuild in about 5 minutes)
46+
47+
Two patched wheels made from the real PyPI wheels: append `\nSAFE = True\n` to `six.py` (→ `<dir>/patched.whl`) and to `idna/__init__.py` (→ `<dir>/patched-idna.whl`), then recompute RECORD. `<dir>/dl/` holds the upstream wheels. Run `python3 -I mock2.py 8766 <dir>`. Env: `SOCKET_API_URL=SOCKET_PATCH_SERVER_URL=SOCKET_VENDOR_URL=http://127.0.0.1:8766 SOCKET_API_TOKEN=fake SOCKET_ORG_SLUG=o SOCKET_VENDOR_SOURCE=service`; leave the PyPI JSON API on real PyPI. Run `vex` with `--product pkg:pypi/app@1 -O vex.json` (`-o` is `--org`). Don't `pkill -f mock2.py` from the same shell.
48+
49+
<details><summary>mock2.py</summary>
50+
51+
```python
52+
import http.server,json,hashlib,base64,sys,os,zipfile,io
53+
PORT=int(sys.argv[1]); D=sys.argv[2]; B=f'http://127.0.0.1:{PORT}'
54+
P={'six':dict(ver='1.16.0',uuid='2b1f6c1e-8d3a-4f6b-9c2d-7e5a9b1c3d11',w='six-1.16.0-py2.py3-none-any.whl',m='six.py',pat='patched.whl'),
55+
'idna':dict(ver='3.7',uuid='4d5e6f70-1111-4222-8333-944455556666',w='idna-3.7-py3-none-any.whl',m='idna/__init__.py',pat='patched-idna.whl')}
56+
def gsha(b): return hashlib.sha256(b'blob %d\0'%len(b)+b).hexdigest()
57+
for n,p in P.items():
58+
p['orig']=open(f"{D}/dl/{p['w']}",'rb').read(); p['patb']=open(f"{D}/{p['pat']}",'rb').read()
59+
p['purl']=f"pkg:pypi/{n}@{p['ver']}"
60+
o=zipfile.ZipFile(io.BytesIO(p['orig'])).read(p['m']); q=zipfile.ZipFile(io.BytesIO(p['patb'])).read(p['m'])
61+
p['route']=f"/patch/pypi/{n}/{p['ver']}/33333333-3333-4333-8333-333333333333/{p['uuid']}/{p['w']}"
62+
p['view']={"uuid":p['uuid'],"purl":p['purl'],"publishedAt":"2026-01-01T00:00:00Z","files":{p['m']:{"beforeHash":gsha(o),"afterHash":gsha(q),"blobContent":base64.b64encode(q).decode()}},"vulnerabilities":{"GHSA-aaaa-bbbb-cccc":{"cves":["CVE-2099-0001"],"summary":"s","severity":"high","description":"d"}},"description":"x","license":"MIT","tier":"free"}
63+
BYU={p['uuid']:p for p in P.values()}
64+
class H(http.server.BaseHTTPRequestHandler):
65+
def log_message(self,*a): open(f'{D}/mock.log','a').write(self.command+' '+self.path+'\n')
66+
def send(self,code,body,ct='application/json'):
67+
if not isinstance(body,bytes): body=json.dumps(body).encode()
68+
self.send_response(code); self.send_header('Content-Type',ct); self.send_header('Content-Length',str(len(body))); self.end_headers(); self.wfile.write(body)
69+
def do_POST(self):
70+
n=int(self.headers.get('Content-Length',0)); body=self.rfile.read(n).decode('utf-8','replace'); p=self.path.split('?')[0]
71+
if p.endswith('/patches/batch'):
72+
return self.send(200,{"packages":[{"purl":q['purl'],"patches":[{"uuid":q['uuid'],"purl":q['purl'],"tier":"free","cveIds":[],"ghsaIds":[],"severity":"high","title":"t"}]} for q in P.values() if q['purl'] in body],"canAccessPaidPatches":False})
73+
if p.endswith('/patches/package'):
74+
res={}
75+
for u,q in BYU.items():
76+
if u in body or q['purl'] in body:
77+
url=B+q['route']
78+
res[u]={"status":"granted","url":url,"purl":q['purl'],"artifacts":[{"kind":"tarball","url":url,"integrity":{"sha256":hashlib.sha256(q['patb']).hexdigest(),"sha512":"sha512-"+base64.b64encode(hashlib.sha512(q['patb']).digest()).decode()}}],"registryOverride":None}
79+
return self.send(200,{"results":res})
80+
self.send(404,{})
81+
def do_GET(self):
82+
p=self.path.split('?')[0]
83+
if '/patches/by-package/' in p:
84+
return self.send(200,{"patches":[{"uuid":q['uuid'],"purl":q['purl'],"publishedAt":"2026-01-01T00:00:00Z","description":"x","license":"MIT","tier":"free","vulnerabilities":{"GHSA-aaaa-bbbb-cccc":{"cves":["CVE-2099-0001"],"summary":"s","severity":"high","description":"d"}}} for q in P.values() if q['purl'].replace('pkg:pypi/','') in p.replace('%40','@') or ('/'+q['purl'].split('/')[1].split('@')[0]) in p],"canAccessPaidPatches":False})
85+
for u,q in BYU.items():
86+
if p.endswith(f'/patches/view/{u}'): return self.send(200,q['view'])
87+
if p==q['route']: return self.send(200,q['patb'],'application/octet-stream')
88+
self.send(404,{})
89+
http.server.ThreadingHTTPServer(('127.0.0.1',PORT),H).serve_forever()
90+
```
91+
</details>
92+
93+
## Next
94+
95+
1. #1794: re-verify once fixed. Also an include outside the root beside a pylock, and a pin whose package already has a vendor line for another version.
96+
2. Re-verify #1104 and #819 with real pip (backlog 1p).
97+
3. Run-49 follow-ups 1q: orphan wiring in a `-c` constraints file / `requirements-dev.txt` sibling under `rollback`; a dotted-name pair (extend mock2 with `zope.event` or `ruamel.yaml`).
98+
4. pip 24–25 hosted / vendored spot-checks (backlog 5).
99+
5. A Windows / macOS probe once branch deletion works.

‎state/pip.md‎

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
[agent] Progress ledger for the scheduled pip / requirements.txt bug-hunt routine (label pm:pip).
22

3-
Last run: 2026-10-28 ~20:30 UTC (fiftieth run), main `b4b6d98`, latest release v5.0.1. #1758 verified fixed by #1759 (real pip 26.2.2). Filed #1794: an extras pin (`six[foo]==X`) in requirements.txt beside a `pip lock` pylock is still un-hosted and left unpatched by the hosted → vendored eject / takeover (exit 0). Harness: the Python mock from run 49, rebuilt and inlined in entry 20261028T203000Z (unpatched wheels come from real PyPI so pylock records PyPI URLs; a 127.0.0.1 index makes the pylock restore refuse, which is correct). Probe branches are still blocked. Open: #1794, #977, #638 (+ #836 enhancement).
3+
Last run: 2026-10-29 ~02:10 UTC (fifty-first run), main `ecd4924`, latest release v5.0.1 (no PyPI code changed since `b4b6d98`). Re-verified the closed fixes #1623, #1602 and #1587 with real pip 26.2.2: all pass. #1794 still reproduces (2/2). No new issues. Harness: the two-package (six + idna) Python mock with vuln metadata inlined in entry 20261029T021000Z. Probe branches are still blocked. Open: #1794, #977, #638 (+ #836 enhancement).
44
## Coverage matrix
55

66
| OS | pip / Python | agent (.venv) | agent egg-info | hosted (unhashed → fragment pin) | vendored | rollback / remove | vendored → hosted takeover | system-site venv | `-r` include, lock-only | PEP 440-equivalent pin (`==1.16`) | global `-g` (`--user`) | lock-only spaced pin (`six == X`) | hosted foreign direct ref (`six @ mirror`) | lock-only PEP 440-equivalent pin (`==1.16`) | no root requirements.txt (`--json`) | hosted → vendored takeover (`vendor`) | `vendor --dry-run` over hosted | `get --mode vendored` over hosted | vendored → hosted, include / transitive pin | UTF-16 requirements.txt (hosted / lock-only) | root file used as `-c` by a sibling | vendored re-vendor to a new uuid | vendored pin removed / bumped (`--prune`) | `pip lock` pylock.toml (hosted / vendored) | agent same-size patch vs `__pycache__` | agent rollback of a patch-added file in a new dir | socket.yml package specs (PyPI spelling) | hosted superseding re-pin (A → B) | `--max-new-patches` (hosted) | `get <name>` PyPI spelling | `--max-new-patches` (vendored) | vendored supersede under cap 0 (hashed include / CRLF marker) | vendored line comment edited, then revert | quoted / `${VAR}` `-r` include (lock-only) | option before `-r` on one line (lock-only) | hosted root pin + duplicate pin in `-r` include (vex / rollback) | PEP 263 coding-line (latin-1) requirements.txt, lock-only | `apply --check` (agent, #1029) | vendored `scan --prune` liveness (#1050) | hosted / vendored file regenerated by `pip freeze` | hashed hosted root + hashed `-r` include (`--require-hashes`) | dangling `\` continuation at EOF (lock-only) | lock-only bare-CR / FF / NEL / U+2028 line breaks | `list` (hosted / marker split / include) | requirements spelling contests Pipfile.lock wiring (vex) |
@@ -58,13 +58,13 @@ Run 50 (Linux, pip 26.2.2 / CPython 3.11, lock-only + pylock): hosted → vendor
5858
## Backlog
5959

6060
1q. Run 49 follow-ups: reuse the Python mock (entry 20261028T134832Z) for a two-package vendored → hosted takeover with dotted names in a CRLF `-r` include, and an orphan wire in a `-c` constraints file / `requirements-dev.txt` sibling under `rollback` (unscoped refusal).
61-
1r. #1794: re-verify once fixed (eject and `scan --mode vendored`; also `--dry-run` previews). Check other `sibling_pin_wirable`-rejected shapes beside a pylock: a symlinked requirements.txt, an include outside the root, a pin whose package also has a vendor line for another version.
61+
1r. #1794: re-verify once fixed (eject and `scan --mode vendored`; also `--dry-run` previews); still reproduces on `ecd4924`. Other `sibling_pin_wirable`-rejected shapes: a symlinked root requirements.txt is refused up front by hosted (`redirect_symlinked_file_unsupported`, nothing written), so the eject can't reach it (run 51); a transitive-only package (not in requirements.txt) is left with the documented `installs driven by requirements.txt retain their existing sources` warning on both hosted and eject (pass). Still open: an include outside the root, a pin whose package also has a vendor line for another version.
6262
1o. done 2026-10-28 run 50: #1758 verified fixed; the `-r` include and `pylock.<name>.toml` cells pass. (The two-package eject refusal / rollback over requirements.txt alone passed on run 48; a failure that only surfaces after the restore, i.e. a true `eject_rolled_back`, wasn't reachable with the fixtures: a corrupt artifact is refused at the grant step.)
63-
1p. Re-verify the maintainer-closed fixes with real pip (closed 2026-10-22 to 10-27): #1623, #1602, #1587, #1104, #819. (#740 verified on the eject path on run 48; #1567's mixed-EOL case passes on the eject path; the plain vendored planner over a mixed file is still to re-check.)
63+
1p. Re-verify the maintainer-closed fixes with real pip (closed 2026-10-22 to 10-27): #1104, #819. (Done 2026-10-29 run 51: #1623 two-package `pip lock` revert byte-identical, comments above restored keys kept, `remove` of the first-vendored package clean; #1602 `===` contests, `==X.0` / `==vX` / `==1.16` / `SIX == X` are wired as siblings; #1587 `wheels/…`, `wheels//…`, `wheels/./…`, `wheels\…`, `file:wheels/…` all contest.) (#740 verified on the eject path on run 48; #1567's mixed-EOL case passes on the eject path; the plain vendored planner over a mixed file is still to re-check.)
6464

6565
1n. #1623 follow-ups: re-verify once fixed (`vendor --revert`, `remove` of the first-vendored package, three packages, CRLF pip-lock file). Also check that comments above restored keys survive (the Poetry handover 20261016T202635Z-from-poetry; root cause noted in #1623), and a hosted → vendored takeover over a two-package `pip lock` file.
6666
1m. #1602 follow-ups: re-verify once fixed. Also check the reverse direction (a requirements.txt wiring vs a hand-edited Pipfile.lock `"version": "===X"` / `"==X.0"`), and get a maintainer call on single-version ranges (`~=X`, `==X.*`, `>=X,<=X`).
67-
1l. #1587 follow-ups: a name-less editable `-e ./src/six` (no `#egg=`; pip reads the name from the project metadata) contests nothing. Check whether that's in scope after #1587 is fixed, along with a Windows `wheels\six-…whl` line. Re-verify #1587 once fixed.
67+
1l. done 2026-10-29: #1587 verified; the Windows `wheels\six-…whl` line contests. The name-less source directory is a deliberate exclusion (see Known non-bugs).
6868
1k. #1567 follow-ups: a mixed CRLF/LF `-r` include, the superseding re-vendor (`plan_rewire`), and the hosted → vendored takeover over a mixed file (hosted keeps per-line endings, so it should be clean). Re-verify once fixed.
6969
1j. #1281 was closed not planned (2026-10-09). Hosted and vendored over CR-only roots were checked on run 37 and are fail-closed or install patched; nothing left to do unless the maintainer reopens it.
7070

@@ -89,6 +89,9 @@ Run 50 (Linux, pip 26.2.2 / CPython 3.11, lock-only + pylock): hosted → vendor
8989

9090
## Known non-bugs
9191

92+
- A name-less local source directory in requirements.txt (`./src/six`, `-e ./src/six` without `#egg=`) doesn't contest another lock's wiring of the same `name@version`, while `six @ ./src/six` and `-e ./src/six#egg=six` do. Deliberate: `vex/discover/pypi_other.rs` `foreign_reference` says a requirements line names its package only through its filename or `#egg=`, "so one with neither (a source directory) is left out" (run 51).
93+
- A symlinked `-r` include (target inside or outside the root) makes vendored refuse that package with `redirect_symlinked_file_unsupported`, exit 1; the other packages are still wired. Fail-closed and loud (run 51).
94+
9295
- Hosted scan with the pin only in a `-r` include beside a `pip lock` pylock skips with `redirect_unattributable` (pylock wired vs include unpatched) and writes nothing, exit 0. This is the same class as "hosted rewrites only the root requirements.txt"; it's fail-closed (2026-10-28, run 50).
9396
- A pylock whose packages carry files from a non-PyPI index and record no `index` makes the hosted → vendored restore refuse (`redirect_revert_failed`, "restore it from version control"). That's correct fail-closed behaviour, and also a harness trap (2026-10-28).
9497

0 commit comments

Comments
 (0)