|
| 1 | +[agent] 2026-10-29 (02:10 UTC): pip / requirements.txt bug-hunt run |
| 2 | + |
| 3 | +**Tested:** main `ecd4924` (11 commits since `b4b6d98`, none touching PyPI code), latest release v5.0.1. Linux only: real pip 26.2.2 / CPython 3.11 and pip 20.3.4 / CPython 3.8.20 (uv). Release binary built from main. |
| 4 | + |
| 5 | +**Probe branches:** none (branch deletion is still blocked). |
| 6 | + |
| 7 | +## Re-triage |
| 8 | + |
| 9 | +- #1794: **still reproduces** (2/2 on `ecd4924`): `idna==3.7` + `six[foo]==1.16.0` + pylock, hosted then `vendor` → requirements.txt un-hosted to `six[foo]==1.16.0`, only pylock wired, exit 0, `vendor --check` exit 1. No new information, so no comment. |
| 10 | +- #977 / #638: not re-run (no related code changed). |
| 11 | + |
| 12 | +## Cells (Linux) |
| 13 | + |
| 14 | +| Cell | Result | |
| 15 | +| --- | --- | |
| 16 | +| #1623: two-package `pip lock` pylock (idna, six), `scan --mode vendored` then `vendor --revert` (pip 26.2.2) | **pass** (byte-identical; fixed by #1625) | |
| 17 | +| #1623: same, with comments above `[[packages.wheels]]` and inside a `[[packages]]` table | pass (byte-identical, comments kept) | |
| 18 | +| #1623: `remove pkg:pypi/idna@3.7` (first vendored) | pass (idna's wheels table restored in place; six stays vendored; `tomllib` parses) | |
| 19 | +| Fresh `pip install -r pylock.toml` of the two-package vendored pylock | pass (both patched) | |
| 20 | +| #1602: pylock vendored + requirements.txt `six===1.16.0` → `vex` | **pass** (`patched_ref_unattributable`, nothing attested) | |
| 21 | +| #1602: `==1.16.0.0`, `==v1.16.0`, `==1.16`, `SIX == 1.16.0` | pass (wired as siblings by the vendored run) | |
| 22 | +| #1602: `~=1.16.0`, `==1.16.*`, `>=1.16.0,<=1.16.0`, bare `six` | `not_affected`: documented in CLI_CONTRACT.md "Contested locks" (ranges don't contest). `>=X,<=X` still waits on the maintainer question in backlog 1m | |
| 23 | +| #1587: `./wheels/…`, `wheels/…`, `wheels//…`, `wheels/./…`, `wheels\…`, `file:wheels/…`, `six @ file:///…` | **pass** (all contest) | |
| 24 | +| `-e ./src/six#egg=six`, `six @ ./src/six` | contest (pass) | |
| 25 | +| `./src/six`, `-e ./src/six` (no name on the line) | don't contest: deliberate, see false positives | |
| 26 | +| Transitive-only six (requirements.txt has `python-dateutil`, pylock locks six), hosted then eject | pass (pylock wired, `installs driven by requirements.txt retain their existing sources` warning, both modes) | |
| 27 | +| Symlinked root requirements.txt + pylock, hosted | refused up front (`redirect_symlinked_file_unsupported`, nothing written): pass | |
| 28 | +| Symlinked `-r` include (target in / outside root), vendored | refused for that package, exit 1, other package wired: pass | |
| 29 | +| Two-package vendored → hosted takeover through a CRLF `-r` include (uppercase `IDNA`, marker, hash continuation) | kept vendored with `redirect_requirements_takeover_unreachable` (documented); fresh install patched; `vendor --revert` byte-identical, CRLF kept: pass | |
| 30 | +| pip 20.3.4 / py3.8: pip-compile CRLF hashed `idna` + `six`, hosted → `--require-hashes` install → `vendor` eject → install → `vendor --check` | pass (patched both times, check 0) | |
| 31 | +| Same, `vendor --revert` after the eject | restores upstream pins re-hashed from PyPI (documented pip-equivalent restore) | |
| 32 | +| Sole-package pylock (six only) + `six[foo]` requirements.txt, hosted → eject | refused: "no sibling registry package shows the registry and artifact fields" (#804 / #410 class) | |
| 33 | +| Lexer: `\` at the end of a line carrying a trailing `# comment` | matches pip (`COMMENT_RE.match` anchors at the line start, so only whole-line comments stop a continuation) | |
| 34 | + |
| 35 | +## Issues |
| 36 | + |
| 37 | +- Filed: none. Commented: none. Closed: none. |
| 38 | + |
| 39 | +## False positives ruled out |
| 40 | + |
| 41 | +- Name-less source directories (`./src/six`, `-e ./src/six`) not contesting a pylock wiring: `foreign_reference` in `vex/discover/pypi_other.rs` documents that a requirements line names its package only through a filename or `#egg=`, so source directories are left out on purpose. Recorded under Known non-bugs. |
| 42 | +- First `vex` attempts printed "No applied patches with vulnerability metadata": the mock's view had `vulnerabilities: {}`. Fixed in the harness below. |
| 43 | +- First #1794 re-run against a sole-package pylock hit the documented sole-package restore refusal, not #1794. With the two-package pylock from the issue, #1794 reproduces. |
| 44 | + |
| 45 | +## Harness (rebuild in about 5 minutes) |
| 46 | + |
| 47 | +Two patched wheels made from the real PyPI wheels: append `\nSAFE = True\n` to `six.py` (→ `<dir>/patched.whl`) and to `idna/__init__.py` (→ `<dir>/patched-idna.whl`), then recompute RECORD. `<dir>/dl/` holds the upstream wheels. Run `python3 -I mock2.py 8766 <dir>`. Env: `SOCKET_API_URL=SOCKET_PATCH_SERVER_URL=SOCKET_VENDOR_URL=http://127.0.0.1:8766 SOCKET_API_TOKEN=fake SOCKET_ORG_SLUG=o SOCKET_VENDOR_SOURCE=service`; leave the PyPI JSON API on real PyPI. Run `vex` with `--product pkg:pypi/app@1 -O vex.json` (`-o` is `--org`). Don't `pkill -f mock2.py` from the same shell. |
| 48 | + |
| 49 | +<details><summary>mock2.py</summary> |
| 50 | + |
| 51 | +```python |
| 52 | +import http.server,json,hashlib,base64,sys,os,zipfile,io |
| 53 | +PORT=int(sys.argv[1]); D=sys.argv[2]; B=f'http://127.0.0.1:{PORT}' |
| 54 | +P={'six':dict(ver='1.16.0',uuid='2b1f6c1e-8d3a-4f6b-9c2d-7e5a9b1c3d11',w='six-1.16.0-py2.py3-none-any.whl',m='six.py',pat='patched.whl'), |
| 55 | + 'idna':dict(ver='3.7',uuid='4d5e6f70-1111-4222-8333-944455556666',w='idna-3.7-py3-none-any.whl',m='idna/__init__.py',pat='patched-idna.whl')} |
| 56 | +def gsha(b): return hashlib.sha256(b'blob %d\0'%len(b)+b).hexdigest() |
| 57 | +for n,p in P.items(): |
| 58 | + p['orig']=open(f"{D}/dl/{p['w']}",'rb').read(); p['patb']=open(f"{D}/{p['pat']}",'rb').read() |
| 59 | + p['purl']=f"pkg:pypi/{n}@{p['ver']}" |
| 60 | + o=zipfile.ZipFile(io.BytesIO(p['orig'])).read(p['m']); q=zipfile.ZipFile(io.BytesIO(p['patb'])).read(p['m']) |
| 61 | + p['route']=f"/patch/pypi/{n}/{p['ver']}/33333333-3333-4333-8333-333333333333/{p['uuid']}/{p['w']}" |
| 62 | + p['view']={"uuid":p['uuid'],"purl":p['purl'],"publishedAt":"2026-01-01T00:00:00Z","files":{p['m']:{"beforeHash":gsha(o),"afterHash":gsha(q),"blobContent":base64.b64encode(q).decode()}},"vulnerabilities":{"GHSA-aaaa-bbbb-cccc":{"cves":["CVE-2099-0001"],"summary":"s","severity":"high","description":"d"}},"description":"x","license":"MIT","tier":"free"} |
| 63 | +BYU={p['uuid']:p for p in P.values()} |
| 64 | +class H(http.server.BaseHTTPRequestHandler): |
| 65 | + def log_message(self,*a): open(f'{D}/mock.log','a').write(self.command+' '+self.path+'\n') |
| 66 | + def send(self,code,body,ct='application/json'): |
| 67 | + if not isinstance(body,bytes): body=json.dumps(body).encode() |
| 68 | + self.send_response(code); self.send_header('Content-Type',ct); self.send_header('Content-Length',str(len(body))); self.end_headers(); self.wfile.write(body) |
| 69 | + def do_POST(self): |
| 70 | + n=int(self.headers.get('Content-Length',0)); body=self.rfile.read(n).decode('utf-8','replace'); p=self.path.split('?')[0] |
| 71 | + if p.endswith('/patches/batch'): |
| 72 | + return self.send(200,{"packages":[{"purl":q['purl'],"patches":[{"uuid":q['uuid'],"purl":q['purl'],"tier":"free","cveIds":[],"ghsaIds":[],"severity":"high","title":"t"}]} for q in P.values() if q['purl'] in body],"canAccessPaidPatches":False}) |
| 73 | + if p.endswith('/patches/package'): |
| 74 | + res={} |
| 75 | + for u,q in BYU.items(): |
| 76 | + if u in body or q['purl'] in body: |
| 77 | + url=B+q['route'] |
| 78 | + res[u]={"status":"granted","url":url,"purl":q['purl'],"artifacts":[{"kind":"tarball","url":url,"integrity":{"sha256":hashlib.sha256(q['patb']).hexdigest(),"sha512":"sha512-"+base64.b64encode(hashlib.sha512(q['patb']).digest()).decode()}}],"registryOverride":None} |
| 79 | + return self.send(200,{"results":res}) |
| 80 | + self.send(404,{}) |
| 81 | + def do_GET(self): |
| 82 | + p=self.path.split('?')[0] |
| 83 | + if '/patches/by-package/' in p: |
| 84 | + return self.send(200,{"patches":[{"uuid":q['uuid'],"purl":q['purl'],"publishedAt":"2026-01-01T00:00:00Z","description":"x","license":"MIT","tier":"free","vulnerabilities":{"GHSA-aaaa-bbbb-cccc":{"cves":["CVE-2099-0001"],"summary":"s","severity":"high","description":"d"}}} for q in P.values() if q['purl'].replace('pkg:pypi/','') in p.replace('%40','@') or ('/'+q['purl'].split('/')[1].split('@')[0]) in p],"canAccessPaidPatches":False}) |
| 85 | + for u,q in BYU.items(): |
| 86 | + if p.endswith(f'/patches/view/{u}'): return self.send(200,q['view']) |
| 87 | + if p==q['route']: return self.send(200,q['patb'],'application/octet-stream') |
| 88 | + self.send(404,{}) |
| 89 | +http.server.ThreadingHTTPServer(('127.0.0.1',PORT),H).serve_forever() |
| 90 | +``` |
| 91 | +</details> |
| 92 | + |
| 93 | +## Next |
| 94 | + |
| 95 | +1. #1794: re-verify once fixed. Also an include outside the root beside a pylock, and a pin whose package already has a vendor line for another version. |
| 96 | +2. Re-verify #1104 and #819 with real pip (backlog 1p). |
| 97 | +3. Run-49 follow-ups 1q: orphan wiring in a `-c` constraints file / `requirements-dev.txt` sibling under `rollback`; a dotted-name pair (extend mock2 with `zope.event` or `ruamel.yaml`). |
| 98 | +4. pip 24–25 hosted / vendored spot-checks (backlog 5). |
| 99 | +5. A Windows / macOS probe once branch deletion works. |
0 commit comments