Repository navigation
Conversation
Merging this PR will not alter performance
Comparing Footnotes
|
There was a problem hiding this comment.
All reported issues were addressed
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
There was a problem hiding this comment.
All reported issues were addressed across 5 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Re-trigger cubic
Read `anyio`, `trio` and `aiofiles` files with `await read()` rather than iterating a line at a time, and set `Content-Length` when the length is known. Multipart uploads from an async file previously looped indefinitely over un-awaited coroutines; they now stream, or raise `TypeError` on a sync `Client`.
Fall back to chunked encoding when an async file reports it can't be rewound, so `Content-Length` can't disagree with the bytes actually sent. Reject text-mode async files passed as `content=`, matching the multipart path. Benchmark async uploads through a descriptor so the length path is covered, and reuse one event loop per benchmark rather than paying loop bootstrap on every sample.
The truthiness guard let an empty `str` read past the binary-mode check, so an empty text-mode file uploaded silently as zero bytes instead of raising.
386c7a7 to
3704e0d
Compare
|
Rebased this on current @Kludex can you have a look at this one when you get a chance? |
Summary
This is my third attempt at this. This PR resolves #597 and implements chunked upload for
anyio.open_file,trio.open_fileandaiofiles.openwhen used ascontent=parameter forpostandputrequest and implements support for multipart file upload for the same libraries. Previous attempts were encode/httpx#3339 and encode/httpx#3698 againstencode/httpx, both went stale while the library was unmaintained. Most of the code is reworked from encode/httpx#3698, with the review comments left on encode/httpx#3339 applied.Currently
content=with an async file iterates it a line at a time instead of reading in chunks, so a file without newlines gets buffered whole, andfiles=with an async file loops forever on both clients, sinceMultipartStream.__aiter__just delegates to the synciter_chunks()andread()returns a coroutine, which is always truthy.changes
_types.AsyncReadableFileprotocol was added, and_types.FileContentwas extended to include it_utils.is_async_readable_filetype predicate function was added to detect and perform type narrowing for trio/anyio/aiofiles async files_utils.aiter_file_chunkswas added to read an async file in chunks from its current position, shared bycontent=and multipart_utils.peek_async_filelike_lengthwas added, async wrappers expose a syncfileno()so the fd can be stat'ed, there's notell()/seek()fallback like the sync version because those would need awaiting, and it only returns a size for regular files so pipes fall back to chunked encoding_content.AsyncIteratorByteStreamupdated to useawait read()for async files instead of looping over linescontent=with an async file is sent withTransfer-Encoding: chunkedfrom the file's current position, matching how sync files respect the cursor since Respect file cursor positions when calculating raw content length #1214_multipart.FileFieldupdated witharender_data, which rewinds the file like the sync path, andget_length/render_datamade async-aware, syncClientnow raisesTypeErrorinstead of looping_multipart.MultipartStreamupdated withaiter_chunks, and__aiter__updated to use it, both wrapped incontextlib.aclosingtest_content.py,test_multipart.pyandtest_utils.py, including a realtrio.open_file()upload and a multipart upload resent on a 307 redirect, docs and changelog updatedperf
content=over 4 MiB with 64-byte lines: 9.18 ms -> 0.33 ms (encoding path, in-memory file, so no disk I/O in the number)content=stream, never per chunkChecklist