Repository navigation
prelude/skstore: avoid temporary collections (Sequence::toArray, direct getArray, more mapToArray sites) - #1715
Merged
Merged
Conversation
Adds Sequence::toArray, returning an Array with the sequence's items. IndexedSequence overrides it with a single Array::fillBy pass, Bytes fills directly from its native buffer, and Array copies via fillBy (the result must be frozen, so aliasing `this` is not sound). Vector, List, Queue, Set and SortedSet already had toArray and now override the base method. Base default: collect(Array). std bumped to 0.7.0 (minor: new prelude API). Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
String::join now goes through Sequence::toArray (dispatching to the direct implementations), and the System poll handler converts the read buffer slice via Bytes::toArray, resolving the last TODO about .collect(Array). Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
IFixedDir::getArray/getArraySourceKey built their result via getAll(...).map(...).collect(Array) (iterator -> Vector -> Array). They now use findAllRange (findFirstBy plus a linear count, shared with findAllBy) and a single Array::fillBy over the matching range, avoiding the intermediate Vector. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Use mapToArray where a mapped Vector was immediately copied to an Array (RangeMap.fromRangeMap, JSON object fields, Doc.inspect), and toArray instead of collect(Array) on Vectors (IO.read_to_string/read_line, Unsafe hex formatting), resolving the existing TODOs -- collect(Array) on a Vector goes iterator -> Vector -> Array. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
Sequence::join previously did values().collect(Vector).join(separator), which routed through Vector::join's toArray(). It now renders each element with toString exactly once, collecting the strings (with separators interleaved) into a single Vector that is then flattened into the output buffer. The Vector::join and List::join overrides, which built intermediate collections just to call Array::join, are removed in favor of the base implementation. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
String::join(pieces) now calls pieces.join(this), dispatching to the direct implementations instead of materializing pieces.toArray() first. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
The poll handler evaluated buf.slice(0, nbytes) twice per read (once for captured.extend, once for the handler string) and then copied the slice again with toArray(); the slice is already an Array, so bind it once and pass it to both. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
The findAllBy docstring ended up stacked above findAllRange and findAllBy itself had none; findAllRange's contract was also imprecise about the no-match case. Document both properly: findAllRange returns (i, n) with matches exactly i..i+n-1 (n == 0 when none), findAllBy enumerates them. Generated by Mistral Vibe. Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
mbouaziz
force-pushed
the
feat/avoid-temporary-collections
branch
from
November 19, 2026 14:21
8b3143c to
3881578
Compare
mbouaziz
added a commit
that referenced
this pull request
Nov 19, 2026
## Summary Optimizes `Sequence::join` (merged in #1715) along the lines suggested in review: 1. **Count during the single pass.** The total output size is now accumulated in the same `each` pass that renders the elements (`toString` still called exactly once per element — there is a single call site), instead of a second pass over the collected strings calling `String::length()`. The separator's length is computed once (`separator.bytes().size()`) and multiplied by `count - 1`. 2. **Byte-level join.** Strings are natively UTF-8 (`skiplang/prelude/runtime/string.c`: `sk_string_t` is a raw byte buffer). The old fill loop iterated *chars* via `String::each`, which goes through `StringIterator::nextCode` — a branchy per-char UTF-8 decode plus `?Char` option boxing per char (`src/native/StringIterator.sk`). The new implementation collects zero-copy `Bytes` views (`String::bytes()`), copies raw bytes with the native `Bytes::get` into an `Array<UInt8>`, and builds the result with `String::fromUtf8`, which is a single native `memcpy` (`sk_string_create`) with no validation. Concatenating valid UTF-8 strings at the byte level is exact, including multibyte chars. ## On slice-copy functions vs iteration Investigated while doing this: - **Array → Array:** element-wise loops are already native — `Array::unsafe_get`/`unsafe_set` are `native` direct accesses, so a simple copy loop compiles to a tight load/store loop that LLVM auto-vectorizes to memcpy speed (the existing private `Array::unsafeMoveSlice` is exactly this pattern). A dedicated native memmove wrapper would save little beyond what `unsafe_*` already skip (bounds checks). - **String → buffer:** char iteration is *not* memcpy-like (UTF-8 decode + `?Char` boxing per char, see above), which is why the byte-level copy matters here. There is currently no `memcpy` exposed in the prelude `Unsafe` module, so a native `Bytes`/`Array` slice-copy would be compiler/native work — not needed for this change since the per-byte loop over native `get`/`set` is already a tight vectorizable loop. ## Validation - `cd skiplang/prelude && SKTEST_JOBS=8 ../../bin/run-sktest.sh` with the pinned stage0 toolchain (CI-equivalent) in a clean worktree at the branch tip: **146/146 pass**, including all `JoinTest`s and a new `testJoinMultibyte` (multibyte elements/separators: `héllo, wörld, 🎉`, `α→β→γ`). - `skfmt` clean; CircleCI on the branch head. - `std` bumped `0.8.0 -> 0.8.1` (patch: internal performance fix, no API change). AI-assisted: implemented with Mistral Vibe (`Co-Authored-By` trailers on the commits).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-ups to #1707 (
Sequence::mapToArray) and the now-redundant #1704: more places where temporary collections can be avoided. Eight small commits, one logical change each:prelude: add Sequence::toArray— newSequence::toArray(): Array<T>API (base defaultcollect(Array)).IndexedSequenceoverrides it with a singleArray::fillBypass,Bytesfills directly from its native buffer viaget, andArraycopies withfillBy(the result must be frozen, so aliasingthisis unsound).Vector/List/Queue/Set/SortedSetalready hadtoArrayand now override the base method. IncludesToArrayTest; bumpsstd0.7.0 -> 0.8.0 (minor: new API).prelude: use toArray instead of collect(Array) in String::join and poll—String::joingoes throughSequence::toArray(dispatching to the direct implementations); theSystempoll handler converts its read slice viaBytes::toArray, resolving aTODO: Get rid of .collect(Array).skstore: build getArray results directly from the match range—IFixedDir::getArray/getArraySourceKeypreviously didgetAll(...).map(...).collect(Array)(iterator -> Vector -> Array). They now use a newfindAllRangehelper (findFirstBy+ linear count, shared withfindAllBy) and oneArray::fillByover the matching index range.prelude: avoid temporary collections at more call sites—mapToArraywhere a mapped Vector was immediately copied to an Array (RangeMap.fromRangeMap, JSON object fields,Doc.inspect), andtoArray()instead ofcollect(Array)on Vectors (IO.read_to_string/read_line,Unsafehex formatting), resolving the remaining collect(Array) TODOs.prelude: implement Sequence::join calling toString once per element—Sequence::joinno longer doesvalues().collect(Vector).join(separator)(Vector + Array temporaries). It renders each element withtoStringexactly once, collecting the rendered strings (with separators interleaved) into a singleVector<String>, then flattens that into the output char buffer. TheVector::join/List::joinoverrides that built intermediate collections to reachArray::joinare removed in favor of the base implementation. AddsJoinTest.prelude: String::join delegates to Sequence::join—String::join(pieces)is nowpieces.join(this), so it no longer materializespieces.toArray()first; Array keeps its optimizedjoinoverride.prelude: slice the popen read buffer once per read— the poll handler evaluatedbuf.slice(0, nbytes)twice per read (once forcaptured.extend, once for the handler string); the slice is bound once and passed to both.skstore: fix findAllBy/findAllRange docstrings—findAllBy's docstring had ended up stacked abovefindAllRange; both are now documented properly (findAllRangereturns(i, n)with matches exactlyi..i+n-1,n == 0when none).Not done (needs compiler/native work, out of scope here)
Array::filter/filterMapsingle-pass builds would need anArrayshrink/truncate native (none exists).items(),keys(),getIter()) can't avoid the Vector temp without indexed access;Vector::mcreateFromIteratoralready presizes fromIterator::sizeHint.Validation
cd skiplang/prelude && SKTEST_JOBS=8 ../../bin/run-sktest.shwith the pinned stage0 toolchain (CI-equivalent) in a clean worktree at the branch tip: 145/145 pass, includingJoinTest(join over Array/Vector/List/Range, empty sequences, customShowelements),ToArrayTest,MapToArrayTest, and alltestCompactFixed*suites (which exercisegetArray/getArraySourceKeyover wide key/source ranges).skfmt.Notes
stdbumped0.7.0 -> 0.8.0in the toArray commit (minor: new API).Co-Authored-Bytrailers on the commits).