Repository navigation
CUDA (GB10): a second carveout source, spark-attic's atticd, opt-in - #2762
bertholomus wants to merge 1 commit into
Conversation
|
Thanks @bertholomus, the atticd client is careful work. Every reply's descriptors are counted and closed, the key
|
|
Agreed on one client for the lending protocol. Here is the shape I would take, and a way to land it as one pull request rather than two. Where my side is: The offer: rather than open a second PR with a second client, give me access to your fork - add @BadAd84 as a collaborator, or name a branch for me to push - and I will fold my client, the refusals and those tests into your The shape, so we agree before either of us edits:
|
|
Thanks @ashhart and @BadAd84. Yes to one client for the protocol, and yes to Adam's offer. 1. One backend. @BadAd84, your shape works for us: 2. Sources. We cannot confirm that. spark-attic's README states it: we learned the protocol from Kindling's published dispram client. In detail: on 2026-10-27 one of our agents read 3 to 5. Adam's shape covers them: a fixed 1792 MiB when unset and a refusal when less is free, the fd size check before the import, the close test, 6. We will add the tests for the new skips after the fold. 7. Receipts on the head are not run yet. Ours will come from a GX10 on stock DGX OS 7.6.0 with atticd, after the fold. I have no date for that run. This stays a draft.
|
0e8576e to
77ef129
Compare
|
Thanks @bertholomus for the straight answer on sources and for stepping your client aside so one can land. Thanks We're happy with the shape you've agreed:
For the fold:
Points 3 to 7 of our earlier review apply to the folded head:
Keeping each of you as the author of your own commits lets us credit you both.
|
d8dc15f to
d8a29b3
Compare
|
Answering your points on the fold. The branch is How the clean room worked, in detail. Four probe scripts, written and run against the running daemon on a GB10 (kindling spark-os 0.9.5, driver 580.178.04), established every wire fact by observation and nothing else:
Those outputs were written up as a specification that describes the daemon's behaviour and says nothing about any implementation. The client was then written from that specification alone by a clean-room agent, in a copy of this tree with Monitored. I read that agent's transcript line by line and listed every path it opened: its protocol reads are the specification, its other reads are files inside that copy, and the only writes are the new module. Two excursions outside it, disclosed rather than omitted: it read The key's default. Kindling's README publishes it, line 32: The refusal walk is carried over verbatim from your branch, renamed to the new settings. My first attempt added skips of its own and crashed
The pin. Points 3 to 7. With Receipts. On this head, with dispramd: 34 of the 36 gate digests identical with the carveout off and on - the two that differ are the resume gate's, whose prompt carries a random uuid per run - 893 MiB of KV planes into the carveout at Tests for the new skips. You offered to add them after the fold; the strict-reader test already exercises the walk's skips, so between us that row is covered either way. The head has had two full review passes. The first found a reply parse bounded more tightly than the datagram it parses, and a test that pinned a serialiser's spacing instead of the protocol's fields. The second found that the ask was sent unrounded, so the daemon's round-up could refuse a size that would have fitted - rounding it down, my first attempt at the fix, would have granted less than planned - plus two missing tests, one for a refusal that carries a |
|
Thanks @BadAd84, this is the account we hoped for: probes first, a behaviour spec, a client written from the spec alone in a tree without the old one, and every excursion disclosed. One practical thing: we can't find
|
|
Thanks @BadAd84 for the fold, and @bertholomus for opening your branch to it. The client is careful: the ask is rounded up and a short grant refused (
|
6098eae to
9565378
Compare
|
Answering your points. The branch is How the names were found, and how anyone can re-derive them. Not from a client: from the daemon's own words and a candidate space nobody curated.
And a note on those logs, because the timestamps are all today: the earlier runs were not kept, so I re-ran every probe, the sweep and the capture to have something to post. The files' timestamps are therefore the re-run, and their contents carry no date of their own; nothing in them was edited after the fact, so what they show is exactly what a fresh run produces. The implementation was then written from that behaviour by an agent working in a copy of the tree with both clients absent, so neither was available to read; its transcript is the record of what it opened, and the two excursions it made (a Your points, in order.
Tests for the new skips. You offered to add them after the fold; they are in: a test drives the walk with each of the four daemon settings set to a bad value, asserting it is skipped while the carveout is off and refused by name when it is on. Receipts for the backend itself, on this pair with dispramd: 34 of the 36 gate digests identical with the carveout off and on - the two that differ are the resume gate's, whose prompt carries a random uuid per run - 893 MiB of KV planes into the carveout at The head has had three review passes. The first found a reply parse bounded more tightly than the datagram it parses and a test that pinned a serialiser's spacing instead of the protocol's fields. The second found the unrounded ask. The third, after porting onto today's main, found the CLI arity, the missing |
|
Thanks @BadAd84, this is a thorough answer. Deriving the protocol from the daemon's own refusals, a sweep over an uncurated vocabulary and a byte-for-byte capture, with the scripts posted so anyone can re-run them, is the clean-room record we asked for. Credit to @bertholomus for the rows and the walk is right too. The off-path checks being identical to main with the carveout off, and failing first because no verb was passed, is a good catch. We'll review it in full once the title and description edit is in and it's out of draft. Main is frozen for the 1.2.7 release this afternoon, so it lands after that.
|
|
The specification the client was written from, as promised in the parent comment - it describes the daemon's behaviour and nothing about any implementation. The specification# Lending-daemon protocol: the specification a client is written from
This is the ONLY source an implementation should use. It states what the daemon does, as observed by running it and
talking to its socket on 2026-11-10 (kindling spark-os 0.9.5 on a GB10, driver 580.178.04), plus what is published in
Kindling's own README. It deliberately says nothing about how any existing client is written, and an implementer must
not read `python/dispram.py`, `dispramd.py`, or any other implementation, in any language.
## 1. Transport
- A Unix domain socket, path given by the operator (`/run/dispram/dispram.sock` is the usual one; another daemon may
listen elsewhere).
- Socket type MUST be `SOCK_SEQPACKET`. `SOCK_STREAM` and `SOCK_DGRAM` are refused at connect time with
`EINVAL` ("Protocol wrong type for socket").
- One request per datagram, one reply per datagram. Both are UTF-8 JSON objects. Byte order and padding are not
involved: the payload is text, and the only binary part is the file descriptor that travels in the ancillary data.
- Requests may be pipelined on one connection only while they are accepted: an `info` followed by an `alloc` works,
and the connection stays usable. A refusal closes the connection (section 6), so a client that continues past one
must reconnect.
## 2. The key
Every request must carry `"key"`, a string. The daemon's key is a per-installation constant; the operator supplies it
(`kindlingai_1` on the reference install). A request with no key, an unknown key, or a payload that is not JSON at all
gets exactly:
{"ok": false, "error": "missing or unknown key"}
So a key error is not distinguishable from a malformed message from the client's side; treat both as "refused".
## 3. `info`
Request:
{"key": "<key>", "op": "info"}
Reply, always `ok:true`-less (it carries no `ok` field at all), four integers in bytes:
{"base": 10739515392, "size": 2145386496, "free": 2145386496, "largest": 2145386496}
- `base`: the reservation's base address (the daemon's own view; not the slice offset).
- `size`: the whole reservation.
- `free`: bytes currently unclaimed.
- `largest`: the largest single claim that would succeed right now (equal to `free` when the reservation is empty,
and it drops by exactly the granted size while a claim is held).
Numbers are printed with `", "` separators as shown, but a client MUST parse JSON rather than the exact spacing.
## 4. `alloc`
Request:
{"key": "<key>", "op": "alloc", "size": <integer bytes>}
Success: the reply carries `ok:true`, the reservation `base`, and `size` - the GRANTED size - and exactly one file
descriptor arrives in the ancillary data as `SCM_RIGHTS`:
{"ok": true, "base": 10739515392, "size": 2097152}
**The daemon rounds the request UP to its 2 MiB granularity.** Observed: request 1 → granted 2097152; 1048576 →
2097152; 2097151 → 2097152; 2097152 → 2097152; 3145728 → 4194304. A client that wants exactly N bytes must ask for a
value that is already a multiple of 2 MiB, and must read the granted `size` from the reply rather than assuming its
request was honoured.
Refusals, with `ok:false`, an `error` string, and NO descriptor:
| Condition | Reply |
|---|---|
| request larger than the whole reservation | `{"ok": false, "error": "size must be an integer from 1 to 2145386496"}` |
| more than `free`/`largest` | `{"ok": false, "error": "not enough free carveout", "base": ..., "size": ..., "free": ..., "largest": ...}` |
| `size` missing, zero, negative, non-integer | `{"ok": false, "error": "size must be an integer from 1 to <reservation>}` |
| unknown `op` (e.g. `"free"`) | `{"ok": false, "error": "unknown op"}` |
| missing/unknown `key` | `{"ok": false, "error": "missing or unknown key"}` |
There is NO op that releases a slice: `free` is an unknown op. See section 6.
## 5. The lent file descriptor
- It is the memory, shared as a POSIX file descriptor. The client maps it with the CUDA virtual-memory API:
`cuMemAddressReserve` for a range of its own, `cuMemImportFromShareableHandle` with handle type
`CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR` (1), `cuMemMap`, then `cuMemSetAccess` read-write for the current device.
- **The descriptor is not seekable**: `lseek(fd, 0, SEEK_END)` fails with ESPIPE (errno 29). A client MUST NOT use the
fd's own length as its size; the granted `size` in the reply is the authority, and the mapping must not exceed it.
- Once imported, the mapping holds its own reference to the memory, so the client may close the descriptor afterwards.
## 6. Lifetime and release
- A claim lives while the connection that asked for it stays open. Observed: with a 4 MiB claim held, `info` on
another connection reports `free` reduced by exactly 4194304; after the holding connection is closed, `free` returns
to the full reservation. So: **closing the window socket is the release**, and a client must keep that socket open
for as long as it wants the memory.
- Both are needed for a clean release, in that order: unmapping the CUDA range releases the client's reference, and
closing the socket gives the memory back to the daemon.
- **The daemon closes the connection after refusing a request.** Observed on the wire: a refused request is answered
and the peer then sends EOF (a zero-byte datagram follows each error), while a successful request leaves the
connection open. A client that would send a second request must reconnect after any refusal; the refusal itself is
already queued, so a client that stops there reads its cause normally.
## 7. What the client must refuse rather than fit
- A daemon that lends less than the ask (only possible if a client asks for a size the daemon then clamps) is a
refusal, not a smaller pool: the placement above depends on the size it planned for.
- A missing socket, a refused connect, a refused claim, a reply that is not the expected JSON, a reply with no file
descriptor, and a `size` in the reply smaller than the ask are all load-stopping refusals with the cause named.
- Never map more than the granted size, and never report a carveout larger than what the daemon granted.
## 8. Values for the reference install
Socket `/run/dispram/dispram.sock`, key `kindlingai_1`, reservation `2145386496` bytes (2046 MiB) at base
`10739515392`, granularity 2 MiB. Another daemon that speaks this protocol may differ in all three; the client takes
the socket and the key from settings and the sizes from the daemon's replies.
## How the vocabulary is obtained (2026-11-11)
Field names: the daemon names them in its refusals (`key` from `{}`, `op` from a key-only body, `size` and its bound
from an `alloc` with no size). Op names: a sweep of a candidate space harvested from standard code on the machine -
`harvest.py` takes every lowercase identifier from `/usr/include`, the Python stdlib and the Zig standard library
(58,337 tokens), `sweep.py` sends each as `op` and records anything that is not `unknown op`. Result in 2 seconds:
exactly `alloc` and `info`, and the `alloc` reply supplies `size` and its bound. `sweep-corpus.log` holds the output.
A plain English dictionary finds `info` only, because "alloc" is not a word in one - the code corpus is what makes
the sweep complete enough to be useful. |
What this adds
Why
Where the client came fromThe protocol was derived from the daemon's own behaviour - field names from its refusals, op names from a sweep of 58,337 identifiers harvested from the C/POSIX headers, the Python stdlib and the Zig stdlib on the machine, and the wire format confirmed by a pass-through capture. The specification, the scripts and their captured output are in the thread on this PR. The implementation was then written from that behaviour by an agent working in a copy of the tree with both clients absent, so neither was available to read. Refusals and mappingA daemon that lends less than asked is refused rather than fitted, so the load stops with the cause named. The reply's granted size is the authority, Gates on this head
ReceiptsMain against this branch with the carveout off, both trees built from source: Based on Co-authored-by: bertholomus 311723197+bertholomus@users.noreply.github.com for the settings rows and the refusal walk. |
The probes, first half
import json, os, socket, struct
PATH = "/run/dispram/dispram.sock"
TYPES = {"SEQPACKET": socket.SOCK_SEQPACKET, "STREAM": socket.SOCK_STREAM, "DGRAM": socket.SOCK_DGRAM}
def try_type(name, typ, payload):
s = socket.socket(socket.AF_UNIX, typ)
s.settimeout(3)
try:
s.connect(PATH)
except OSError as e:
s.close()
return f"{name}: connect failed: {e}"
try:
if typ == socket.SOCK_DGRAM:
s.send(payload)
else:
s.sendall(payload)
if typ == socket.SOCK_STREAM:
data = s.recv(4096)
else:
data, anc, flags, addr = s.recvmsg(4096, socket.CMSG_SPACE(4))
fds = [struct.unpack("i", c[2][:4])[0] for c in anc if c[0] == socket.SOL_SOCKET and c[1] == socket.SCM_RIGHTS] if typ != socket.SOCK_STREAM else []
return f"{name}: reply={data!r} fds={fds}"
except Exception as e:
return f"{name}: {type(e).__name__}: {e}"
finally:
s.close()
print("== socket type probe, empty object ==")
for name, typ in TYPES.items():
print(try_type(name, typ, b"{}"))
print("\n== request vocabulary probe on the type that answered ==")
for body in (b'{"op":"info"}', b'{"op":"Info"}', b'{"op":"hello"}', b'{"key":"kindlingai_1","op":"info"}', b'{"op":"alloc","size":2097152}'):
print(try_type("SEQPACKET", socket.SOCK_SEQPACKET, body))
import os, socket, struct
PATH = "/run/dispram/dispram.sock"
def call(body, want_fd=False):
s = socket.socket(socket.AF_UNIX, socket.SOCK_SEQPACKET)
s.settimeout(5)
s.connect(PATH)
s.sendall(body)
data, anc, flags, addr = s.recvmsg(4096, socket.CMSG_SPACE(4))
fds = [struct.unpack("i", c[2][:4])[0] for c in anc if c[0] == socket.SOL_SOCKET and c[1] == socket.SCM_RIGHTS]
return s, data, fds
print("== alloc with the key ==")
s, data, fds = call(b'{"key":"kindlingai_1","op":"alloc","size":2097152}', True)
print("reply:", data)
print("fds:", fds)
if fds:
fd = fds[0]
print("fd size (lseek):", os.lseek(fd, 0, os.SEEK_END))
# the socket closing is the release: the daemon frees the slice when the last reference goes
s.close()
print("closed the window socket; daemon should have the slice back")
print("\n== an oversized ask ==")
s, data, fds = call(b'{"key":"kindlingai_1","op":"alloc","size":4294967296}', True)
print("reply:", data, "fds:", fds); s.close()
print("\n== a malformed body ==")
s, data, fds = call(b'not json', True)
print("reply:", data, "fds:", fds); s.close()
print("\n== an unknown op ==")
s, data, fds = call(b'{"key":"kindlingai_1","op":"free"}', True)
print("reply:", data, "fds:", fds); s.close()
import json, os, socket, struct, time
PATH = "/run/dispram/dispram.sock"
KEY = "kindlingai_1"
def conn():
s = socket.socket(socket.AF_UNIX, socket.SOCK_SEQPACKET)
s.settimeout(5)
s.connect(PATH)
return s
def say(s, body, want_fd=False):
s.sendall(body)
data, anc, flags, addr = s.recvmsg(4096, socket.CMSG_SPACE(4) if want_fd else 0)
fds = [struct.unpack("i", c[2][:4])[0] for c in anc if c[0] == socket.SOL_SOCKET and c[1] == socket.SCM_RIGHTS]
return data, fds
print("1. two requests on one connection (info then alloc):")
s = conn()
print(" info :", say(s, f'{{"key":"{KEY}","op":"info"}}'.encode())[0])
print(" alloc:", say(s, f'{{"key":"{KEY}","op":"alloc","size":2097152}}'.encode(), True))
s.close()
print("\n2. key on every message? (info ok, then a keyless alloc on the same connection):")
s = conn()
print(" info :", say(s, f'{{"key":"{KEY}","op":"info"}}'.encode())[0])
try:
print(" alloc :", say(s, '{"op":"alloc","size":2097152}'.encode(), True))
except Exception as e:
print(" alloc :", type(e).__name__, e)
s.close()
print("\n3. size handling (largest={}), asks of 1, 2MiB-1, 3MiB:".format(2145386496))
for ask in (1, 2097151, 3145728):
s = conn()
d, fds = say(s, f'{{"key":"{KEY}","op":"alloc","size":{ask}}}'.encode(), True)
print(f" ask={ask}: reply={d} fds={fds}")
s.close()
print("\n4. over-ask (largest+1):")
s = conn()
print(" ", say(s, f'{{"key":"{KEY}","op":"alloc","size":2145386497}}'.encode(), True))
s.close()
print("\n5. malformed and unknown-op replies:")
for body in (b'not json', b'{"key":"' + KEY.encode() + b'"}', b'{"key":"' + KEY.encode() + b'","op":"free"}', b'{"key":"wrong","op":"info"}'):
s = conn()
print(" ", body, "->", say(s, body)[0])
s.close()
print("\n6. does closing a connection with a claim free it (alloc, drop socket, alloc the whole reservation):")
s = conn()
d, fds = say(s, f'{{"key":"{KEY}","op":"alloc","size":2145386496}}'.encode(), True)
print(" first claim:", d, "fds:", fds)
for fd in fds: os.close(fd)
s.close()
time.sleep(0.3)
s = conn()
print(" after close:", say(s, f'{{"key":"{KEY}","op":"alloc","size":2145386496}}'.encode(), True))
s.close()
print("\n7. key length / unknown key:")
s = conn()
print(" ", say(s, b'{"key":"x","op":"info"}')[0])
s.close()
import os, socket, struct
PATH = "/run/dispram/dispram.sock"
KEY = "kindlingai_1"
def conn():
s = socket.socket(socket.AF_UNIX, socket.SOCK_SEQPACKET); s.settimeout(5); s.connect(PATH); return s
def say(s, body, want_fd=True):
s.sendall(body)
data, anc, flags, addr = s.recvmsg(4096, socket.CMSG_SPACE(4) if want_fd else 0)
fds = [struct.unpack("i", c[2][:4])[0] for c in anc if c[0] == socket.SOL_SOCKET and c[1] == socket.SCM_RIGHTS]
return data, fds
s = conn()
print("A. info with the carveout free:", say(s, f'{{"key":"{KEY}","op":"info"}}'.encode(), False)[0])
for ask in (1, 1048576, 2097151, 2097152, 3145728):
c = conn()
d, fds = say(c, f'{{"key":"{KEY}","op":"alloc","size":{ask}}}'.encode())
got = None
if fds:
try:
got = ("lseek", os.lseek(fds[0], 0, os.SEEK_END))
except OSError as e:
got = ("lseek-failed", e.errno)
for fd in fds: os.close(fd)
print(f"B. ask={ask}: reply={d} fd={fds} fdsize={got}")
c.close()
print("C. after those closes, info again:", say(conn(), f'{{"key":"{KEY}","op":"info"}}'.encode(), False)[0])
c = conn()
d, fds = say(c, f'{{"key":"{KEY}","op":"alloc","size":4194304}}'.encode())
print("D. hold a 4 MiB claim:", d, "fd", fds)
d2, fds2 = say(conn(), f'{{"key":"{KEY}","op":"info"}}'.encode(), False)
print(" daemon sees:", d2)
c.close()
print("E. after closing the holder, info:", say(conn(), f'{{"key":"{KEY}","op":"info"}}'.encode(), False)[0]) |
The probes, second half, and the sweep
import socket
PATH = "/run/dispram/dispram.sock"
K = "kindlingai_1"
def ask(body):
s = socket.socket(socket.AF_UNIX, socket.SOCK_SEQPACKET); s.settimeout(5); s.connect(PATH)
s.sendall(body); d = s.recv(4096); s.close(); return d.decode(errors="replace")
print("1. {} ->", ask(b"{}"))
print("2. key only, no op ->", ask(f'{{"key":"{K}"}}'.encode()))
print("3. unknown op 'bogus' ->", ask(f'{{"key":"{K}","op":"bogus"}}'.encode()))
print("4. unknown op 'list' ->", ask(f'{{"key":"{K}","op":"list"}}'.encode()))
print("5. unknown op 'help' ->", ask(f'{{"key":"{K}","op":"help"}}'.encode()))
print("6. op 'alloc' with no size ->", ask(f'{{"key":"{K}","op":"alloc"}}'.encode()))
print("7. op 'alloc' size string ->", ask(f'{{"key":"{K}","op":"alloc","size":"x"}}'.encode()))
print("8. op 'info' extra junk ->", ask(f'{{"key":"{K}","op":"info","zzz":1}}'.encode()))
"""Harvest a generic software vocabulary - identifiers from the C headers and the Python stdlib on this machine.
Nothing here is chosen by hand: it is every lowercase identifier in corpora any developer already has. The point is
that a protocol's op names can be searched without anyone naming them first.
"""
import os, re, sys
TOKEN = re.compile(r"\b[a-z][a-z0-9_]{1,23}\b")
roots = [
"/usr/include", # C/POSIX headers
"/usr/lib/python3.12", # the Python standard library
"/usr/lib/python3",
]
out = set()
files = 0
for root in roots:
for dirpath, dirnames, filenames in os.walk(root):
if "/test" in dirpath or "/tests" in dirpath:
continue
for fn in filenames:
if not fn.endswith((".h", ".py", ".c")):
continue
try:
with open(os.path.join(dirpath, fn), "r", errors="ignore") as f:
text = f.read(2_000_000)
except OSError:
continue
files += 1
out.update(TOKEN.findall(text))
words = sorted(out)
with open(sys.argv[1], "w") as f:
f.write("\n".join(words) + "\n")
print(f"files: {files}, distinct tokens: {len(words)}")
import re, sys
TOKEN = re.compile(r"\b[a-z][a-z0-9_]{1,23}\b")
tokens = set(open(sys.argv[1]).read().split())
tokens.update(TOKEN.findall(open(sys.argv[2], errors="ignore").read()))
tokens.discard("")
with open(sys.argv[3], "w") as f:
f.write("\n".join(sorted(tokens)) + "\n")
print("merged tokens:", len(tokens), "| has alloc:", "alloc" in tokens, "| has info:", "info" in tokens)
"""Discover the daemon's `op` vocabulary black-box: every word in a standard list, one request each."""
import socket, sys, time
PATH = "/run/dispram/dispram.sock"
K = "kindlingai_1"
def reply(op):
s = socket.socket(socket.AF_UNIX, socket.SOCK_SEQPACKET); s.settimeout(2)
try:
s.connect(PATH)
s.sendall(f'{{"key":"{K}","op":"{op}"}}'.encode())
return s.recv(512).decode(errors="replace")
except Exception as e:
return f"ERR {type(e).__name__}"
finally:
s.close()
words = [w.strip() for w in open(sys.argv[1], encoding="utf-8", errors="ignore") if w.strip()]
print(f"candidates: {len(words)}", flush=True)
hits, t0 = [], time.time()
for i, w in enumerate(words, 1):
if len(w) > 24 or not w.isascii():
continue
r = reply(w)
if r != '{"ok": false, "error": "unknown op"}':
hits.append((w, r.strip()))
print(f"HIT {w!r} -> {r.strip()}", flush=True)
if i % 50000 == 0:
print(f" {i}/{len(words)} in {time.time()-t0:.0f}s, hits={len(hits)}", flush=True)
print(f"done in {time.time()-t0:.0f}s; recognised ops: {[h[0] for h in hits]}", flush=True) |
The capture
"""A pass-through proxy for the daemon's socket: logs every request and reply, forwards descriptors.
The client talks to /tmp/lend-proxy.sock; this forwards to the daemon's own socket, pumping whichever side is
readable so both directions are captured byte for byte, including the SCM_RIGHTS descriptor pass. The daemon may
close a connection after refusing a request, so this accepts successive connections until it is killed.
"""
import os, select, socket, struct, sys
UPSTREAM = "/run/dispram/dispram.sock"
LISTEN = "/tmp/lend-proxy.sock"
log = open(sys.argv[1], "w", buffering=1)
def pump(src, dst, direction):
data, anc, flags, addr = src.recvmsg(65536, socket.CMSG_SPACE(4))
fds = [struct.unpack("i", c[2][:4])[0] for c in anc
if c[0] == socket.SOL_SOCKET and c[1] == socket.SCM_RIGHTS]
log.write(f"{direction} {len(data)}B data={data!r} fds={fds}\n")
if not data and not fds:
return False
if fds:
dst.sendmsg([data], [(socket.SOL_SOCKET, socket.SCM_RIGHTS, struct.pack("i", fds[0]))])
for fd in fds:
os.close(fd)
else:
dst.send(data)
return True
def serve_one(conn, up):
alive = True
while alive:
ready, _, _ = select.select([conn, up], [], [], 5)
if not ready:
log.write("idle\n")
return
for sock in ready:
src, dst, direction = (conn, up, "client->daemon") if sock is conn else (up, conn, "daemon->client")
try:
alive = pump(src, dst, direction)
except OSError as error:
log.write(f"closed: {error}\n")
return
if not alive:
return
try:
os.unlink(LISTEN)
except OSError:
pass
srv = socket.socket(socket.AF_UNIX, socket.SOCK_SEQPACKET)
srv.bind(LISTEN)
srv.listen(4)
log.write(f"listening on {LISTEN}, forwarding to {UPSTREAM}\n")
while True:
try:
conn, _ = srv.accept()
except OSError:
break
up = socket.socket(socket.AF_UNIX, socket.SOCK_SEQPACKET)
up.connect(UPSTREAM)
log.write("connected\n")
serve_one(conn, up)
conn.close()
up.close()
log.write("connection done\n")
"""Exercise the protocol through the proxy, using only names the sweep and the refusals produced."""
import socket, sys
PATH = "/tmp/lend-proxy.sock"
K = "kindlingai_1"
def call(sock, body, want_fd=False):
sock.sendall(body)
d, anc, flags, addr = sock.recvmsg(4096, socket.CMSG_SPACE(4) if want_fd else 0)
print(f" sent {body!r}\n got {d!r} anc={len(anc)}")
return d
for body, want_fd in ((b"{}", False),
(f'{{"key":"{K}"}}'.encode(), False),
(f'{{"key":"{K}","op":"info"}}'.encode(), False),
(f'{{"key":"{K}","op":"alloc","size":4194304}}'.encode(), True)):
s = socket.socket(socket.AF_UNIX, socket.SOCK_SEQPACKET); s.settimeout(5); s.connect(PATH)
call(s, body, want_fd)
s.close() # the daemon may have closed it already; either way the next request reconnects
print("client done; closing the connection is the release")
|
The recorded output of every run
|
|
Thanks @BadAd84 for the scripts and the new description, and @bertholomus for lending the branch. Before the full read, the pushed head (
And a few things in the code itself:
@bertholomus, once that's pushed, could you paste @BadAd84's description from the thread into the PR body, update the title and mark it ready? Then we'll do the full read.
|
TF_CUDA_CARVEOUT stays 0 or 1. TF_CUDA_CARVEOUT_SOURCE picks where the display reservation comes from: drm, the default and the existing dumb-buffer route, or daemon, where a display-memory daemon lends it in whole 2 MiB slices over a unix socket. Which daemon a box runs is a setting, not a code path: TF_CUDA_CARVEOUT_SOCKET (default /run/dispram/dispram.sock, kindling spark-os's dispramd, whose README publishes the default key), and TF_CUDA_CARVEOUT_KEY, or TF_CUDA_CARVEOUT_KEY_FILE for a daemon whose key is per box: spark-attic's atticd keeps /etc/spark-attic/key and listens on /run/spark-attic/attic.sock. The client, cuda/carveout_lend.zig, was written from a specification of the daemon's observed behaviour, by an agent working in a copy of this tree that held no client for the protocol. The specification, the probe scripts and their output are in the thread of pull request 2762. An ask is rounded up to a whole slice before it is sent, and a grant smaller than the ask is refused rather than fitted. The reply's granted size is the authority, ok must be true (a refusal carries a size field too), and a seekable descriptor's own length is checked as well. The slice is mapped through arena.zig's Vmm. One pair of calls serves every caller. carveout.requested decides whether the carveout is on and reads its rows through the settings table, where TF_DRM_CARD gains a row. Carveout.from opens the source the request names and logs a refusal once. The CLI and Nemotron make those two calls and nothing else. The key file is read only for the daemon source, as its claim is made. A file that is missing, not a regular file, empty on its first line or 256 bytes long there is refused with TF_CUDA_CARVEOUT_KEY_FILE named, and neither the path nor the file's bytes are shown. The startup walk validates the carveout's rows when it is on and skips them when it is off, as it already did for TF_CUDA_CARVEOUT_MIB, and reads TF_DRM_CARD only while the carveout is on and the card is its source. Off Linux the claim returns NoDaemon before any of the protocol code is analysed, so the Windows host tests build. The client gives the daemon 10 seconds to answer, and a daemon's rows are read only while the daemon is the source, as the card's are only while the card is. Host tests, in carveout.zig and carveout_lend_test.zig (the second runs on Linux only): the request's rows and their defaults, the key file's accepted and refused forms, the key file left unopened for the drm source, and a fake daemon on a real SEQPACKET socket for a granted claim, the descriptor by SCM_RIGHTS, a refused claim, a refusal that carries a size, a malformed reply, a short grant, an ask between two slices, a missing descriptor, a missing socket, a daemon that never answers, a daemon that has gone away, the key file's line on the wire, and the daemon getting the memory back when the connection closes. zig/tests/fixtures/gradient-serving.tsv takes carveout.zig's new hash and a row for carveout_lend.zig. docs/recipes/cuda.md says what each source costs, measured on a GB10, and CHANGELOG.md gets its Unreleased bullet. Co-authored-by: bertholomus <311723197+bertholomus@users.noreply.github.com>
9565378 to
73a8895
Compare
|
You were right to check. The branch is now one signed commit, Your points in order:
Receipts on this head. One GB10 (kindling spark-os, dispramd), Nemotron 3.5 Lightning MLX 4-bit,
Three more things in my 12:43 note were wrong. I found them while redoing the gates the way CI runs them:
While doing this work we found a speed cost in one of the two sources, and I am putting the receipts here so it is on record. It is outside this PR: the output is identical and the carveout is off by default. We will look into whether it is fixable, and if we find anything worth it, it goes in a separate PR. The same 108,012-token Nemotron prompt, greedy, 96 tokens,
Every run has the same digest, A bandwidth probe on each box says where it comes from. 1,536 MiB of each memory, GB/s, higher is faster:
Gates on
@bertholomus the atticd cell is still yours. If the key file's shape does not fit atticd, say so and I will change it. |
|
@bertholomus the description for the final head, to paste into the PR body (it starts with Closes #2747 What this adds
Why
Where the client came fromThe protocol was derived from the daemon's own behaviour: field names from its refusals, op names from a sweep of 58,337 identifiers harvested from the C/POSIX headers, the Python stdlib and the Zig stdlib on the machine, and the wire format confirmed by a pass-through capture. The specification, the scripts and their captured output are in the thread on this PR. The implementation was then written from that behaviour by an agent working in a copy of the tree with both clients absent, so neither was available to read. Shape
Gates on this head
ReceiptsOne GB10 (kindling spark-os 0.9.5, driver 580.178.04, dispramd), Nemotron 3.5 Lightning MLX 4-bit,
Based on Co-authored-by: bertholomus 311723197+bertholomus@users.noreply.github.com for the settings rows and the refusal walk. |
|
Thanks @BadAd84 for the rework, and for correcting the receipts plainly. The pushed head now matches your note:
The Nemotron digests on
Once those are in and CI is green on the new head, we'll run the card path on a GB10 before landing: Nemotron with the carveout off and with
|
Closes #2747
What this adds
TF_CUDA_CARVEOUTstays0or1. A newTF_CUDA_CARVEOUT_SOURCEsays where the display reservation comes from:drm, the default and the existing dumb-buffer route, ordaemon, where a display-memory daemon lends it in whole 2 MiB slices over a unix socket. Which daemon a box runs is a setting, not a code path:TF_CUDA_CARVEOUT_SOCKET(default/run/dispram/dispram.sock, kindling spark-os's dispramd),TF_CUDA_CARVEOUT_KEY(default is Kindling's published one, README line 32), orTF_CUDA_CARVEOUT_KEY_FILEfor a daemon whose key is per box: spark-attic's atticd keeps/etc/spark-attic/keyand listens on/run/spark-attic/attic.sock. One client for the protocol both daemons speak, which is what #2747 asks for.Why
DRM_IOCTL_MODE_CREATE_DUMBon/dev/dri/card0returnsENOSYSunder driver 580.178.04 on the GB10s here (kindling spark-os 0.9.5), so the DRM backend cannot claim the reservation even though the firmware keepsDISPLAY_FRM(2046 MiB at 0x280200000) out of system RAM. The daemon lends that memory to unprivileged CUDA processes: a slice is claimed over the socket, arrives as a descriptor bySCM_RIGHTS, and is mapped with the CUDA VMM API.Where the client came from
The protocol was derived from the daemon's own behaviour: field names from its refusals, op names from a sweep of 58,337 identifiers harvested from the C/POSIX headers, the Python stdlib and the Zig stdlib on the machine, and the wire format confirmed by a pass-through capture. The specification, the scripts and their captured output are in the thread on this PR. The implementation was then written from that behaviour by an agent working in a copy of the tree with both clients absent, so neither was available to read.
Shape
carveout.requested(Settings)decides whether the carveout is on and reads its rows through the settings table, andCarveout.from(d, request)opens the source the request names and logs a refusal once. The CLI and Nemotron make those two calls and nothing else.TF_DRM_CARD's only while the card is.TF_CUDA_CARVEOUT_KEY_FILEnamed, and neither the path nor the file's bytes are shown.okmust be true, since a refusal carries asizefield too). The mapping goes througharena.zig'sVmm.NoDaemonbefore any protocol code is analysed.Gates on this head
73a88956b, on upstreammainat6c5cef894, in the forms CI uses:zig fmt --check build.zig zig: clean on Linux and Windows.tools/zig/lean_check.pyandlean_new.py, base and head each from its own tree: 345 on the base, 343 on the head, 0 new.tools/pr_check.py --base 6c5cef894 --pr 2762: 0 errors, 0 warnings.zig build -Dfixed-cpu test: 667 of 667 steps on Linux x86_64, 643 of 643 on Windows.test-kernel-contract,test-golden,test-server-security,test-pull-securityandfuzz-cortheonpass on Linux, andtest-goldenandfuzz-cortheonon Windows.test-gradient-servingfails onmainitself, onzig/src/cuda/cublaslt.zig, whose row is stale there. Our two rows are re-pinned and that one is left alone.check-native-cuda-hostdoes not compile the CLI'smain(a deliberate bad call in it still passed), so the CLI was compiled by the device build below.Receipts
One GB10 (kindling spark-os 0.9.5, driver 580.178.04, dispramd), Nemotron 3.5 Lightning MLX 4-bit,
main(6c5cef894) and this commit both built from source on the box, greedy, 96 tokens with--ignore-eos:TF_CUDA_CARVEOUT=1 TF_CUDA_CARVEOUT_SOURCE=daemon(2 runs)--context 1310726d94d3ec88d76d94d3ec88d76d94d3ec88d7,6d94d3ec88d7f19784e7f9d3f19784e7f9d3f19784e7f9d3,f19784e7f9d3TF_CUDA_CARVEOUT_MIB=512they do not fit, 0 MiB are carved and the digest is the same. WithTF_CUDA_CARVEOUT_KEY_FILEthe digest is the same.tensorfold-native serveare identical with the carveout off and on.DumbBufferRefusedon this OS, onmainand on this head.atticd.Based on
6c5cef894.Co-authored-by: bertholomus 311723197+bertholomus@users.noreply.github.com for the settings rows and the refusal walk.