Key embedded DWARF data by ELF build-id instead of package version#98
Open
taodd wants to merge 12 commits into
Open
Key embedded DWARF data by ELF build-id instead of package version#98taodd wants to merge 12 commits into
taodd wants to merge 12 commits into
Conversation
Two libelf-based helpers that the embedded DWARF lookup will use as its new key material: std::string get_elf_build_id(const std::string& path); std::string get_host_arch(); get_elf_build_id walks the ELF .note.gnu.build-id section and hex-encodes the note descriptor. Doesn't validate descriptor length (GNU ld emits 160-bit SHA1, LLD's xxhash variant is 128-bit) — just encodes whatever bytes are there. Returns empty string on any failure (unreadable file, not an ELF, no build-id note) so callers can treat "no key available" as a soft signal to fall through to live DWARF parsing. get_host_arch normalises uname.machine to the dpkg --print-architecture naming convention (amd64, arm64, ppc64el, …) so values round-trip with package metadata and the existing files/<distro>/ directory layout. libelf is already linked into every cephtrace binary via the Makefile's LIBS := … -lelf; no build-system changes needed.
Adds a public mod_path map (basename → full path on disk) populated in add_module(). Mirrors the existing public mod_func2pc / mod_func2vf maps in style and scope. Reason: the upcoming JSON schema includes a per-module build_id field, and export_to_json needs to read each module's ELF to compute its build-id. Today export_to_json only has access to module basenames (the keys of mod_func2pc / mod_func2vf) — no way to locate the file on disk. Stashing the full path at add_module() time, before any DWARF parsing happens, also avoids a TOCTOU window where the on-disk binary could be replaced (e.g. mid-upgrade) between parse and export. No call-site changes; mod_path is populated as a side effect of the existing add_module flow.
Adds two new fields to the exported JSON:
- top-level "arch": uname.machine normalised to dpkg's naming
(amd64, arm64, ppc64el, …)
- per-module "build_id": the GNU build-id hex of the on-disk ELF
that produced the func2pc / func2vf data for this module
Both fields are written from the side-effect-free helpers added in the
previous commit (get_host_arch, get_elf_build_id) so the export step
gains no new I/O failure modes beyond what add_module already performed.
build_id is omitted if mod_path lacks an entry for the module (e.g. the
parser's data was loaded via import_from_json rather than add_module +
parse). The downstream embedded loader will treat empty build_id as
"never matches", so legacy JSONs round-tripped through this code stay
inert to the new build-id lookup path.
The loop iterating top-level JSON keys to populate mod_func2pc / mod_func2vf used to skip a hard-coded "version" entry, which would silently treat new metadata fields (arch, future additions) as if they were modules — populating empty entries in the maps and noising up fill_map_hprobes downstream. Replace the hard-coded skip with a positive predicate: only iterate keys whose value is an object containing func2pc or func2vf. Adding new top-level metadata (arch, build_id, etc.) no longer requires updating this loop.
Extends the generated EmbeddedModule / EmbeddedVersion C structs and
their initializers to carry the two new metadata fields that
DwarfParser::export_to_json now writes:
struct EmbeddedModule { …; const char* build_id; … };
struct EmbeddedVersion { …; const char* arch; … };
Both fall back to an empty string when the source JSON lacks the field.
Legacy JSONs without arch / build_id therefore round-trip through the
generator unchanged on disk and produce empty values in the embedded
arrays, which the runtime matcher will treat as never-matches once the
key switches from version to build-id.
Also factors a shared is_module_entry() predicate so analyze_limits and
generate_version_entry stop using a hard-coded "skip the version key"
heuristic that would silently treat new metadata keys as modules.
Switches DwarfParser::import_from_embedded() from version-string keying
(dpkg/rpm shell-out, host-namespace bound, arch-blind) to ELF GNU
build-id keying (read from the target's .note.gnu.build-id, namespace-
agnostic, arch-specific by construction).
New signature:
bool import_from_embedded(
const std::vector<std::pair<std::string /*basename*/,
std::string /*build_id hex*/>>& modules,
const std::string& trace_type,
std::string* matched_version_out = nullptr);
An entry matches iff its modules[] set equals the caller's
(basename, build-id) set exactly. Any empty build-id on either side
disqualifies that entry — legacy JSONs predating the build-id scheme
therefore never match, falling cleanly through to live DWARF parsing.
For osdtrace there is one (basename, build-id) pair (ceph-osd). For
radostrace there are three (librbd.so.1, librados.so.2,
libceph-common.so.2); all three build-ids must match the same embedded
entry so we never silently load a JSON whose libraries don't agree.
radostrace's squid-or-above struct-offset gate previously called
get_package_version() directly; it now consumes the matched embedded
entry's version string via the matched_version_out parameter, removing
the need for a second dpkg shell-out on the fast path. Live-parse and
-i/--import-json paths still use get_package_version / get_version_from_json
respectively — the dpkg/rpm path is preserved for those.
osdtrace and radostrace call-site updates included in the same commit
because they're inseparable from the new signature.
Closes the three known correctness failure modes of version-keying:
- snap/container deployments (host dpkg returns wrong version)
- cross-architecture (x86 JSON silently loaded on arm64)
- custom rebuilds at same version string
…names Three coupled test-side changes for the build-id-keyed JSON migration: 1. tests/compare_dwarf_json.py: replace the keys1 == keys2 top-level equality check with a positive predicate matching the C++ side's new "module entry = dict containing func2pc or func2vf" rule. Metadata keys (version, arch, and any future addition) are no longer flagged as diffs; differences are surfaced as informational notes the same way version differences already are. 2. tests/dwarf-compare.sh: glob the reference filename instead of hard-coding it, so reference JSONs that carry an optional arch suffix (osd-VERSION_<arch>_dwarf.json) are picked up alongside the legacy arch-less naming. 3. tests/functional-test-microceph.sh: same glob fix for the runtime DWARF JSON discovery used by the --import-json path. All three changes are no-ops against existing JSONs and existing filename layouts; they only matter once the migration adds the metadata fields and arch-suffixed filenames begin appearing.
Migrates 37 of 49 reference DWARF JSONs from the legacy version-keyed
schema to the new build-id-keyed schema by adding two fields:
- top-level "arch": "amd64" (all checked-in JSONs are x86_64)
- per-module "build_id": <hex> from the on-disk binary in the
matching distribution package
Generated by running tools/migrate_jsons_to_build_id.py, which fetches
each .deb / .rpm and reads the GNU build-id via readelf -n.
Also fixes the package mapping inside migrate_jsons_to_build_id.py:
libceph-common.so.2 is shipped by librados2 (not libceph-common* / not
ceph-base — neither package exists / contains it on Ubuntu). Verified
by inspecting the contents of the 19.2.3-0ubuntu0.24.04.3 ceph-base,
ceph-common, ceph-mon, librados2, librbd1, etc. .debs.
12 JSONs remain unmigrated and are documented in the script's output:
4 × CentOS Stream entries (centos-stream/{osdtrace,radostrace}/
{osd-,rados-}2:18.2.7-0.el9_dwarf.json,
*2:19.2.3-0.el9_dwarf.json)
— RPM mirror URL pattern in the script does not currently locate
these. They keep working via --import-json.
7 × Ubuntu Cloud Archive entries (~cloud0 suffix on jammy)
— the cloud archive uses a different pool layout than the
cephadm/security PPAs the script knows about.
1 × Debian unstable entry (18.2.7+ds-1)
— sourced from a different archive entirely.
These twelve unmigrated JSONs will not match the build-id-keyed
embedded fast path; they keep working via osdtrace -i / radostrace -i
exactly as today. Generating their build-ids is a follow-up exercise
(needs the cloud-archive pool path + a CentOS Stream mirror that still
carries the rotated versions).
Three small additions to migrate_jsons_to_build_id.py close the
remaining 12-of-49 gap by adding archive-specific URL discovery and a
pure-Python RPM extractor:
1. Ubuntu Cloud Archive (~cloudN versions). These don't appear in
ubuntu/+source/<pkg> on Launchpad — they're published in PPAs
under launchpad.net/~ubuntu-cloud-archive (caracal-staging,
yoga-staging, etc.). Probe a candidate PPA list for each ~cloudN
version's ceph_<ver>.dsc until one matches, then fetch the .deb
from the same PPA's +files endpoint.
2. Debian unstable (e.g. 18.2.7+ds-1). Pull from snapshot.debian.org
via its JSON binfiles API: look up the binary by name+version,
read the SHA-1 hash, fetch /file/<hash>.
3. CentOS Stream RPMs (2:VER-0.el9). Add download.ceph.com/rpm-<X.Y.Z>/
(upstream version-pinned archive) ahead of the mirror.stream.centos.org
paths in the candidate list. Older versions long rotated out of
CentOS Stream's distro pools are still kept here by the Ceph
project itself.
To avoid forcing rpm2cpio onto every contributor's machine, also add
_extract_rpm_pure_python(): a stdlib-only fallback that locates the xz
magic inside the RPM (skipping lead+signature+header), decompresses
with lzma, and walks the newc cpio payload writing regular files into
dest_dir. rpm2cpio + cpio are still tried first when available.
End-to-end result: 49 of 49 reference JSONs now carry "arch" and
per-module "build_id" fields. The build-id-keyed embedded fast path
now covers every supported Ceph version cephtrace ships data for.
Two unrelated regressions surfaced by the first CI run on PR #98: 1. tests/functional-test-embedded-dwarf.sh greps for the string "Using embedded DWARF data" as the embedded-mode marker (assertions 8.1 and 9.1). My new import_from_embedded() log line said "Found embedded DWARF data (...)" — neither marker was present in the log, so the test failed even when the embedded path actually matched the target binary's build-id. Restore the "Using embedded DWARF data" prefix; the new per-module build-id breakdown still follows on subsequent lines. 2. flake8 in tools/ reported one F401 (unused 'os' import), two E231 (missing whitespace after commas), and a handful of E501 (lines exceeding the 79-char default). Trivial mechanical fixes — shorten URL strings via factored-out base-URL variables, drop the unused import, split a multi-line list literal. No behavioural change beyond the log-prefix tweak. Verified locally: make osdtrace radostrace kfstrace builds clean; flake8-equivalent 80-col scan reports no remaining violations in tools/.
taodd
added a commit
that referenced
this pull request
May 14, 2026
Two unrelated regressions surfaced by the first CI run on PR #98: 1. tests/functional-test-embedded-dwarf.sh greps for the string "Using embedded DWARF data" as the embedded-mode marker (assertions 8.1 and 9.1). My new import_from_embedded() log line said "Found embedded DWARF data (...)" — neither marker was present in the log, so the test failed even when the embedded path actually matched the target binary's build-id. Restore the "Using embedded DWARF data" prefix; the new per-module build-id breakdown still follows on subsequent lines. 2. flake8 in tools/ reported one F401 (unused 'os' import), two E231 (missing whitespace after commas), and a handful of E501 (lines exceeding the 79-char default). Trivial mechanical fixes — shorten URL strings via factored-out base-URL variables, drop the unused import, split a multi-line list literal. No behavioural change beyond the log-prefix tweak. Verified locally: make osdtrace radostrace kfstrace builds clean; flake8-equivalent 80-col scan reports no remaining violations in tools/. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI runs both flake8 and pylint with --fail-on-anything semantics; the
prior commit only addressed flake8. Pylint additionally flagged:
- C0116 missing-function-docstring on six helpers in
tools/migrate_jsons_to_build_id.py — added one-liners.
- C0415 import-outside-toplevel for "import lzma" inside the
pure-Python RPM extractor — hoisted to module top.
- R1732 consider-using-with for the subprocess.Popen() wrapper around
rpm2cpio — wrapped in `with`.
- C0103 invalid-name for META_KEYS in tests/compare_dwarf_json.py —
pylint enforces snake_case for module-and-function-local "constants"
that aren't truly module-level; renamed to lowercase meta_keys.
- R0914/R0912/R0915 too-many-{locals,branches,statements} for
migrate_one_json() — left as one function (breaking it up would
obscure the per-module loop); silenced with an inline
pylint-disable plus comment.
- W0718 broad-exception-caught for the migration main loop's
Exception-as-fallback — kept the broad except (migration must
not abort the whole run on one bad JSON) but switched to the
explicit pylint-disable spelling.
Local `python3 -m pylint` now reports 10.00/10 across all three files.
taodd
force-pushed
the
embedded-dwarf-build-id
branch
from
521e6f1 to
d75f547
Compare
…iners
When -p <pid> is given and the target is containerized (cephadm /
podman / docker / k8s), the path resolved from /proc/<pid>/exe (for
osdtrace) or find_library_path() (for radostrace) is the path as it
appears inside the container's mount namespace — e.g. /usr/bin/ceph-osd
or /usr/lib/x86_64-linux-gnu/librados.so.2. The host doesn't have those
files, so get_elf_build_id() previously opened nothing, returned an
empty string, and the embedded fast path was skipped.
Reach the actual on-disk binary via /proc/<pid>/root/ (the target's
mount-namespace view exposed read-only by procfs) so the build-id read
sees the same inode the kernel uprobe will attach to.
Verified end-to-end against a cephadm-bootstrapped single-host cluster
(Ceph v19.2.3, podman-launched ceph-osd):
Using embedded DWARF data (version 2:19.2.3-0.el9, arch amd64):
ceph-osd build-id 39ec76a70203385f8c9fbf812b95f32821290c8b
That build-id matches the centos-stream 19.2.3 reference JSON entry
exactly, so the embedded path is selected and 75+ fully-decoded trace
rows flow under sustained rbd-bench traffic.
Live-parse fallback continues to use osd_path / library paths directly —
that path was already broken for containerized targets, but that's a
pre-existing bug orthogonal to the build-id keying change.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Switches the embedded DWARF lookup from package-version string keying to GNU build-id keying, closing three correctness gaps in the current scheme:
dpkg -sran against the host's package db, returning either"unknown"or, worse, the host's version which then matched a JSON whose offsets didn't correspond to the snap's binary (silent wrong-offset trace data).func2pcaddresses andfunc2vfregister numbers are arch-specific (DWARF register numbers and SysV/AArch64 calling conventions differ), but the version string is arch-agnostic. An x86-derived JSON was silently selected on arm64..debadvertising the official version triggered a false match.ELF GNU build-id (in
.note.gnu.build-id) is unique per(source, toolchain, arch)build and is always readable from the binary itself — nodpkg/rpmshell-out, no host-namespace assumption.Design
EmbeddedVersionentry matches iff the set of (module-basename, build-id) tuples the caller provides matches the entry'smodules[]set exactly. osdtrace passes one tuple (ceph-osd); radostrace three (librbd.so.1,librados.so.2,libceph-common.so.2).get_package_version()is kept — radostrace's squid struct-offset gate still needs a version comparison, now driven by the matched embedded entry'sversionfield (passed back through the newmatched_version_outout-param). dpkg/rpm shell-out remains for the live-parse path only.files/<distro>/<trace>/<existing-version>_dwarf.json. Filenames are for human grep convenience, not for runtime lookup. Reference lookup in test scripts uses globbing now so a future arch-suffixed file (e.g.osd-19.2.3-…_arm64_dwarf.json) coexists without script changes.archfield, new per-modulebuild_idfield. Legacy JSONs without these are loaded but never match build-id lookup; still usable with-i/--import-json.Commits
JSON migration coverage
The one-shot
tools/migrate_jsons_to_build_id.pywalks everyfiles/**/*.json, fetches the matching.deb/.rpmfrom archive.ubuntu.com / ubuntu-cloud.archive.canonical.com / Launchpad PPA build artefacts / CentOS Stream mirrors, extracts the binary, and reads its build-id viareadelf -n.Result on the existing 49 reference files:
-i)The 12 unmigrated split into three groups documented in
bd372b8:~cloud0suffix on jammy) — cloud-archive pool layout not implemented in the script.18.2.7+ds-1).These keep working through
--import-jsonexactly as today; only the embedded fast path won't match them (falling cleanly through to live parse).Test plan
make osdtrace radostrace kfstraceclean on a noble host.src/embedded_dwarf_data.hcarries the newbuild_id/archfields and the 37 migrated entries' build-ids../osdtrace --versionand./radostrace --versionstill work.tests/dwarf-compare.shandtests/functional-test-microceph.shpick up arch-suffixed reference filenames via the new glob (today's references still match too).tests/functional-test-embedded-dwarf.shshould pass: against microceph's snap-confined ceph-osd, the new build-id lookup either matches (if the migrated JSON's build-id corresponds to the snap's binary) OR cleanly falls through to live parse — both are valid per the test's 3-way assertion.Out of scope
osdtrace -j/radostrace -jon an aarch64 host. Follow-up PR: enabledwarf-compare.shonbuild-ubuntu-arm64first, then commit arm64 JSONs.--skip-version-checkto--skip-buildid-check: backward-compat noise; left for later.🤖 Generated with Claude Code