feat: performance benchmarking module + baseline (KOJAK-68)#34
Merged
feat: performance benchmarking module + baseline (KOJAK-68)#34
Conversation
Add `okapi-benchmarks` module using JMH (1.37) via the me.champeau.jmh Gradle plugin. Provides reproducible measurements of fanout throughput across transports and configurations. Benchmarks: - KafkaThroughputBenchmark — real Postgres + real Kafka via Testcontainers - HttpThroughputBenchmark — real Postgres + WireMock with @param httpLatencyMs injecting 0/20/100ms server-side delay (covers library-only ceiling vs realistic webhook scenarios) - DelivererMicroBenchmark — single-entry deliver() with mocked I/O, measures pure code overhead (~1.8M ops/s for Kafka with MockProducer) Methodology: - @OperationsPerInvocation(1000) on throughput benchmarks so JMH ms/op directly reflects per-message cost (reciprocal = msg/s) - @setup(Level.Trial) starts containers; @setup(Level.Invocation) populates fresh PENDING entries before each timed run - Scheduler bypassed: `processor.processNext` called in a loop until drained to measure processing capacity, not polling cadence Baseline results (smoke run, MacBook M3 Max, JDK 25, fork=1, warmup=1, iter=2): - Kafka: ~109-115 msg/s (flat across batchSize — confirms sync .get() bottleneck) - HTTP @0ms: ~1,500-3,300 msg/s (library + DB ceiling; WireMock has no real I/O) - HTTP @20ms: ~33-36 msg/s (flat — sequential blocking dominates) - HTTP @100Ms: ~9 msg/s (flat — close to theoretical 1000/100 = 10 msg/s) These numbers establish a reference point for upcoming optimizations (KOJAK-70..79) — every subsequent change re-runs the suite and documents before/after in `benchmarks/results-postopt-*.md`. For release-quality numbers with confidence intervals, run `./gradlew :okapi-benchmarks:jmh` (default fork=2, warmup=3, iterations=5).
Adds a Performance section between Compatibility and Build summarizing the KOJAK-68 baseline throughput across transports and batch sizes, plus a link to the full benchmarks/ directory with methodology and reproduction instructions. Also extends the Build section with the JMH command for completeness.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Establishes a reproducible JMH-based performance benchmarking setup and captures the pre-optimization baseline for okapi. Provides a reference point against which all subsequent optimizations (KOJAK-70..79) will be measured and documented.
KOJAK-68 — Performance benchmarking module + baseline measurement
What's included
okapi-benchmarksmoduleme.champeau.jmhGradle pluginfork=2, warmup=3, iter=5for release-quality runsokapi-benchmarks/(not published, likeokapi-integration-tests)Benchmarks
KafkaThroughputBenchmark@Param batchSize ∈ {10, 50, 100}HttpThroughputBenchmark@Param batchSize × httpLatencyMs ∈ {0, 20, 100}DelivererMicroBenchmarkdeliver()withMockProducer/ WireMock — measures pure code overheadMethodology touches
@OperationsPerInvocation(1000)so JMH ms/op directly reflects per-message cost@Setuptruncates outbox and re-populates fresh PENDING entriesDocumentation
benchmarks/README.md— methodology + how to run + caveats (WireMock in-JVM, localhost ≠ production)benchmarks/results-baseline-2026-04.md— full baseline tables with interpretationBaseline numbers (smoke run on M3 Max, JDK 25 LTS)
Kafka
Flat across batchSize — confirms sync
producer.send().get()blocking is the bottleneck.HTTP
At non-zero latency, throughput is flat — proves sequential
httpClient.send()blocking dominates. Library + DB ceiling visible only atlatencyMs=0.Microbenchmark
KafkaMessageDeliverer.deliver()withMockProducer: ~1.8M ops/s — pure code overhead is negligibleHow to reproduce
Test plan
./gradlew :okapi-benchmarks:jmhClassescompiles cleanly./gradlew :okapi-benchmarks:ktlintCheckpasses-lflag)DelivererMicroBenchmark.kafkaDeliver)@Parammatrixbenchmarks/results-baseline-2026-04.mdNotes for reviewers
MockWebServerfor tighter measurements.github-action-benchmarkconsuming the JSON output..logfiles are gitignored — verbose console output is not a useful artifact; structured JSON is sufficient for archival.