Repository navigation
feat(events): persist events to disk so they survive the process (tier 3) - #538
abelonogov-ld wants to merge 193 commits into
Conversation
Tiers 1 and 2 both need the process to survive. When it does not, everything recorded since the last delivery goes with it -- including the crash report an application was recording when it died. Tier 3 is the answer to that: events are written to a log under Application Support, and a later launch delivers whatever the previous run left behind. The write is a plain append into the page cache, never an fsync, so a commit point costs tens of microseconds rather than milliseconds. track and identify are commit points: at .immediate the write happens on the calling thread, so a process that dies the instant after track returns still reports that event. Evaluations are not -- they are held as objects and encoded in runs, so the hottest path in the SDK pays a counter bump and nothing else. This is off by default. Writing events changes what a track costs and leaves a directory of the application's data on the device, and a minor version cannot decide either on an application's behalf. LDConfig.eventPersistence has three levels: disabled, which is what Tiers 1 and 2 describe and is the default; deferred, which writes off the calling thread; and immediate, which is the whole of the above. Delivery is held for the round trip rather than just its start, because a batch now stays on disk until its response is handled and a second delivery beginning inside that window would send it again under the same payload ID. Spec: Event Durability §13, §14; Page-Cache Event Persistence. Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence # Conflicts: # LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
Tier 3 stages encoded bytes into the store instead of holding events in memory, so the assertion that nothing was buffered has to ask the store.
…y/event-durability-tier3-persistence # Conflicts: # LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
…y/event-durability-tier3-persistence Also renames handWrittenEncoder to eventJSONWriter, after its type.
…y/event-durability-tier3-persistence
…y/event-durability-tier3-persistence Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # LaunchDarkly.xcodeproj/project.pbxproj
…y/event-durability-tier3-persistence Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # LaunchDarkly/LaunchDarkly/ServiceObjects/EventJSONWriter.swift # LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift # LaunchDarkly/LaunchDarklyTests/ServiceObjects/EventReporterSpec.swift
…y/event-durability-tier3-persistence Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence
…y/event-durability-tier3-persistence
…atomically; reserve capacity during encode - Without persistence, nothing but a delivery drains the held run: no threshold commit, and track, identify, flush and backgrounding no longer encode on the caller's thread. - An evaluation's summary counter and its full/debug events are recorded under pendingLock, and a commit takes both under it. - A commit reserves capacity for the run it takes, so events being encoded still count against eventCapacity. Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift
…y/event-durability-tier3-persistence Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # LaunchDarkly/LaunchDarkly/LDClient.swift # LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift # LaunchDarkly/LaunchDarklyTests/Mocks/DarklyServiceMock.swift
…y/event-durability-tier3-persistence Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # LaunchDarkly/LaunchDarkly/LDClient.swift # LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift # LaunchDarkly/LaunchDarklyTests/ServiceObjects/EventReporterSpec.swift
…y/event-durability-tier3-persistence Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # LaunchDarkly/LaunchDarkly/LDClient.swift # LaunchDarkly/LaunchDarkly/ServiceObjects/EventReporter.swift # LaunchDarkly/LaunchDarklyTests/ServiceObjects/EventReporterSpec.swift
… failed write left in memory A read can fail on a file that is intact, as it does in a process out of file descriptors, and every failure was treated as a batch that can never be delivered and deleted. Listing now skips such a batch until a later listing can read it, delivery retries it, and startup recovery closes the previous run's log off uncounted rather than deleting it. Only a file that is gone, or one written in a format this version does not read, is nothing to keep. closeBatch also closed only the log where a failed write had left events both there and in memory, so a flush could report success and close() discard the rest. Co-authored-by: Cursor <cursoragent@cursor.com>
…d not be kept A batch LaunchDarkly refuses in a way that may pass is kept and retried, so it is not a loss. What is lost is what the store never received: an event or a summary that could not be serialized when it was committed, or that serialized too large to store. A delivery that finds nothing to send cannot tell those apart from events that arrived, so flushAndWait reported success for them. Co-authored-by: Cursor <cursoragent@cursor.com>
Android pins the failed write at the processor level, where the bug was visible: the flush reports success and both the written and the held events arrive. It also pins that a body reads correctly once reads work again, and that a previous run's log moved aside leaves this run the name to append to. Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence Tier 2 replaced the delivery gate this branch was built on, so the two designs had to be reconciled rather than taken side by side. Coalescing comes from tier 2: flushes arriving before a delivery starts join it, and one that has already started is never joined, so nothing holds the drain back while a retry runs. What tier 2 did not need is a way to keep an in-flight batch from being sent twice. Tier 2 empties its buffer before publishing, so a second delivery finds nothing; here a batch stays on disk until its response arrives, which is what `isDelivering` used to guard. `batchesInFlight` takes its place. A delivery claims the batches it lists and releases them when it has an answer, so a later delivery still runs and still sends what it closed for itself, and only the batches already on their way are left out of it. A caller whose own events are all that is left unsent is answered `true`, which is the contract tier 2 documents: an answer is about the batch the caller's own delivery took. The loss flag is now taken once per delivery rather than once per caller, so everyone joined to it hears about a loss instead of only whichever was answered first, and a scheduled delivery with no waiters leaves it for the caller who is owed it. `flush` stays a commit point on both paths. This branch had also fixed the duplicate event reporter on its own, as `eventSyncLogger`; that now comes from the precursor branch instead, so the local copy goes. Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence
Functions that must run under a lock now have plain names, keep "Requires `xLock`" in their docs, and check it with UnfairLock.assertOwned(), which traps in debug builds when the calling thread does not hold the lock. Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # LaunchDarkly/LaunchDarkly/UnfairLock.swift
… several threads Co-authored-by: Cursor <cursoragent@cursor.com>
…nit event whether or not events persist Co-authored-by: Cursor <cursoragent@cursor.com>
…and say where they are The message said the events remain on disk even where persistence is off, and was at debug level only. Where the store persists it stays at debug and says they remain on disk for the next launch; otherwise it is at info and says they are only in memory and lost if the application is terminated. Co-authored-by: Cursor <cursoragent@cursor.com>
…ies, and event backgrounding into its own class Three places each had their own platform list: LDClient delivered under an assertion on iOS and tvOS, BackgroundActivity took assertions on iOS, tvOS and watchOS, and SystemCapabilities posted lifecycle notifications on iOS, tvOS and macOS. SystemCapabilities now says, per platform, whether a backgrounded process is suspended and how to get an assertion, beside the notifications. BackgroundEventDelivery owns what events do on backgrounding: commit, then deliver under an assertion where the platform suspends, and log what did not get out. LDClient and BackgroundActivity carry no platform conditions. watchOS is now stated rather than implied: it suspends and has assertions, but posts no notification the SDK observes, so nothing happens there. Co-authored-by: Cursor <cursoragent@cursor.com>
…y/event-durability-tier3-persistence Co-authored-by: Cursor <cursoragent@cursor.com> # Conflicts: # LaunchDarkly/LaunchDarklyTests/Networking/DarklyServiceSpec.swift
The accessors only tests read -- the delivery counts, pending events, the summarizer and response-date hooks, the queue occupier and the batch lock -- move into DEBUG extensions. The delivery counter itself stays in every build: it is one increment under a lock already held. Co-authored-by: Cursor <cursoragent@cursor.com>
Backgrounding commits every started environment before the notification returns and, where the process is suspended, delivers them all under one background activity assertion, released once every environment has answered. Closed clients the application still holds are no longer reached. Run mode stays per client. Co-authored-by: Cursor <cursoragent@cursor.com>
flush(completion:), flushAndWait and the background delivery each started every environment's delivery and counted answers. They now share flushReportingOutcomes, which tracks each environment, so a reporter answering twice can no longer end flushAndWait before every environment has answered. Co-authored-by: Cursor <cursoragent@cursor.com>
flushReportingOutcomes takes an optional deadline, after which the environments that have not answered count as undelivered and a late answer is ignored. The background delivery uses the assertion's 30s cap, so a delivery that never answers no longer keeps the activity held and every later backgrounding from delivering. Pins the once-only contract with never-answering and double-answering reporters. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
|
Current version of PR was reviewed by /review-bugbot with additional instructions on Nov 5, 23:10 PST. It flagged 0 findings. Bugbot on commit |
It was queued unpaced, so it sent a backlog back to back and ended the pacing of a delivery already running, competing with flag requests. Co-authored-by: Cursor <cursoragent@cursor.com>
A block on deliveryQueue cannot run below the queue's own userInitiated QoS, so asking for utility with .enforceQoS left the recovery at userInitiated. Run it on a utility global queue instead, with deliveryQueue suspended until it is done, so deliveries, the report timer and the launch gate still run after it. Co-authored-by: Cursor <cursoragent@cursor.com>
close() held instancesQueue while each environment's flushAtClose() waited up to closeCommitBudget for its final commit, so LDClient.get(), flush() and every other call reading instances waited behind those writes. Take the clients and clear instances under the queue, then close them outside it. Co-authored-by: Cursor <cursoragent@cursor.com>
close() takes the clients out of instances before closing them, so a start() on another thread could build a new client, and open the same event store, while the old one was still committing. start() now waits for any close in progress, outside instancesQueue, so get() and flush() still do not. Co-authored-by: Cursor <cursoragent@cursor.com>
| import OSLog | ||
|
|
||
| /// Posts a batch of events to LaunchDarkly once and says what the response means. Android's `AnalyticsEventSender` | ||
| /// plays the same part. |
There was a problem hiding this comment.
No need to mention Android here. This class comment is very AI.
| // Answered away from `deliveryQueue`, and from whichever thread the delivery ended on, so that every | ||
| // completion reaches its caller the same way. A completion is free to flush again: it is never running | ||
| // on the queue the delivery it asks for would be queued on. | ||
| DispatchQueue.global().async { |
There was a problem hiding this comment.
AI review comment: Medium, carried from the lock review and grown since (8 sites then, 12 now, including the give-up hop). These unqualified DispatchQueue.global() hand-offs run at .default QoS: under sustained .userInitiated load they starve the answer path, and the earlier review measured a flushAndWait going unanswered to its cap while the batch was already accepted. The one-argument fix (global(qos: .userInitiated)) applies at each site; the waiters these answer are typically blocked application threads.
| bufferLock.withLock { countedBytes < eventByteCapacity } | ||
| } | ||
|
|
||
| func admitsEvent(heldElsewhere: Int, capacity: Int) -> Bool { |
There was a problem hiding this comment.
AI review comment: Medium, spec decision; flagging so the choice is recorded as deliberate. The byte-capacity check here is ungated by persistEvents, so it binds at .disabled too. Durability 4.2/D14 require exactly that ('applies where events are held only in memory'); append-log 4.1 says the opposite ('Neither applies to a store kept off the disk... bounded by the event capacity alone'). The code (and Android, identically) implements durability's side; the two spec pages need to agree, and whichever way it lands should be stated in the eventCapacityBytes docs.
There was a problem hiding this comment.
Pinned 5 cases to DispatchQueue.global(qos: .userInitiated)
|
|
||
| /// - Parameter flushInterval: the normal ceiling; never below the initial delay, so a short interval does not make | ||
| /// a failing service be posted to more often than once a second. | ||
| convenience init(flushInterval: TimeInterval) { |
There was a problem hiding this comment.
AI review comment: Medium, spec decision. The RETRY spec requires each component's own spec to bind six parameters (normal and extended initial delays and ceilings, reset threshold, healthy operation), and append-log 9 binds none of them, so these choices are currently unadjudicable: extended base ~512s via the attempts bump (RETRY 1.5.6's minDelay promotion is not implemented), normal ceiling = max(eventFlushInterval, 1) which is user-configurable, reset on one success. Conformance item C8b cannot be tested as written. Asking for the bindings in append-log 9 turns all of this into checkable facts.
There was a problem hiding this comment.
Conformed to Retry 1.5.6
| attempts += 1 | ||
| } | ||
| let wait: TimeInterval | ||
| if let retryAfter, retryAfter >= 0 { |
There was a problem hiding this comment.
AI review comment: Medium, code plus spec. A server-directed Retry-After is jittered downward (wait = directed minus random()*(directed/2)), so a 600s directive can be honored at 300s: the SDK can post before the instant the server asked for. RETRY has an internal tension here (1.4.3 requires jitter for externally-synchronized retries; 1.11.4's rationale says server-directed values are applied verbatim), raised as a spec defect; once settled, this line should follow (verbatim, or jitter upward only).
There was a problem hiding this comment.
Made jitter upword
| /// reads. | ||
| enum EventLogFormat { | ||
| /// Bump this whenever the framing or the meaning of a frame changes. | ||
| static let version: UInt16 = 1 |
There was a problem hiding this comment.
AI review comment: Medium, test gap (both SDKs). Nothing pins the on-disk bytes: the iOS tests reference EventLogFormat.fileHeader itself (self-referential, so a drifted constant still passes) and Android's format tests are structural round-trips. For a format specified to the byte precisely so two implementations produce identical files across platforms, versions, and processes, a shared golden-byte vector (a hex literal of a known log, asserted by both suites) is cheap insurance; today a one-byte drift on either side is uncatchable.
It extends the SDK's own Crc32, not Foundation's Data. Co-authored-by: Cursor <cursoragent@cursor.com>
slowCrc32 only runs where zlib cannot be loaded, so no test reached it. Compare it with zlib's crc32 across sizes and starting values, and check that advance(withBytes:) matches advance(withChunk:). Co-authored-by: Cursor <cursoragent@cursor.com>
The hand-offs that carry a delivery on (the next paced batch, the wait between batches, the backoffs before a retry, and answering waiters) ran on an unqualified global queue, which takes the QoS of whoever submits the block. A flush from a utility thread that ended the pacing lowered the rest of the delivery to utility. Pin them to userInitiated, the delivery queue's own QoS. Co-authored-by: Cursor <cursoragent@cursor.com>
RETRY 1.5.6 makes the extended initial delay the base of the backoff once an unexpected failure is seen. Delivery reached the same first wait by raising the attempt count to at least 10 on a 1 s base, which a refusal after ten or more ordinary failures overshot. Bind the extended initial delay to 512 s, start the count at 1 on the first refusal since a reset, and floor the extended delays at the initial delay (RETRY 1.5.4.1). Co-authored-by: Cursor <cursoragent@cursor.com>
A Retry-After wait was jittered down by up to half, so an instance could post before the service asked. Jitter it up by up to half instead, stopping at the hour a server-directed wait is capped at. A wait asked for of an hour or more has no room above the cap, so it is still jittered down from the hour. Co-authored-by: Cursor <cursoragent@cursor.com>
The format tests built their expectations from EventLogFormat itself, so a drifted constant still passed. Assert the page-cache spec's Appendix A log as a hex literal: the header constant, what the store writes for its two events, what the reader makes of it, and that a next run delivers it. The Android suite asserts the same bytes. Co-authored-by: Cursor <cursoragent@cursor.com>
…without one The privacy, unserializable-event, evaluation-recording and refusal examples built a persisting store whatever the spec, so the run without persistence repeated them with a disk. They now take the spec's persistence, and the examples whose subject is the disk are left out where there is none. Co-authored-by: Cursor <cursoragent@cursor.com>
Without a disk: a delivery never encoding an evaluation's counter apart from its full event, and nothing encoded for a flush or a close while offline. Either way: a flush bypassing the backoff behind an ordinary one, what is flushed after a 400 or 413 drop, a batch an unheard delivery kept, a summary that could not be serialized, a flush joining a queued delivery, and summaries at close never outrunning their events. At .deferred: writes for a waiting flush and at close while a delivery holds the queue, and the directory worked out off the caller's thread. Co-authored-by: Cursor <cursoragent@cursor.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.
There are 3 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 71fc143. Configure here.
| reporter.isOnline = false | ||
| } | ||
| expect(runsWithCounters) > 5 | ||
| } |
There was a problem hiding this comment.
Flaky concurrent encoding split test
Medium Severity
The new test requires runsWithCounters > 5 after five trials, so it only passes when a flush splits the in-flight recordings. Recording 2000 lock-protected evaluations can finish before the first commitAndCloseBatch, leaving exactly one counted run per trial and failing even when each run’s summary still matches its feature events.
Reviewed by Cursor Bugbot for commit 71fc143. Configure here.
|
|
||
| expect(located.wait(timeout: .now() + 5)) == .success | ||
| expect(lock.withLock { onCallersThread }).toNot(contain(true)) | ||
| } |
There was a problem hiding this comment.
Directory location test races flush
Medium Severity
At .deferred, flush still commits on the caller’s thread. This example records one event and then flushes, so the first disk access — and locatingDirectory — can run on that same thread if init/online recovery has not already located the path, causing onCallersThread to contain true.
Reviewed by Cursor Bugbot for commit 71fc143. Configure here.


Third in the event-durability stack, on top of #536. Recorded events go to an on-disk log rather than an in-memory buffer, so an application that dies moments after recording one still reports it: the next run delivers what the last one left behind.
What changes
EventStoreis a new on-disk log, with capacity, framing, interrupted-log recovery, and multi-process safety.NullEventStorestands in where the platform gave the SDK nowhere to write, which is the behaviour the SDK had before.Recording no longer encodes on the caller's thread. An evaluation appends to a held run and updates a counter; a commit turns the whole run into bytes at once. Recording a custom or identify event is a commit point, because an application reporting something it chose to report is saying it matters more than the microseconds it costs. Evaluations commit once
pendingCommitThresholdof them accumulate, when a delivery starts, or on flush.LDConfig.eventPersistencelets the application choose whether a commit point runs on the caller's thread (.immediate), off it (.deferred), or not at all (.disabled).Delivery sends batches oldest first and stops at the first retryable failure, so the rest keep their place in the log instead of the SDK spending the session's requests on a service that is refusing them.
Reconciling with tier 2
Tier 2 replaced the delivery gate this branch was built on, so the merge had to pick one design rather than keep both.
Coalescing comes from tier 2: flushes arriving before a delivery starts join it, one already started is never joined, and nothing holds the drain back while a retry runs. What tier 2 did not need is a way to keep an in-flight batch from being sent twice — it empties its buffer before publishing, so a second delivery finds nothing. Here a batch stays on disk until its response arrives, which is what
isDeliveringguarded.batchesInFlighttakes its place. A delivery claims the batches it lists and releases them once it has an answer, so a later delivery still runs and still sends what it closed for itself; only the batches already on their way are left out. A caller whose own events are all that remain unsent is answeredtrue, which is exactly the contract tier 2 documents: an answer is about the batch the caller's own delivery took, never one an earlier delivery lost.The loss flag is now taken once per delivery rather than once per caller, so everyone joined to it hears about a loss instead of only whoever was answered first, and a scheduled delivery with no waiters leaves it for the caller who is owed it. This branch had also fixed the duplicate event reporter independently; that now comes from #537, so the local copy is gone.
Verification
does not send a batch again when a delivery is asked for while one is in flight.DiagnosticReporterSpec.testRetriesflaked once under load and passed on re-run; it has a 0.1 s timeout and is unrelated to these changes.Note
Overview
This PR replaces
EventReporterwith anEventProcessorpipeline (DirectEventProcessor,NullEventProcessor,EventCommitter,EventDeliveryScheduler,AnalyticsEventSender) and adds durable on-disk event storage so crashes and abrupt process exits can still be reported on the next launch.Persistence and recording: New
EventStore/DiskEventLogstack (framed logs, recovery, multi-process safety on macOS) holds committed batches until delivery succeeds.LDConfiggainseventPersistence(.disabled/.deferred/.immediate),eventCapacityBytes,eventMaxAge, andeventDeliveryDelay, with defaulteventCapacityraised to 2000. Evaluation events batch-encode off the hot path; commits happen at thresholds, flush, backgrounding, and ontrack/identifydepending on persistence mode.Delivery behavior: Batches are sent oldest-first with in-flight claims so overlapping deliveries do not double-post the same payload.
flushAndWait/flush(completion:)can bypass backoff, aggregate outcomes across environments, and treat retriable failures differently from permanently refused or unserializable events.BackgroundEventDeliverycommits all environments on background entry and attempts delivery under a shared background assertion on platforms that suspend.Client integration:
LDClientroutes all recording througheventProcessor, coordinates start vs. close viaclosingClients/claimInstances(), flushes at close withflushAtClose, and ties the diagnostic init to the event launch delay gate.Reviewed by Cursor Bugbot for commit 71fc143. Bugbot is set up for automated code reviews on this repo. Configure here.