Repository navigation
Conversation
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Problem
ActiveRecordLogSubscriber#sqlappends every query toLograge::Sql.store[:lograge_sql_queries], in any thread. The list is only drained when Lograge logs a request. In a thread that never serves a request, it grows for the life of the process.Some examples of such threads:
connect,subscribed, ...).This is the same leak as #59. The Solid Cable denylist added in #60 covers one source of it; any other query source outside a request still leaks.
In our case, a heap dump of a Sidekiq process that had been running for about 3 hours held 100k entries in a single worker thread's store. Memory grew by about 150 MB/h until the pod was OOM-killed.
Fix
Lograge::Sql.store_active?reports whether events should be stored right now. The subscriber skips the event when it returns false.RequestStoreis loaded (Lograge depends on it), the method returnsRequestStore.active?. That is true only betweenRequestStore::Middleware'sbegin!andend!, which is exactly the span of a request that Lograge will log and drain.RequestStoreis not loaded, or is too old to haveactive?, the behaviour doesn't change.Requests are logged exactly as before.
Specs