fix(tap-agent): eagerly initialize metrics at startup #888
+422
−19
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Prometheus metrics in tap-agent are defined using LazyLock,
which means
they are only registered with the Prometheus registry when
first
accessed. If no senders have escrow accounts or pending
allocations at
startup, no SenderAccount actors are spawned, the metrics are
never
accessed, and /metrics returns empty.
This fix creates init_metrics() functions that force all
LazyLock metric
statics to initialize at startup, ensuring they are
registered with
Prometheus before any sender activity occurs.
Changes:
- Make all metric static definitions pub(crate) in
sender_account.rs,
sender_allocation.rs, and sender_accounts_manager.rs
- Add init_metrics() function to each module that
dereferences each
LazyLock to force initialization
- Add public init_metrics() to agent.rs that calls all
module-level
init_metrics functions
- Call agent::init_metrics() in main.rs before spawning the
metrics
server
- Add test to verify all metrics are properly registered
Clean up sender and allocation-level gauge metrics when
actors stop to
prevent stale values from accumulating in Prometheus.
Changes:
- Clean up UNAGGREGATED_FEES_BY_VERSION gauge on allocation
stop
- Clean up INVALID_RECEIPT_FEES gauge on allocation stop
- Clean up sender-level gauges (SENDER_DENIED,
ESCROW_BALANCE,
SENDER_FEE_TRACKER, MAX_FEE_PER_SENDER,
RAV_REQUEST_TRIGGER_VALUE)
in SenderAccount post_stop
- Add tests to verify metric cleanup behavior
suchapalaver
changed the title
Pull Request Test Coverage Report for Build 19990119371Details
💛 - Coveralls |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR improves Prometheus metrics handling in tap-agent with two fixes:
Joseph Livesey joseph@semiotic.ai