Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve performance and usability of telemetry batching implementation #7838

Closed
akhenry opened this issue Sep 11, 2024 · 2 comments · Fixed by #7837
Closed

Improve performance and usability of telemetry batching implementation #7838

akhenry opened this issue Sep 11, 2024 · 2 comments · Fixed by #7837
Assignees
Labels
notable_change A change which should be noted in the changelog severity:critical type:enhancement verified Tested or intentionally closed

Comments

@akhenry
Copy link
Contributor

akhenry commented Sep 11, 2024

Is your feature request related to a problem? Please describe.
In real-world use of the initial implementation of telemetry batching it was found that bursty telemetry easily overran the buffers causing regular telemetry drops.

The buffering implementation should be modified to be more flexible in the face of bursty data.

Additionally, the current implementation requires plugin developers to define a server-specific "batching strategy". This presents a fairly significant impediment to adoption. A shared, rather than a per-parameter buffer removes the opportunity for a "one-size-fits-all" buffering approach.

Describe the solution you'd like
The initial implementation of real-time telemetry buffering used per-parameter buffers in an attempt to optimize for the case where a noisy parameter causing a buffer overrun squeezes out updates from lower-frequency parameters.
What was discovered in real-world testing was that telemetry did not arrive as regularly as expected, and instead would often arrive in "bursts". These bursts would easily overrun a shorter per-parameter buffer, causing regular telemetry drops.

A large shared buffer, rather than smaller per-parameter buffers is likely to offer more overhead in the face of temporary bursts of telemetry, as the buffer space is more efficiently shared between high and low frequency parameters.

@akhenry
Copy link
Contributor Author

akhenry commented Sep 11, 2024

Testing Instructions on VIPERGC-490

@akhenry akhenry added the notable_change A change which should be noted in the changelog label Sep 30, 2024
@akhenry akhenry added this to the Build 9 RC11 milestone Oct 2, 2024
@ozyx
Copy link
Contributor

ozyx commented Oct 2, 2024

Verified -- Testathon 2024-10-02

@unlikelyzero unlikelyzero added the verified Tested or intentionally closed label Oct 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
notable_change A change which should be noted in the changelog severity:critical type:enhancement verified Tested or intentionally closed
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants