Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent collected exceptions from being dropped due to null event values #7301

Conversation

schmittjoseph
Copy link
Member

@schmittjoseph schmittjoseph commented Sep 10, 2024

Summary

We have a bug in our in-proc exception collection event source code that results in malformed event payloads whenever an exception doesn't have an activity id. This results in the event pipeline processing the exceptions to be unable to parse the data and drop exceptions.

Below describes the issue, covering what the in-proc event source code is doing and then our pipeline.

In-proc flow:

  • Our in-proc code used nullable strings for ActivityId and ExceptionMessage (side note: ExceptionMessage could never actually be null) when declaring the ExceptionInstance event.
  • When either of these had a null value, we'd instantiate a PinnedData with a null value.
  • When event source goes to serialize this, it doesn't send anything for this field since it's a 0 length payload.

Event pipeline flow:

  • When attempting to parse an ExceptionInstance event, we call traceEvent.GetPayload<string>(ExceptionEvents.ExceptionInstancePayloads.{ExceptionMessage,ActivityId}).
  • When the value of this field is null on the in-proc side of things, a 0 length payload was used for the field. However since strings are variable length, TraceEvent needs to calculate the length of the string. This is done by searching for a null terminator in the message payload starting at the field offset.
  • No null terminator is sent from our in-proc code when the string is null. This means TraceEvent will read other fields payloads until it finds what it thinks is a null terminator, mangling the offset and often times entering a faulted state.

The fix:

  • On first glance it might seem like we want to update our traceEvent.GetPayload call to pass in a nullable string (string?). However upon testing this has no impact on TraceEvent's understanding of the fields. It still treats the field as a non-nullable string and expects a null-terminator. There does not appear to be an affordance for TraceEvent to detect that the string has a null value since it's dynamic length.
  • Instead we must send empty strings instead of null values. Luckily all of our code in dotnet-monitor gracefully handles either null or empty strings for an exception's ActivityId so we don't have to update that.
Release Notes Entry

Fixed an issue that could sometimes result in exceptions not being reported in exception history.

@schmittjoseph schmittjoseph requested a review from a team as a code owner September 10, 2024 18:02
@schmittjoseph schmittjoseph changed the title Prevent exception event source events being malformed from null values Prevent collected exceptions from being dropped due to null event values Sep 10, 2024
@schmittjoseph schmittjoseph added the update-release-notes Pull requests that should be mentioned in the release notes label Sep 10, 2024
@schmittjoseph schmittjoseph merged commit 08eed9b into dotnet:main Sep 10, 2024
26 checks passed
@schmittjoseph
Copy link
Member Author

/backport to release/8.0

Copy link
Contributor

Started backporting to release/8.0: https://github.com/dotnet/dotnet-monitor/actions/runs/10800213525

schmittjoseph added a commit that referenced this pull request Sep 11, 2024
schmittjoseph added a commit that referenced this pull request Sep 12, 2024
* Update dependencies from https://github.com/dotnet/diagnostics build 20240906.1 (#7285)

[main] Update dependencies from dotnet/diagnostics

* Add Docker Compose sample and cross link (#7276)

* Update public Linux build machines to Ubuntu 22.04 (#7283)

* Update release tool to .NET 8 and update dependencies (#7284)

* [main] Bump Swashbuckle.AspNetCore in /eng/dependabot/independent (#7288)

Bumps [Swashbuckle.AspNetCore](https://github.com/domaindrivendev/Swashbuckle.AspNetCore) from 6.7.1 to 6.7.3.
- [Release notes](https://github.com/domaindrivendev/Swashbuckle.AspNetCore/releases)
- [Commits](domaindrivendev/Swashbuckle.AspNetCore@v6.7.1...v6.7.3)

---
updated-dependencies:
- dependency-name: Swashbuckle.AspNetCore
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* generate release notes (#7294)

* Update dependencies from https://github.com/dotnet/diagnostics build 20240909.1 (#7295)

[main] Update dependencies from dotnet/diagnostics

* Update dependencies from https://github.com/dotnet/arcade build 20240909.4 (#7296)

[main] Update dependencies from dotnet/arcade

* Prevent collected exceptions from being dropped due to null event values (#7301)

* Register v9.0.0-rc.1.24453.3 release information (#7303)

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

* Update dependencies from https://github.com/dotnet/diagnostics build 20240910.1 (#7304)

[main] Update dependencies from dotnet/diagnostics

* Update dependencies from https://github.com/dotnet/arcade build 20240910.4 (#7305)

[main] Update dependencies from dotnet/arcade

* Restore branch-specific files

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dotnet-maestro[bot] <42748379+dotnet-maestro[bot]@users.noreply.github.com>
Co-authored-by: Justin Anderson <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Joe Schmitt <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
update-release-notes Pull requests that should be mentioned in the release notes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants