API Proposal: Support for OpenTelemetry Events

[samsp-msft]

[Issue writing in progress]

Add support for OpenTelemetry Events

Events is a newish spec from OpenTelemetry to describe events. The goal is to replace the events on Spans, and be able to use them for client-side events (such as a tracking button clicks), gen_ai telemetry etc.

Events are a specialization of log messages, and have a different schema:

• Events all have a name
• Events don't have a format string
• Event data should be structured data as part of the Body which should be transmitted on the wire using AnyValue.

AnyValue is a distributed union type defined in the gRPC protos for OTLP. It has a lot of commonality with what can be used in JSON - the main difference is more options for numbers.

How Events should be exposed to .NET

• There should be new extensions to ILogger to emit events
• Should be able to supply the body as types wrapable in an AnyValue
  • bool, int, long, double, string, byte[]
  • array of supported types
  • dictionary<string, object containing wrappable types>
  • AnyValue
  • JSON?

Proposals

New Library: Microsoft.Extensions.OpenTelemetry

Contains:

AnyValue Definition

We could potentially defer the AnyValue support to the OTel library and have a type checker on what is passed in, but I think we should probably embrace AnyValue and enable its use in other places such as tags on Activity, that according to OTLP can have AnyValue members.

We should use the affinity to JSON to override the ToString() functionality to return JSON formatted data when emitting AnyValue data in places that are not aware of it.

Extension to ILogger for Event API

ILogger.LogEvent(string eventName, string body, IList<<IReadOnlyList<KeyValuePair<string, object?>> tags=null)

With other overloads for encouraging the body to be implemented using AnyValue compatible types

Strongly typed Event method generation

Logging has the LoggerMessage attribute to create optimized logging methods to write standard log messages. Events needs to have the same kind of support, so you can create rich events.

One of the differences is that you should be able to define a range of parameters and map those into dictionary members in the body. The keys for dictionary members can include invalid characters for member names, primarily .. There should be a way to attribute the parameters to map them to the names in the dictionary.

ILogger TState

ILogger uses a struct FormattedLogValues to store the log message data. It may be possible to use this for Events, or implement an alternative type to store the data. As its typical to convert values to strings, we can use the ability to render an AnyValue as JSON as the way to enable downlevel support.

[RussKie]

@samsp-msft what's the right area label for this issue?

[stephentoub]

Why would this need to be a new library rather than adding to one of the existing ones?

[samsp-msft]

Why would this need to be a new library rather than adding to one of the existing ones?

Events are functionality that is defined by OpenTelemetry and while event data will come through ILogger, their output is not going to be nearly as understandable if the collector is not aware of them. As its specific to OTel, I am not sure we want to put it in the ILogger libs, nor require consumers to pickup the R9 extension lib.

Anyvalue is a OTel defined distributed union. We need a definition that can be used by the Events code, that can then be referenced by the OTel libraries. I don't think we can take a dependency in .NET to OTel, which is on a completely different schedule, ownership etc.

[samsp-msft]

@lmolkova, @CodeBlanch, @rajkumar-rangaraj, @cijothomas, @reyang, @tarekgh

[cijothomas]

OTel defines Events as Logs with EventName.
ILogger already supports emitting logs with EventName.

ILogger does not restrict the types of attributes (its objects). It is upto providers like OpenTelemetry to support more complex attributes.

I don't think this it is a good idea to introduce a totally new thing, just to support Events, which looks to be already supported by ILogger. If there are limitations, then is it possible to look at fixing them in the ILogger itself?

(I am no longer a maintainer for OTel .NET, so my comments may not be based on most up-to-date information!
However, this is an issue I am seeing in OTel Rust too - Rust's Logging API already supports EventName (just like ILogger). It also suffers the same limitation as ILogger that it does not support complex attributes natively. We are hoping to address it in existing library itself without introducing any new package/library.)

[lmolkova]

From OTel Logs/Events side we're currently in process of updating a few things:

• what goes to attributes vs body - Logs/Events guidance on what to put to attributes vs body open-telemetry/semantic-conventions#1651. In the past ~3 months we've reached the consensus that we'd rather leverage attributes for all OTel semantic-conventions things.
• whether we support complex attribute types - the consensus is yes

We'll migrate existing events to use attributes instead of the body. The body use-case remains unclear. Given that this work is not complete, there is a small(?) chance we'll have to come back to the body, especially for large/binary/opaque things not defined by OTel.

---

How I think it applies to .NET:

ILogger.Log and variations supports parameters of arbitrary types, so no ILogger API change is necessary. An existing example like FoodPriceChanged in otel dotnet almost works.

What's still missing:

• No body/no formatter. OTel Events in common case won't have any body and won't need a formatter or string description at all - I hope compile-time logging source generators can support this option better.
• Complex parameter types. We'll need to figure out the contract for OTel-dotnet ILogger -> LogRecord conversion to support complex types. I think it calls ToString on everything it does not recognize.

The latter is where the AnyValue support would be helpful. Without it, OTel would need to explore the parameter and convert it to AnyValue. It's likely to be easy, but not performant.

Given that things are in flux around the body and it probably won't be necessary, I think we should focus on the attribute/tag/parameter type.

Here we can either:

• contain the object -> OTel AnyValue conversion inside OTel-dotnet
• invent AnyValue as an optimization

[samsp-msft]

From OTel Logs/Events side we're currently in process of updating a few things:

• what goes to attributes vs body - Logs/Events guidance on what to put to attributes vs body open-telemetry/semantic-conventions#1651. In the past ~3 months we've reached the consensus that we'd rather leverage attributes for all OTel semantic-conventions things.
• whether we support complex attribute types - the consensus is yes

We'll migrate existing events to use attributes instead of the body. The body use-case remains unclear. Given that this work is not complete, there is a small(?) chance we'll have to come back to the body, especially for large/binary/opaque things not defined by OTel.

How I think it applies to .NET:

ILogger.Log and variations supports parameters of arbitrary types, so no ILogger API change is necessary. An existing example like FoodPriceChanged in otel dotnet almost works.

The lack of body which is a key part of the logging concept feels kind of broken when using the existing APIs. What you have to do today is kind of ugly. Eg

extensions/src/Libraries/Microsoft.Extensions.AI/ChatCompletion/OpenTelemetryChatClient.cs

Line 497 in 2f973c9

private void Log(EventId id, [StringSyntax(StringSyntaxAttribute.Json)] string eventBodyJson)

What's still missing:

• No body/no formatter. OTel Events in common case won't have any body and won't need a formatter or string description at all - I hope compile-time logging source generators can support this option better.
• Complex parameter types. We'll need to figure out the contract for OTel-dotnet ILogger -> LogRecord conversion to support complex types. I think it calls ToString on everything it does not recognize.

The latter is where the AnyValue support would be helpful. Without it, OTel would need to explore the parameter and convert it to AnyValue. It's likely to be easy, but not performant.

This is where having more knowledge of the types that can be passed into AnyValue comes into play. Ideally when you pass in values into the APIs that take an object, it will validate those against what is packable into an AnyValue, so you get an error in the code that emits the log message, rather than later when OTEL is trying to emit it.

Given that things are in flux around the body and it probably won't be necessary, I think we should focus on the attribute/tag/parameter type.

Here we can either:

• contain the object -> OTel AnyValue conversion inside OTel-dotnet
• invent AnyValue as an optimization

Having AnyValue support "in the extensions" so that apps can formulate their richer bodies using AnyValue I think is going to be a more robust developer experience than stuffing values into object typed variables and hoping for the best that it can be pushed over the wire.