[Prototype] OpenTelemetry and Tokio Tracing bridge that properly activates contexts and spans

[bantonsson]

This is a follow up ticket to discussions in #1571 about OpenTelemetry Tracing API vs Tokio-Tracing API.

The aim is to prototype a Tokio Tracing subscriber bridge with OpenTelemetry tracing/logging that will use the overlapping context scopes fix as a basis and keep the OpenTelemetry Context up to date with the Tokio Tracing notion of the active Span. This bridge will activate/deactivate the OpenTelemetry Context and Span on the threads at the proper points instead of only briefly activating the OpenTelemetry Context or Span when the Tokio Tracing Span is finished. More concretely there will be a stack that mimics the stack in Tokio Tracing that deactivates the right OpenTelemetry Context corresponding to the Tokio Tracing Span.

This will not (at least not initially) provide seamless two-way interoperability as described in Option 3 here, but instead allow the OpenTelemetry Context to travel with Tokio Tracing Spans, so that standard OpenTelemetry API calls will find the right Context and Span

The goal is to make code like this broken example work seamlessly from the OpenTelemetry point of view, allowing end user code to use OpenTelemetry API calls that will work with frameworks and libraries that are using Tokio Tracing.

I would love to get feedback on the idea and hear about other peoples experience in trying to fix the interoperability.

[cijothomas]

From CommunityMeeting on Dec 17th:

The key idea is to use tokio::tracing::subscriber to activate/deactivate OTel Context. #2378 is a requirement, but okay to manually test with the changes brought in by hand, so not a blocker for prototype.

#2438 can be still offered experimental, as this issue will fix the correlation without requiring the change from #2438