-
Notifications
You must be signed in to change notification settings - Fork 332
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make sure OpenCensusTracer from tracing package can be correctly shut down (flushed) #2475
Comments
This looks like super easy to fix. I think we should force change the SetupStaticPublishing or SetupDynamicPublishing funcs, so they will return the Such incompatible change will enforce all Knative components to align, and in the process fix this issue, by calling |
This issue is stale because it has been open for 90 days with no |
/reopen |
The issue knative/pkg#2475 has been resolved. It's not necessary to send multiple events anymore. Each event should be exported to Zipking because of the new fix for tracer shutdown.
The issue knative/pkg#2475 has been resolved. It's not necessary to send multiple events anymore. Each event should be exported to Zipking because of the new fix for tracer shutdown.
The issue knative/pkg#2475 has been resolved. It's not necessary to send multiple events anymore. Each event should be exported to Zipking because of the new fix for tracer shutdown.
The issue knative/pkg#2475 has been resolved. It's not necessary to send multiple events anymore. Each event should be exported to Zipking because of the new fix for tracer shutdown.
The issue knative/pkg#2475 has been resolved. It's not necessary to send multiple events anymore. Each event should be exported to Zipking because of the new fix for tracer shutdown.
* Do not call deprecated eventshub.AddTracing * Remove workaround for incorrect tracing shutdown The issue knative/pkg#2475 has been resolved. It's not necessary to send multiple events anymore. Each event should be exported to Zipking because of the new fix for tracer shutdown. * Update reconciler-test dependency
The problem is that currently the components that use SetupStaticPublishing or SetupDynamicPublishing can drop some Spans/Traces when being shut down or evicted (e.g. during upgrades).
The goal is to prevent dropping buffered traces before they're sent as indicated by upstream issue in OpenCensus.
The OpenCensusTracer has a Finish function that can be called to properly Close the OpenZipkin http reporter and flush buffered traces. However, the SetupStaticPublishing/SetupDynamicPublishing functions don't make the OpenCensusTracer available so that anyone can call the Finish function. Another problem is that the Finish function passes nil which triggers nil pointer exception when calling the Finish function.
I've verified that calling the Finish function on the returned OpenCensusTracer sends all remaining traces and makes them available in Zipkin backend. It is only required to pass
&config.Config{}
on this line instead of nil.It would be good to enforce calling Finish before components are shutdown as long as there's no sleep time in that component by default before it is shut down or evicted (I think some components have the sleep time such as Knative Serving Activator).
Related discussion here and here
Expected Behavior
It is possible to flush traces on demand (possibly before shutting down a component).
Actual Behavior
Traces that were buffered before the component is shutdown are not properly sent to tracing backend.
Additional Info
The text was updated successfully, but these errors were encountered: