Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export failure [exception] Export retry limit exceeded [previous] Idle timeout reached for "my_otltp_collector_host". #1448

Open
motoronik opened this issue Dec 10, 2024 · 1 comment

Comments

@motoronik
Copy link

Hello, I am using open telemetry of this version

  • open-telemetry/exporter-otlp: "1.0.4"
  • open-telemetry/sdk: "1.0.8"

I recently ran into a problem. My collector, which accepts span from the exporter, was unavailable for a while and therefore the application was also unavailable and a large number of requests fell with an error.
It seems to me that it is a good practice not to trigger errors in case of unavailability of the collector host, but to jam them. Usually, all systems for collecting metrics and errors behave like this.

Tell me please

  • Maybe I did not carefully study the documentation and did not find an opportunity to drown out such problems at the configuration level?
  • If there is no way to jam errors, do you plan to implement this, because if the application crashes if the metric collector is unavailable, this is a dangerous situation and forces you to abandon opentelemetry or redefine the exporter code to drown out exceptions that arise in the case of sending spans.

Will be answer, thank you

Error trace:

NOTICE: PHP message: OpenTelemetry: [error] Export failure [exception] Export retry limit exceeded [previous] Idle timeout reached for "my_otltp_collector_host".
#0 /my_project_dir/vendor/open-telemetry/exporter-otlp/SpanExporter.php(41): OpenTelemetry\SDK\Common\Export\Http\PsrTransport->send('{"resourceSpans...', NULL)
#1 /my_project_dir/vendor/open-telemetry/sdk/Trace/SpanProcessor/BatchSpanProcessor.php(255): OpenTelemetry\Contrib\Otlp\SpanExporter->export(Array)
#2 /my_project_dir/vendor/open-telemetry/sdk/Trace/SpanProcessor/BatchSpanProcessor.php(198): OpenTelemetry\SDK\Trace\SpanProcessor\BatchSpanProcessor->flush('shutdown', NULL)
#3 /my_project_dir/vendor/open-telemetry/sdk/Trace/TracerSharedState.php(98): OpenTelemetry\SDK\Trace\SpanProcessor\BatchSpanProcessor->shutdown(NULL)
#4 /my_project_dir/vendor/open-telemetry/sdk/Trace/TracerProvider.php(92): OpenTelemetry\SDK\Trace\TracerSharedState->shutdown(NULL)
#5 /my_project_dir/vendor/open-telemetry/sdk/Common/Util/functions.php(36): OpenTelemetry\SDK\Trace\TracerProvider->shutdown()
#6 [internal function]: OpenTelemetry\SDK\Trace\TracerProvider->OpenTelemetry\SDK\Common\Util\{closure}()
#7 /my_project_dir/vendor/open-telemetry/sdk/Common/Util/functions.php(50): Closure->call(Object(OpenTelemetry\SDK\Trace\TracerProvider))
#8 /my_project_dir/vendor/open-telemetry/sdk/Common/Util/ShutdownHandler.php(76): OpenTelemetry\SDK\Common\Util\{closure}()
#9 [internal function]: OpenTelemetry\SDK\Common\Util\ShutdownHandler::OpenTelemetry\SDK\Common\Util\{closure}(Array)
@brettmc
Copy link
Collaborator

brettmc commented Dec 13, 2024

I think that the exporter should log and continue. The trace you've provided looks like the error being logged, and you can control where it gets logged to if that helps, or turn off logging entirely (see OTEL_PHP_LOG_DESTINATION)

If you're using a shared-nothing webserver setup, then I suspect that your problem was that because each request was timing out, the workers were not being freed up to service the next request, and you ran out of available workers/slots/whatever? That's unfortunately a symptom of the shared-nothing approach of those setups.

Recently, the spec has the addition of a file-based exporter. We haven't implemented it yet, but I wonder if you could combine it with the collector's filelogreceiver to give you that protection?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants