Unit test to reproduce data race when reloading TLS config

[chahatsagarmain]

Which problem is this PR solving?

• Part of [Bug]: race condition when reloading TLS certificates #4316

Description of the changes

• Added unit to test to check for data race

How was this change tested?

Checklist

• I have read https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
• I have signed all commits
• I have added unit tests for the new functionality
• I have run lint and test steps successfully
  • for jaeger: make lint test
  • for jaeger-ui: yarn lint and yarn test

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 48.76%. Comparing base (2b7cf3a) to head (ee64cb4).
Report is 11 commits behind head on main.

❗ There is a different number of reports uploaded between BASE (2b7cf3a) and HEAD (ee64cb4). Click for more details.

HEAD has 1 upload less than BASE

Flag	BASE (2b7cf3a)	HEAD (ee64cb4)
unittests	1	0

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #6213       +/-   ##
===========================================
- Coverage   96.50%   48.76%   -47.75%     
===========================================
  Files         354      179      -175     
  Lines       20127    10803     -9324     
===========================================
- Hits        19424     5268    -14156     
- Misses        520     5092     +4572     
- Partials      183      443      +260     

Flag	Coverage Δ
badger_v1	8.31% <ø> (ø)
badger_v2	1.67% <ø> (-0.01%)	⬇️
cassandra-4.x-v1	14.39% <ø> (ø)
cassandra-4.x-v2	1.61% <ø> (-0.01%)	⬇️
cassandra-5.x-v1	14.39% <ø> (ø)
cassandra-5.x-v2	1.61% <ø> (-0.01%)	⬇️
elasticsearch-6.x-v1	18.59% <ø> (ø)
elasticsearch-7.x-v1	18.68% <ø> (ø)
elasticsearch-8.x-v1	18.85% <ø> (ø)
elasticsearch-8.x-v2	1.66% <ø> (-0.02%)	⬇️
grpc_v1	9.44% <ø> (-0.04%)	⬇️
grpc_v2	6.97% <ø> (-0.04%)	⬇️
kafka-v1	8.88% <ø> (ø)
kafka-v2	1.67% <ø> (-0.01%)	⬇️
memory_v2	1.67% <ø> (+<0.01%)	⬆️
opensearch-1.x-v1	18.73% <ø> (ø)
opensearch-2.x-v1	18.72% <ø> (-0.01%)	⬇️
opensearch-2.x-v2	1.67% <ø> (-0.01%)	⬇️
tailsampling-processor	0.46% <ø> (-0.01%)	⬇️
unittests	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

---

🚨 Try these New Features:

• Flaky Tests Detection - Detect and resolve failed and flaky tests
• JS Bundle Analysis - Avoid shipping oversized bundles

[yurishkuro]

Have you observe this new test fail? It doesn't look like it's testing race condition deterministically.

[chahatsagarmain]

Have you observe this new test fail? It doesn't look like it's testing race condition deterministically.

In the previous test, the race condition occurred because the mutex was not used. Adding the mutex resolved the issue. However, in the updated test, new TLS configurations are being applied and its seems to be causing data race during client connection and TLS handshake error .

[yurishkuro]

so I see this test is failing on detected race condition, but the stack traces indicate that the race is caused by the test itself (the Write part), not by the production code:

WARNING: DATA RACE
Write at 0x00c000138400 by goroutine 163:
  github.com/jaegertracing/jaeger/pkg/config/tlscfg.TestCertificateRaceCondition.func4()
      /home/runner/work/jaeger/jaeger/pkg/config/tlscfg/options_test.go:392 +0xdc

Previous read at 0x00c000[138](https://github.com/jaegertracing/jaeger/actions/runs/11900851926/job/33162577763?pr=6213#step:8:139)400 by goroutine 162:
  github.com/jaegertracing/jaeger/pkg/config/tlscfg.(*Options).Config()
      /home/runner/work/jaeger/jaeger/pkg/config/tlscfg/options.go:75 +0x676

[yurishkuro]

@chahatsagarmain an alternative to fixing the race condition from reloading is to eliminate the use of this package altogether. We already migrated several endpoints to use OTEL helpers which internally handle TLS reloading differently (on a timeout rather than on file change). It would be interesting to see which parts of Jaeger still use the tlscfg package and switch to OTEL helpers.

[chahatsagarmain]

@yurishkuro So i can use configtls from OTEL and replace the usage of the tlscfg package ?
Also , there is usage of tlscfg in collector and agent and mostly test files .

[yurishkuro]

yes, that would be good. You can start small, e.g. can we remove tlscfg dependency from cmd/es-rollover?