Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[core] TSAN failure for GcsHealthCheckManagerTest #49469

Closed
dentiny opened this issue Dec 28, 2024 · 0 comments
Closed

[core] TSAN failure for GcsHealthCheckManagerTest #49469

dentiny opened this issue Dec 28, 2024 · 0 comments
Assignees
@dentiny
Labels
bug Something that is supposed to be working; but isn't core Issues that should be addressed in Ray Core

Comments

@dentiny
Copy link
Contributor

dentiny commented Dec 28, 2024

What happened + What you expected to happen


[2024-12-27T20:44:08Z] ==================
--
  | [2024-12-27T20:44:08Z] WARNING: ThreadSanitizer: data race (pid=1309)
  | [2024-12-27T20:44:08Z]   Write of size 8 at 0x7b2400001b00 by main thread:
  | [2024-12-27T20:44:08Z]     #0 free /tmp/llvm/utils/release/final/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:707:3 (gcs_health_check_manager_test+0x9c958)
  | [2024-12-27T20:44:08Z]     #1 std::__1::default_delete<ray::gcs::GcsHealthCheckManager>::operator()(ray::gcs::GcsHealthCheckManager*) const /opt/llvm/bin/../include/c++/v1/memory:1423:5 (gcs_health_check_manager_test+0x1105e4)
  | [2024-12-27T20:44:08Z]     #2 std::__1::unique_ptr<ray::gcs::GcsHealthCheckManager, std::__1::default_delete<ray::gcs::GcsHealthCheckManager> >::reset(ray::gcs::GcsHealthCheckManager*) /opt/llvm/bin/../include/c++/v1/memory:1684:7 (gcs_health_check_manager_test+0x1105e4)
  | [2024-12-27T20:44:08Z]     #3 std::__1::unique_ptr<ray::gcs::GcsHealthCheckManager, std::__1::default_delete<ray::gcs::GcsHealthCheckManager> >::~unique_ptr() /opt/llvm/bin/../include/c++/v1/memory:1638:19 (gcs_health_check_manager_test+0x1105e4)
  | [2024-12-27T20:44:08Z]     #4 GcsHealthCheckManagerTest::~GcsHealthCheckManagerTest() /proc/self/cwd/src/ray/gcs/gcs_server/test/gcs_health_check_manager_test.cc:53:7 (gcs_health_check_manager_test+0x1105e4)
  | [2024-12-27T20:44:08Z]     #5 GcsHealthCheckManagerTest_MarkHealthAndSkipCheck_Test::~GcsHealthCheckManagerTest_MarkHealthAndSkipCheck_Test() /proc/self/cwd/src/ray/gcs/gcs_server/test/gcs_health_check_manager_test.cc:165:1 (gcs_health_check_manager_test+0x110429)
  | [2024-12-27T20:44:08Z]     #6 testing::Test::DeleteSelf_() /proc/self/cwd/external/com_google_googletest/googletest/include/gtest/gtest.h:336:24 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xd67a2)
  | [2024-12-27T20:44:08Z]     #7 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2612:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xd4a1f)
  | [2024-12-27T20:44:08Z]     #8 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2648:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xd4a1f)
  | [2024-12-27T20:44:08Z]     #9 testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2842:5 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xd6442)
  | [2024-12-27T20:44:08Z]     #10 testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:3015:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xd79e4)
  | [2024-12-27T20:44:08Z]     #11 testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5920:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xecc44)
  | [2024-12-27T20:44:08Z]     #12 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2612:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xec05f)
  | [2024-12-27T20:44:08Z]     #13 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2648:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xec05f)
  | [2024-12-27T20:44:08Z]     #14 testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5484:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xebe4c)
  | [2024-12-27T20:44:08Z]     #15 RUN_ALL_TESTS() /proc/self/cwd/external/com_google_googletest/googletest/include/gtest/gtest.h:2317:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x60df5)
  | [2024-12-27T20:44:08Z]     #16 main /proc/self/cwd/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x60df5)
  | [2024-12-27T20:44:08Z]
  | [2024-12-27T20:44:08Z]   Previous read of size 8 at 0x7b2400001b00 by thread T30:
  | [2024-12-27T20:44:08Z]     #0 ray::gcs::GcsHealthCheckManager::HealthCheckContext::StartHealthCheck()::$_1::operator()(grpc::Status) const /proc/self/cwd/src/ray/gcs/gcs_server/gcs_health_check_manager.cc:139:19 (liblibgcs_Userver_Ulib.so+0x510cc2)
  | [2024-12-27T20:44:08Z]     #1 decltype(std::__1::forward<ray::gcs::GcsHealthCheckManager::HealthCheckContext::StartHealthCheck()::$_1&>(fp)(std::__1::forward<grpc::Status>(fp0))) std::__1::__invoke<ray::gcs::GcsHealthCheckManager::HealthCheckContext::StartHealthCheck()::$_1&, grpc::Status>(ray::gcs::GcsHealthCheckManager::HealthCheckContext::StartHealthCheck()::$_1&, grpc::Status&&) /opt/llvm/bin/../include/c++/v1/type_traits:3694:1 (liblibgcs_Userver_Ulib.so+0x510cc2)
  | [2024-12-27T20:44:08Z]     #2 void std::__1::__invoke_void_return_wrapper<void, true>::__call<ray::gcs::GcsHealthCheckManager::HealthCheckContext::StartHealthCheck()::$_1&, grpc::Status>(ray::gcs::GcsHealthCheckManager::HealthCheckContext::StartHealthCheck()::$_1&, grpc::Status&&) /opt/llvm/bin/../include/c++/v1/__functional_base:348:9 (liblibgcs_Userver_Ulib.so+0x510cc2)
  | [2024-12-27T20:44:08Z]     #3 std::__1::__function::__alloc_func<ray::gcs::GcsHealthCheckManager::HealthCheckContext::StartHealthCheck()::$_1, std::__1::allocator<ray::gcs::GcsHealthCheckManager::HealthCheckContext::StartHealthCheck()::$_1>, void (grpc::Status)>::operator()(grpc::Status&&) /opt/llvm/bin/../include/c++/v1/functional:1558:16 (liblibgcs_Userver_Ulib.so+0x510cc2)
  | [2024-12-27T20:44:08Z]     #4 std::__1::__function::__func<ray::gcs::GcsHealthCheckManager::HealthCheckContext::StartHealthCheck()::$_1, std::__1::allocator<ray::gcs::GcsHealthCheckManager::HealthCheckContext::StartHealthCheck()::$_1>, void (grpc::Status)>::operator()(grpc::Status&&) /opt/llvm/bin/../include/c++/v1/functional:1732:12 (liblibgcs_Userver_Ulib.so+0x510cc2)
  | [2024-12-27T20:44:08Z]     #5 std::__1::__function::__value_func<void (grpc::Status)>::operator()(grpc::Status&&) const /opt/llvm/bin/../include/c++/v1/functional:1885:16 (liblibnode_Umanager_Ucc_Ugrpc.so+0x8a597)
  | [2024-12-27T20:44:08Z]     #6 std::__1::function<void (grpc::Status)>::operator()(grpc::Status) const /opt/llvm/bin/../include/c++/v1/functional:2560:12 (liblibnode_Umanager_Ucc_Ugrpc.so+0x8a597)
  | [2024-12-27T20:44:08Z]     #7 void grpc::internal::CatchingCallback<std::__1::function<void (grpc::Status)>, grpc::Status>(std::__1::function<void (grpc::Status)>&&, grpc::Status&&) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/support/callback_common.h:43:5 (liblibnode_Umanager_Ucc_Ugrpc.so+0x8a597)
  | [2024-12-27T20:44:08Z]     #8 grpc::internal::CallbackWithStatusTag::Run(bool) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/support/callback_common.h:128:5 (liblibnode_Umanager_Ucc_Ugrpc.so+0x8a3cf)
  | [2024-12-27T20:44:08Z]     #9 grpc::internal::CallbackWithStatusTag::StaticRun(grpc_completion_queue_functor*, int) /proc/self/cwd/external/com_github_grpc_grpc/include/grpcpp/support/callback_common.h:112:46 (liblibnode_Umanager_Ucc_Ugrpc.so+0x8a0d4)
  | [2024-12-27T20:44:08Z]     #10 grpc::(anonymous namespace)::CallbackAlternativeCQ::Ref()::'lambda'(void*)::operator()(void*) const /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/common/completion_queue_cc.cc:94:17 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xd13df)
  | [2024-12-27T20:44:08Z]     #11 grpc::(anonymous namespace)::CallbackAlternativeCQ::Ref()::'lambda'(void*)::__invoke(void*) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/common/completion_queue_cc.cc:62:13 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xd13df)
  | [2024-12-27T20:44:08Z]     #12 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::operator()(void*) const /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gprpp/posix/thd.cc:145:11 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgpr.so+0x2c46a)
  | [2024-12-27T20:44:08Z]     #13 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&)::'lambda'(void*)::__invoke(void*) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gprpp/posix/thd.cc:115:9 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgpr.so+0x2c46a)
  | [2024-12-27T20:44:08Z]
  | [2024-12-27T20:44:08Z]   Thread T30 'nexting_thread' (tid=1347, running) created by main thread at:
  | [2024-12-27T20:44:08Z]     #0 pthread_create /tmp/llvm/utils/release/final/llvm-project/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:965:3 (gcs_health_check_manager_test+0x9db2b)
  | [2024-12-27T20:44:08Z]     #1 grpc_core::(anonymous namespace)::ThreadInternalsPosix::ThreadInternalsPosix(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gprpp/posix/thd.cc:113:30 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgpr.so+0x2bf36)
  | [2024-12-27T20:44:08Z]     #2 grpc_core::Thread::Thread(char const*, void (*)(void*), void*, bool*, grpc_core::Thread::Options const&) /proc/self/cwd/external/com_github_grpc_grpc/src/core/lib/gprpp/posix/thd.cc:199:15 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgpr.so+0x2bf36)
  | [2024-12-27T20:44:08Z]     #3 void std::__1::allocator<grpc_core::Thread>::construct<grpc_core::Thread, char const (&) [15], grpc::(anonymous namespace)::CallbackAlternativeCQ::Ref()::'lambda'(void*), grpc::CompletionQueue*&>(grpc_core::Thread*, char const (&) [15], grpc::(anonymous namespace)::CallbackAlternativeCQ::Ref()::'lambda'(void*)&&, grpc::CompletionQueue*&) /opt/llvm/bin/../include/c++/v1/memory:911:28 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xd0c7a)
  | [2024-12-27T20:44:08Z]     #4 void std::__1::allocator_traits<std::__1::allocator<grpc_core::Thread> >::construct<grpc_core::Thread, char const (&) [15], grpc::(anonymous namespace)::CallbackAlternativeCQ::Ref()::'lambda'(void*), grpc::CompletionQueue*&, void>(std::__1::allocator<grpc_core::Thread>&, grpc_core::Thread*, char const (&) [15], grpc::(anonymous namespace)::CallbackAlternativeCQ::Ref()::'lambda'(void*)&&, grpc::CompletionQueue*&) /opt/llvm/bin/../include/c++/v1/__memory/allocator_traits.h:288:13 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xd0c7a)
  | [2024-12-27T20:44:08Z]     #5 void std::__1::vector<grpc_core::Thread, std::__1::allocator<grpc_core::Thread> >::__emplace_back_slow_path<char const (&) [15], grpc::(anonymous namespace)::CallbackAlternativeCQ::Ref()::'lambda'(void*), grpc::CompletionQueue*&>(char const (&) [15], grpc::(anonymous namespace)::CallbackAlternativeCQ::Ref()::'lambda'(void*)&&, grpc::CompletionQueue*&) /opt/llvm/bin/../include/c++/v1/vector:1667:5 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xd0c7a)
  | [2024-12-27T20:44:08Z]     #6 grpc_core::Thread& std::__1::vector<grpc_core::Thread, std::__1::allocator<grpc_core::Thread> >::emplace_back<char const (&) [15], grpc::(anonymous namespace)::CallbackAlternativeCQ::Ref()::'lambda'(void*), grpc::CompletionQueue*&>(char const (&) [15], grpc::(anonymous namespace)::CallbackAlternativeCQ::Ref()::'lambda'(void*)&&, grpc::CompletionQueue*&) /opt/llvm/bin/../include/c++/v1/vector:1687:9 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xd0c7a)
  | [2024-12-27T20:44:08Z]     #7 grpc::(anonymous namespace)::CallbackAlternativeCQ::Ref() /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/common/completion_queue_cc.cc:60:26 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xd0c7a)
  | [2024-12-27T20:44:08Z]     #8 grpc::CompletionQueue::CallbackAlternativeCQ() /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/common/completion_queue_cc.cc:194:36 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xd0c7a)
  | [2024-12-27T20:44:08Z]     #9 grpc::Server::CallbackCQ() /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/server/server_cc.cc:1386:19 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xf1112)
  | [2024-12-27T20:44:08Z]     #10 grpc::Server::RegisterService(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const*, grpc::Service*) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/server/server_cc.cc:1074:35 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xee3f5)
  | [2024-12-27T20:44:08Z]     #11 grpc::Server::Start(grpc::ServerCompletionQueue**, unsigned long) /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/server/server_cc.cc:1172:5 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xef5e7)
  | [2024-12-27T20:44:08Z]     #12 grpc::ServerBuilder::BuildAndStart() /proc/self/cwd/external/com_github_grpc_grpc/src/cpp/server/server_builder.cc:446:11 (libexternal_Scom_Ugithub_Ugrpc_Ugrpc_Slibgrpc++_Ubase.so+0xe71f7)
  | [2024-12-27T20:44:08Z]     #13 ray::rpc::GrpcServer::Run() /proc/self/cwd/src/ray/rpc/grpc_server.cc:122:21 (liblibgrpc_Ucommon_Ulib.so+0x7495a)
  | [2024-12-27T20:44:08Z]     #14 GcsHealthCheckManagerTest::AddServer(bool) /proc/self/cwd/src/ray/gcs/gcs_server/test/gcs_health_check_manager_test.cc:90:13 (gcs_health_check_manager_test+0x10da06)
  | [2024-12-27T20:44:08Z]     #15 GcsHealthCheckManagerTest_MarkHealthAndSkipCheck_Test::TestBody() /proc/self/cwd/src/ray/gcs/gcs_server/test/gcs_health_check_manager_test.cc:166:18 (gcs_health_check_manager_test+0x10dfe6)
  | [2024-12-27T20:44:08Z]     #16 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2612:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xd4a1f)
  | [2024-12-27T20:44:08Z]     #17 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2648:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xd4a1f)
  | [2024-12-27T20:44:08Z]     #18 testing::Test::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2687:5 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xd4901)
  | [2024-12-27T20:44:08Z]     #19 testing::TestInfo::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2836:11 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xd63c8)
  | [2024-12-27T20:44:08Z]     #20 testing::TestSuite::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:3015:30 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xd79e4)
  | [2024-12-27T20:44:08Z]     #21 testing::internal::UnitTestImpl::RunAllTests() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5920:44 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xecc44)
  | [2024-12-27T20:44:08Z]     #22 bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2612:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xec05f)
  | [2024-12-27T20:44:08Z]     #23 bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:2648:14 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xec05f)
  | [2024-12-27T20:44:08Z]     #24 testing::UnitTest::Run() /proc/self/cwd/external/com_google_googletest/googletest/src/gtest.cc:5484:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest.so+0xebe4c)
  | [2024-12-27T20:44:08Z]     #25 RUN_ALL_TESTS() /proc/self/cwd/external/com_google_googletest/googletest/include/gtest/gtest.h:2317:73 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x60df5)
  | [2024-12-27T20:44:08Z]     #26 main /proc/self/cwd/external/com_google_googletest/googlemock/src/gmock_main.cc:71:10 (libexternal_Scom_Ugoogle_Ugoogletest_Slibgtest_Umain.so+0x60df5)
  | [2024-12-27T20:44:08Z]
  | [2024-12-27T20:44:08Z] SUMMARY: ThreadSanitizer: data race /opt/llvm/bin/../include/c++/v1/memory:1423:5 in std::__1::default_delete<ray::gcs::GcsHealthCheckManager>::operator()(ray::gcs::GcsHealthCheckManager*) const
  | [2024-12-27T20:44:08Z] ==================

Versions / Dependencies

master

Reproduction script

N/A

Issue Severity

Low: It annoys or frustrates me.
@dentiny dentiny added bug Something that is supposed to be working; but isn't triage Needs triage (eg: priority, bug/not-bug, and owning component) core Issues that should be addressed in Ray Core and removed triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Dec 28, 2024
@dentiny dentiny self-assigned this Dec 28, 2024
This was referenced Dec 28, 2024
Merged
Closed
rynewang pushed a commit that referenced this issue Dec 30, 2024
@dentiny
[core] Fix GCS health check manager TSAN (#49471)
66cec2f 
Fix issue: #49469

The bug is:
- We have two eventloops in the GCS health check manager, one for asio
io context, one for grpc
- In the unit test and production code elsewhere, we only synchronize on
io context by `io_context::stop`, but not grpc
- leading to grpc still accessing `GcsHealthCheckManager`, while we
**mistakenly think** all async operation have been properly synchronized

In this PR, I use shared pointer to make sure all accesses to gcs health
check manager is valid, even if io context has been stopped. Also
contain a fix to data member declaration order to respect the usage
dependency.

---------

Signed-off-by: dentiny <[email protected]>
@dentiny dentiny closed this as completed Dec 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dentiny dentiny

Labels
bug Something that is supposed to be working; but isn't core Issues that should be addressed in Ray Core
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@dentiny