[DocDB] SnapshotTxnTest.DeleteOnLoad flaky in master

[bmatican]

Jira Link: DB-1072

Description

https://detective-gcp.dev.yugabyte.com/stability/test?branch=master&buckets=20&class=SnapshotTxnTest&name=DeleteOnLoad

[bmatican]

Detective link: https://detective-gcp.dev.yugabyte.com/stability/test?branch=master&build_type=all&class=SnapshotTxnTest&fail_tag=tsan&name=DeleteOnLoad&platform=all

Jenkins log: https://jenkins.dev.yugabyte.com/job/github-yugabyte-db-centos-master-clang12-tsan/606/artifact/build/tsan-clang12-dynamic-ninja/yb-test-logs/tests-client__snapshot-txn-test/SnapshotTxnTest_DeleteOnLoad.log

TSAN race

  Write of size 8 at 0x7b6c00380a88 by thread T49 (mutexes: write M624728677111302072):
    #0 std::__1::enable_if<(is_move_constructible<yb::tablet::TransactionStatusResolver**>::value) && (is_move_assignable<yb::tablet::TransactionStatusResolver**>::value), void>::type std::__1::swap<yb::tablet::TransactionStatusResolver**>(yb::tablet::TransactionStatusResolver**&, yb::tablet::TransactionStatusResolver**&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/type_traits:3953:9 (libtablet.so+0x565d32)
    #1 std::__1::__split_buffer<yb::tablet::TransactionStatusResolver*, std::__1::allocator<yb::tablet::TransactionStatusResolver*> >::swap(std::__1::__split_buffer<yb::tablet::TransactionStatusResolver*, std::__1::allocator<yb::tablet::TransactionStatusResolver*> >&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/__split_buffer:416:5 (libtablet.so+0x5847c0)
    #2 std::__1::__deque_base<yb::tablet::TransactionStatusResolver, std::__1::allocator<yb::tablet::TransactionStatusResolver> >::swap(std::__1::__deque_base<yb::tablet::TransactionStatusResolver, std::__1::allocator<yb::tablet::TransactionStatusResolver> >&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/deque:1237:12 (libtablet.so+0x584733)
    #3 std::__1::deque<yb::tablet::TransactionStatusResolver, std::__1::allocator<yb::tablet::TransactionStatusResolver> >::swap(std::__1::deque<yb::tablet::TransactionStatusResolver, std::__1::allocator<yb::tablet::TransactionStatusResolver> >&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/deque:2955:13 (libtablet.so+0x584700)
    #4 yb::tablet::TransactionParticipant::Impl::CompleteShutdown() ${BUILD_ROOT}/../../src/yb/tablet/transaction_participant.cc:170:24 (libtablet.so+0x542d33)
    #5 yb::tablet::TransactionParticipant::CompleteShutdown() ${BUILD_ROOT}/../../src/yb/tablet/transaction_participant.cc:1793:10 (libtablet.so+0x53c4c5)
    #6 yb::tablet::Tablet::CompleteShutdown(yb::StronglyTypedBool<yb::tablet::DisableFlushOnShutdown_Tag>) ${BUILD_ROOT}/../../src/yb/tablet/tablet.cc:1010:31 (libtablet.so+0x41b9ca)
    #7 yb::tablet::TabletPeer::CompleteShutdown(yb::StronglyTypedBool<yb::tablet::DisableFlushOnShutdown_Tag>) ${BUILD_ROOT}/../../src/yb/tablet/tablet_peer.cc:522:14 (libtablet.so+0x4d6e13)
    #8 yb::tablet::TabletPeer::Shutdown(yb::StronglyTypedBool<yb::tablet::ShouldAbortActiveTransactions_Tag>, yb::StronglyTypedBool<yb::tablet::DisableFlushOnShutdown_Tag>) ${BUILD_ROOT}/../../src/yb/tablet/tablet_peer.cc:585:5 (libtablet.so+0x4d76e2)
    #9 yb::tserver::TSTabletManager::DeleteTablet(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, yb::tablet::TabletDataState, yb::StronglyTypedBool<yb::tablet::ShouldAbortActiveTransactions_Tag>, boost::optional<long> const&, bool, boost::optional<yb::tserver::TabletServerErrorPB_Code>*) ${BUILD_ROOT}/../../src/yb/tserver/ts_tablet_manager.cc:1270:3 (libtserver.so+0x504728)

...

  Previous read of size 8 at 0x7b6c00380a88 by thread T150 (mutexes: write M821479685832051288):
    #0 std::__1::__split_buffer<yb::tablet::TransactionStatusResolver*, std::__1::allocator<yb::tablet::TransactionStatusResolver*> >::begin() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/__split_buffer:79:78 (libtablet.so+0x555d2a)
    #1 std::__1::__deque_base<yb::tablet::TransactionStatusResolver, std::__1::allocator<yb::tablet::TransactionStatusResolver> >::begin() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/deque:1142:33 (libtablet.so+0x558db0)
    #2 std::__1::deque<yb::tablet::TransactionStatusResolver, std::__1::allocator<yb::tablet::TransactionStatusResolver> >::begin() /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/deque:1357:60 (libtablet.so+0x5608ca)
    #3 yb::tablet::TransactionParticipant::Impl::LoadFinished(std::__1::unordered_map<yb::StronglyTypedUuid<yb::TransactionId_Tag>, yb::tablet::ApplyStateWithCommitHt, boost::hash<yb::StronglyTypedUuid<yb::TransactionId_Tag> >, std::__1::equal_to<yb::StronglyTypedUuid<yb::TransactionId_Tag> >, std::__1::allocator<std::__1::pair<yb::StronglyTypedUuid<yb::TransactionId_Tag> const, yb::tablet::ApplyStateWithCommitHt> > > const&) ${BUILD_ROOT}/../../src/yb/tablet/transaction_participant.cc:1098:34 (libtablet.so+0x5453c7)

[bmatican]

Passing you another txn layer issue Eric.

[bmatican]

Also seems to appear in https://detective-gcp.dev.yugabyte.com/stability/test?branch=master&build_type=all&class=BackupTxnTest&fail_tag=tsan&name=DeleteTable&platform=all as well

cc @sanketkedia

[bmatican]

Maybe another flavor of this
Detective link: https://detective-gcp.dev.yugabyte.com/stability/test?branch=master&build_type=all&class=BackupTxnTest&fail_tag=tsan&name=DeleteTableWithMastersRestart&platform=all

Jenkins log: https://jenkins.dev.yugabyte.com/job/github-yugabyte-db-centos-master-clang12-tsan/629/artifact/build/tsan-clang12-dynamic-ninja/yb-test-logs/tests-client__backup-txn-test/BackupTxnTest_DeleteTableWithMastersRestart.log

    #0 yb::rpc::Poller::Start(yb::rpc::Scheduler*, yb::MonoDelta) ${BUILD_ROOT}/../../src/yb/rpc/poller.cc:31:14 (libyrpc.so+0x32a2e4)
    #1 yb::tablet::TransactionParticipant::Impl::LoadFinished(std::__1::unordered_map<yb::StronglyTypedUuid<yb::TransactionId_Tag>, yb::tablet::ApplyStateWithCommitHt, boost::hash<yb::StronglyTypedUuid<yb::TransactionId_Tag> >, std::__1::equal_to<yb::StronglyTypedUuid<yb::TransactionId_Tag> >, std::__1::allocator<std::__1::pair<yb::StronglyTypedUuid<yb::TransactionId_Tag> const, yb::tablet::ApplyStateWithCommitHt> > > const&) ${BUILD_ROOT}/../../src/yb/tablet/transaction_participant.cc:1103:13 (libtablet.so+0x545493)
    #2 non-virtual thunk to yb::tablet::TransactionParticipant::Impl::LoadFinished(std::__1::unordered_map<yb::StronglyTypedUuid<yb::TransactionId_Tag>, yb::tablet::ApplyStateWithCommitHt, boost::hash<yb::StronglyTypedUuid<yb::TransactionId_Tag> >, std::__1::equal_to<yb::StronglyTypedUuid<yb::TransactionId_Tag> >, std::__1::allocator<std::__1::pair<yb::StronglyTypedUuid<yb::TransactionId_Tag> const, yb::tablet::ApplyStateWithCommitHt> > > const&) ${BUILD_ROOT}/../../src/yb/tablet/transaction_participant.cc (libtablet.so+0x545bb7)
    #3 yb::tablet::TransactionLoader::Executor::Execute()::'lambda'()::operator()() const ${BUILD_ROOT}/../../src/yb/tablet/transaction_loader.cc:90:15 (libtablet.so+0x538466)
    #4 yb::ScopeExitLambda<yb::tablet::TransactionLoader::Executor::Execute()::'lambda'()>::~ScopeExitLambda() ${BUILD_ROOT}/../../src/yb/util/scope_exit.h:38:7 (libtablet.so+0x531308)
    #5 yb::tablet::TransactionLoader::Executor::Execute() ${BUILD_ROOT}/../../src/yb/tablet/transaction_loader.cc:97:3 (libtablet.so+0x5301b7)
    #6 decltype(*(std::__1::forward<yb::tablet::TransactionLoader::Executor*>(fp0)).*fp()) std::__1::__invoke<void (yb::tablet::TransactionLoader::Executor::*)(), yb::tablet::TransactionLoader::Executor*, void>(void (yb::tablet::TransactionLoader::Executor::*&&)(), yb::tablet::TransactionLoader::Executor*&&) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/type_traits:3635:1 (libtablet.so+0x5399dc)
    #7 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (yb::tablet::TransactionLoader::Executor::*)(), yb::tablet::TransactionLoader::Executor*, 2ul>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (yb::tablet::TransactionLoader::Executor::*)(), yb::tablet::TransactionLoader::Executor*>&, std::__1::__tuple_indices<2ul>) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/thread:280:5 (libtablet.so+0x5398f0)
    #8 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (yb::tablet::TransactionLoader::Executor::*)(), yb::tablet::TransactionLoader::Executor*> >(void*) /opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20220630123401-af96d73e39-centos7-x86_64-clang12/installed/tsan/libcxx/include/c++/v1/thread:291:5 (libtablet.so+0x53907a)

  Previous read of size 8 at 0x7b6c00040ab0 by main thread (mutexes: write M910425692569622004, write M1028645268687686336):
    #0 yb::rpc::Poller::Shutdown() ${BUILD_ROOT}/../../src/yb/rpc/poller.cc:44:9 (libyrpc.so+0x32a48d)
I0722 18:36:58.366695 30111 docdb_rocksdb_util.cc:390] Overriding FLAGS_rocksdb_max_background_flushes to 1
    #1 yb::tablet::TransactionParticipant::Impl::StartShutdown() ${BUILD_ROOT}/../../src/yb/tablet/transaction_participant.cc:151:13 (libtablet.so+0x542b5c)
    #2 yb::tablet::TransactionParticipant::StartShutdown() ${BUILD_ROOT}/../../src/yb/tablet/transaction_participant.cc:1789:10 (libtablet.so+0x53c455)
    #3 yb::tablet::Tablet::StartShutdown() ${BUILD_ROOT}/../../src/yb/tablet/tablet.cc:982:31 (libtablet.so+0x41b827)
I0722 18:36:58.367177 30111 docdb_rocksdb_util.cc:423] FLAGS_rocksdb_max_background_compactions was not set, automatically configuring 1 background compactions.
    #4 yb::tablet::TabletPeer::StartShutdown() ${BUILD_ROOT}/../../src/yb/tablet/tablet_peer.cc:460:16 (libtablet.so+0x4d67b0)
    #5 yb::tserver::TSTabletManager::StartShutdown() ${BUILD_ROOT}/../../src/yb/tserver/ts_tablet_manager.cc:1591:15 (libtserver.so+0x4ffdf1)
W0722 18:36:58.360709 30253 async_rpc_tasks.cc:368] DeleteTablet RPC for tablet {ql_client_test_table_tablet_id3} (ql_client_test_table [id=62ca1f29bba24eb28a90e29b2dd4de20]) on TS=33b327e6527e4a4daaabe3fedea8ebaf (task=0x00007b4c003ce358, state=kRunning): Timed out (yb/master/async_rpc_tasks.cc:367): Request timed out
    #6 yb::tserver::TabletServer::Shutdown() ${BUILD_ROOT}/../../src/yb/tserver/tablet_server.cc:449:22 (libtserver.so+0x47615d)
I0722 18:36:58.368772 30111 docdb_rocksdb_util.cc:423] FLAGS_rocksdb_max_background_compactions was not set, automatically configuring 1 background compactions.
    #7 yb::tserver::enterprise::TabletServer::Shutdown() ${BUILD_ROOT}/../../ent/src/yb/tserver/tablet_server_ent.cc:74:10 (libtserver.so+0x577d0e)
    #8 yb::tserver::MiniTabletServer::Shutdown() ${BUILD_ROOT}/../../src/yb/tserver/mini_tablet_server.cc:203:14 (libtserver_test_util.so+0x7cc08)