Session run timeout causes memory leak and saving stuck

I am using distributed tensorflow, one of my machines was something wrong and caused almost all session run timeout, eventually memory reached 80G and been killed. This also caused worker0 process failed to save model, saver.save() stuck forever.
I am using 0.12.1 windows build. I searched and nobody ever reported issues like this. Any idea and advice to confirm and fix it?

I don't think we'll be able to debug this with the information provided.  Can you reproduce the problem?  It may be worth checking if TensorFlow 1.0 solves it for you.

I ran into this issue two times, however I will try 1.0 version. But calling saving.save without shard=True stuck forever each time when I have lots of workers, for example 40 ps machines and 80 worker machines, each worker machine starts 10 worker processes.
here is the stack of master worker that calls saving, maybe you guys could get some idea:
Child-SP          RetAddr           Call Site
00 0000000fdf7ec918 00007ffbbefff58f ntdll!ZwRemoveIoCompletion+0x14 [d:\rs1.obj.amd64fre\minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm @ 251]
01 0000000fdf7ec920 00007ffb9957fdd9 KERNELBASE!GetQueuedCompletionStatus+0x3f [d:\rs1\minkernel\kernelbase\error.c @ 844]
02 0000000fdf7ec980 00007ffb995693e6 _pywrap_tensorflow!grpc_iocp_work+0x69 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\iomgr\iocp_windows.c @ 87]
03 0000000fdf7eca10 00007ffb99549466 _pywrap_tensorflow!grpc_pollset_work+0xb6 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\iomgr\pollset_windows.c @ 152]
04 0000000fdf7ecac0 00007ffb99537fa1 _pywrap_tensorflow!grpc_completion_queue_pluck+0x256 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\surface\completion_queue.c @ 510]
05 0000000fdf7ecc00 00007ffb9751127b _pywrap_tensorflow!grpc::CoreCodegen::grpc_completion_queue_pluck+0x41 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\cpp\common\core_codegen.cc @ 66]
06 (Inline Function) ---------------- _pywrap_tensorflow!grpc::CompletionQueue::Pluck+0x53 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\include\grpc++\impl\codegen\completion_queue.h @ 207] 07 0000000fdf7ecc60 00007ffb97515f96 _pywrap_tensorflow!grpc::BlockingUnaryCall<tensorflow::RunStepRequest,tensorflow::RunStepResponse>+0x32b [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\include\grpc++\impl\codegen\client_unary_call.h @ 72] 08 0000000fdf7ece80 00007ffb97518af4 _pywrap_tensorflow!tensorflow::grpc::MasterService::Stub::RunStep+0x36 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\rpc\grpc_master_service_impl.cc @ 80] 09 0000000fdf7eced0 00007ffb97533fa4 _pywrap_tensorflow!tensorflow::GrpcRemoteMaster::RunStep+0xa4 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\rpc\grpc_remote_master.cc @ 60] 0a 0000000fdf7ed0a0 00007ffb9753387b _pywrap_tensorflow!tensorflow::GrpcSession::RunProto+0x124 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\rpc\grpc_session.cc @ 238] 0b 0000000fdf7ed100 00007ffb971bc0a0 _pywrap_tensorflow!tensorflow::GrpcSession::Run+0x4cb [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\rpc\grpc_session.cc @ 188] 0c 0000000fdf7ed430 00007ffb971c8a7a _pywrap_tensorflow!TF_Run_Helper+0x1c0 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\c\c_api.cc @ 543] 0d 0000000fdf7ed5d0 00007ffb9719a509 _pywrap_tensorflow!TF_Run+0x1ba [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\c\c_api.cc @ 606] 0e 0000000fdf7ed6a0 00007ffb97199e1a _pywrap_tensorflow!tensorflow::TF_Run_wrapper_helper+0x6c9 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\python\client\tf_session_helper.cc @ 500] 0f 0000000fdf7ed970 00007ffb971d07ac _pywrap_tensorflow!tensorflow::TF_Run_wrapper+0x4a [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\python\client\tf_session_helper.cc @ 550] 10 0000000fdf7ed9d0 000000006eb0e512 _pywrap_tensorflow!_wrap_TF_Run+0x40c [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\pywrap_tensorflow.cc @ 5862] 11 0000000fdf7edc00 000000006eb89e20 python35!PyCFunction_Call+0x122 12 0000000fdf7edc30 000000006eb86bc1 python35!PyEval_GetFuncDesc+0x3b0 13 0000000fdf7edc90 000000006eb88a9e python35!PyEval_EvalFrameEx+0x3241 14 0000000fdf7edd90 000000006eaf8751 python35!PyEval_EvalFrameEx+0x511e 15 0000000fdf7ede30 000000006eac6058 python35!PyFunction_SetAnnotations+0xb71 16 0000000fdf7edee0 000000006eb8a7cd python35!PyObject_Call+0x78 17 0000000fdf7edf10 000000006eb86c66 python35!PyEval_GetFuncDesc+0xd5d 18 0000000fdf7edf80 000000006eb88a9e python35!PyEval_EvalFrameEx+0x32e6 19 0000000fdf7ee080 000000006eb8a070 python35!PyEval_EvalFrameEx+0x511e 1a 0000000fdf7ee120 000000006eb89ea1 python35!PyEval_GetFuncDesc+0x600 1b 0000000fdf7ee1c0 000000006eb86bc1 python35!PyEval_GetFuncDesc+0x431 1c 0000000fdf7ee220 000000006eb88a9e python35!PyEval_EvalFrameEx+0x3241 1d 0000000fdf7ee320 000000006eb8a070 python35!PyEval_EvalFrameEx+0x511e 1e 0000000fdf7ee3c0 000000006eb89ea1 python35!PyEval_GetFuncDesc+0x600 1f 0000000fdf7ee460 000000006eb86bc1 python35!PyEval_GetFuncDesc+0x431 20 0000000fdf7ee4c0 000000006eb89fe8 python35!PyEval_EvalFrameEx+0x3241 21 0000000fdf7ee5c0 000000006eb89ea1 python35!PyEval_GetFuncDesc+0x578 22 0000000fdf7ee660 000000006eb86bc1 python35!PyEval_GetFuncDesc+0x431 23 0000000fdf7ee6c0 000000006eb88a9e python35!PyEval_EvalFrameEx+0x3241 24 0000000fdf7ee7c0 000000006eb8a070 python35!PyEval_EvalFrameEx+0x511e 25 0000000fdf7ee860 000000006eb89ea1 python35!PyEval_GetFuncDesc+0x600 26 0000000fdf7ee900 000000006eb86bc1 python35!PyEval_GetFuncDesc+0x431 27 0000000fdf7ee960 000000006eb88a9e python35!PyEval_EvalFrameEx+0x3241 28 0000000fdf7eea60 000000006eb8a070 python35!PyEval_EvalFrameEx+0x511e 29 0000000fdf7eeb00 000000006eb89ea1 python35!PyEval_GetFuncDesc+0x600 2a 0000000fdf7eeba0 000000006eb86bc1 python35!PyEval_GetFuncDesc+0x431 2b 0000000fdf7eec00 000000006eb89fe8 python35!PyEval_EvalFrameEx+0x3241 2c 0000000fdf7eed00 000000006eb89ea1 python35!PyEval_GetFuncDesc+0x578 2d 0000000fdf7eeda0 000000006eb86bc1 python35!PyEval_GetFuncDesc+0x431 2e 0000000fdf7eee00 000000006eb88a9e python35!PyEval_EvalFrameEx+0x3241 2f 0000000fdf7eef00 000000006eb8a070 python35!PyEval_EvalFrameEx+0x511e 30 0000000fdf7eefa0 000000006eb89ea1 python35!PyEval_GetFuncDesc+0x600 31 0000000fdf7ef040 000000006eb86bc1 python35!PyEval_GetFuncDesc+0x431 32 0000000fdf7ef0a0 000000006eb88a9e python35!PyEval_EvalFrameEx+0x3241 33 0000000fdf7ef1a0 000000006eb8a070 python35!PyEval_EvalFrameEx+0x511e 34 0000000fdf7ef240 000000006eb89ea1 python35!PyEval_GetFuncDesc+0x600 35 0000000fdf7ef2e0 000000006eb86bc1 python35!PyEval_GetFuncDesc+0x431 36 0000000fdf7ef340 000000006eb88a9e python35!PyEval_EvalFrameEx+0x3241 37 0000000fdf7ef440 000000006ebd423a python35!PyEval_EvalFrameEx+0x511e 38 0000000fdf7ef4e0 000000006ebd3f78 python35!PyRun_FileExFlags+0x38a 39 0000000fdf7ef560 000000006ebd273e python35!PyRun_FileExFlags+0xc8 3a 0000000fdf7ef5d0 000000006ebd19b3 python35!PyRun_SimpleFileExFlags+0x2be 3b 0000000fdf7ef660 000000006ea5f215 python35!PyRun_AnyFileExFlags+0x63 3c 0000000fdf7ef690 000000006ea5fc5d python35!Py_hashtable_copy+0x71f5 3d 0000000fdf7ef6d0 000000001cb01264 python35!Py_Main+0xa0d 3e 0000000fdf7ef7c0 00007ffbc1488364 python+0x1264 3f 0000000fdf7ef800 00007ffbc2b270d1 kernel32!BaseThreadInitThunk+0x14 40 0000000fdf7ef830 00000000`00000000 ntdll!RtlUserThreadStart+0x21 [d:\rs1\minkernel\ntdll\rtlstrt.c @ 997]
Child-SP          RetAddr           Call Site
00 0000000fdffaed28 00007ffbbeffc4a7 ntdll!ZwDelayExecution+0x14 [d:\rs1.obj.amd64fre\minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm @ 595]
01 0000000fdffaed30 000000006ea8de6e KERNELBASE!SleepEx+0xa7 [d:\rs1\minkernel\kernelbase\thread.c @ 2247]
02 0000000fdffaedd0 000000006ea8c437 python35!PyOS_SigintEvent+0x698e
03 0000000fdffaee20 000000006eb89d00 python35!PyOS_SigintEvent+0x4f57
04 0000000fdffaee50 000000006eb86bc1 python35!PyEval_GetFuncDesc+0x290
05 0000000fdffaeeb0 000000006eb88a9e python35!PyEval_EvalFrameEx+0x3241
06 0000000fdffaefb0 000000006eaf8751 python35!PyEval_EvalFrameEx+0x511e
07 0000000fdffaf050 000000006eac6058 python35!PyFunction_SetAnnotations+0xb71
08 0000000fdffaf100 000000006eb8a7cd python35!PyObject_Call+0x78
09 0000000fdffaf130 000000006eb86c66 python35!PyEval_GetFuncDesc+0xd5d
0a 0000000fdffaf1a0 000000006eb89fe8 python35!PyEval_EvalFrameEx+0x32e6
0b 0000000fdffaf2a0 000000006eb89ea1 python35!PyEval_GetFuncDesc+0x578
0c 0000000fdffaf340 000000006eb86bc1 python35!PyEval_GetFuncDesc+0x431
0d 0000000fdffaf3a0 000000006eb89fe8 python35!PyEval_EvalFrameEx+0x3241
0e 0000000fdffaf4a0 000000006eb89ea1 python35!PyEval_GetFuncDesc+0x578
0f 0000000fdffaf540 000000006eb86bc1 python35!PyEval_GetFuncDesc+0x431
10 0000000fdffaf5a0 000000006eb88a9e python35!PyEval_EvalFrameEx+0x3241
11 0000000fdffaf6a0 000000006eaf8751 python35!PyEval_EvalFrameEx+0x511e
12 0000000fdffaf740 000000006eac6058 python35!PyFunction_SetAnnotations+0xb71
13 0000000fdffaf7f0 000000006eadd82d python35!PyObject_Call+0x78
14 0000000fdffaf820 000000006eac6058 python35!PyMethod_New+0xa9d
15 0000000fdffaf860 000000006eb899ec python35!PyObject_Call+0x78
16 0000000fdffaf890 000000006ea89193 python35!PyEval_CallObjectWithKeywords+0xcc
17 0000000fdffaf8c0 000000006ebde352 python35!PyOS_SigintEvent+0x1cb3
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for ucrtbase.dll -
18 0000000fdffaf900 00007ffbbf3bcab0 python35!PyThread_init_thread+0x4b2
19 0000000fdffaf930 00007ffbc1488364 ucrtbase!o__realloc_base+0x60
1a 0000000fdffaf960 00007ffbc2b270d1 kernel32!BaseThreadInitThunk+0x14
1b 0000000fdffaf990<PHONE_NUMBER>000000 ntdll!RtlUserThreadStart+0x21 [d:\rs1\minkernel\ntdll\rtlstrt.c @ 997]
Child-SP          RetAddr           Call Site
00 0000000fe019f668 00007ffbc2b2cfb5 ntdll!ZwWaitForAlertByThreadId+0x14 [d:\rs1.obj.amd64fre\minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm @ 3731]
01 0000000fe019f670 00007ffbbf018d28 ntdll!RtlSleepConditionVariableSRW+0xf5 [d:\rs1\minkernel\ntos\rtl\condition.c @ 1109]
*** ERROR: Symbol file could not be found.  Defaulted to export symbols for msvcp140.dll -
02 0000000fe019f6f0 00007ffbb6978368 KERNELBASE!SleepConditionVariableSRW+0x28 [d:\rs1\minkernel\kernelbase\synch.c @ 2020]
03 0000000fe019f730 00007ffbb6978530 msvcp140!std::_Syserror_map+0x198
04 0000000fe019f760 00007ffb971e89e7 msvcp140!Cnd_destroy+0x70
05 (Inline Function) ---------------- _pywrap_tensorflow!std::_Cnd_waitX+0xd [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 95] 06 (Inline Function) ---------------- _pywrap_tensorflow!std::condition_variable::wait+0xd [c:\program files (x86)\microsoft visual studio 14.0\vc\include\mutex @ 565]
07 0000000fe019f7e0 00007ffb971e80f9 _pywrap_tensorflow!Eigen::EventCount::Park+0x67 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\eventcount.h @ 208]
08 0000000fe019f830 00007ffb971e98a0 _pywrap_tensorflow!Eigen::EventCount::CommitWait+0xc9 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\eventcount.h @ 103]
09 0000000fe019f870 00007ffb971e9bc3 _pywrap_tensorflow!Eigen::NonBlockingThreadPoolTempltensorflow::thread::EigenEnvironment::WaitForWork+0x180 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\nonblockingthreadpool.h @ 233]
0a 0000000fe019f8b0 00007ffb971e9f6e _pywrap_tensorflow!Eigen::NonBlockingThreadPoolTempltensorflow::thread::EigenEnvironment::WorkerLoop+0x293 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\nonblockingthreadpool.h @ 160]
0b (Inline Function) ---------------- _pywrap_tensorflow!std::_Func_class<void>::operator()+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 279] 0c (Inline Function) ---------------- _pywrap_tensorflow!tensorflow::thread::EigenEnvironment::CreateThread::__l2::<lambda_3e5df898eaffef9b52307b4af9ee50ab>::operator()+0x2c [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\lib\core\threadpool.cc @ 57]
0d (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoker_functor::_Call+0x2c [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398] 0e (Inline Function) ---------------- _pywrap_tensorflow!std::invoke+0x2c [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466]
0f (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoke_ret+0x2c [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1484] 10 0000000fe019f920 00007ffb9721ce25 _pywrap_tensorflow!std::_Func_impl<<lambda_3e5df898eaffef9b52307b4af9ee50ab>,std::allocator<int>,void>::_Do_call+0x3e [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 212] 11 (Inline Function) ---------------- _pywrap_tensorflow!std::_Func_class::operator()+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 279]
12 (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoker_functor::_Call+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398] 13 (Inline Function) ---------------- _pywrap_tensorflow!std::invoke+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466]
14 (Inline Function) ---------------- _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Execute+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 240] 15 0000000fe019f960 00007ffb9721cab9 _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Run+0x85 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 247] 16 0000000fe019f9a0 00007ffbbf3bcab0 _pywrap_tensorflow!std::_Pad::_Call_func+0x9 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 210] 17 0000000fe019f9d0 00007ffbc1488364 ucrtbase!o__realloc_base+0x60 18 0000000fe019fa00 00007ffbc2b270d1 kernel32!BaseThreadInitThunk+0x14 19 0000000fe019fa30 00000000`00000000 ntdll!RtlUserThreadStart+0x21 [d:\rs1\minkernel\ntdll\rtlstrt.c @ 997]
Child-SP          RetAddr           Call Site
00 0000000fe038f6f8 00007ffbc2b2d119 ntdll!ZwWaitForAlertByThreadId+0x14 [d:\rs1.obj.amd64fre\minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm @ 3731]
01 0000000fe038f700 00007ffbbf018d78 ntdll!RtlSleepConditionVariableCS+0xc9 [d:\rs1\minkernel\ntos\rtl\condition.c @ 927]
02 0000000fe038f770 00007ffb99585d6e KERNELBASE!SleepConditionVariableCS+0x28 [d:\rs1\minkernel\kernelbase\synch.c @ 1985]
03 0000000fe038f7a0 00007ffb995694a9 _pywrap_tensorflow!gpr_cv_wait+0x11e [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\support\sync_windows.c @ 103]
04 0000000fe038f800 00007ffb99548fb2 _pywrap_tensorflow!grpc_pollset_work+0x179 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\iomgr\pollset_windows.c @ 182]
05 0000000fe038f8b0 00007ffb99537cae _pywrap_tensorflow!grpc_completion_queue_next+0x1f2 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\surface\completion_queue.c @ 380]
06 0000000fe038f9f0 00007ffb975359e1 _pywrap_tensorflow!grpc::CompletionQueue::AsyncNextInternal+0x3e [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\cpp\common\completion_queue.cc @ 56]
07 (Inline Function) ---------------- _pywrap_tensorflow!grpc::CompletionQueue::Next+0xa5 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\include\grpc++\impl\codegen\completion_queue.h @ 147] 08 0000000fe038fa40 00007ffb9721ce25 _pywrap_tensorflow!<lambda_ea64bb373946421891242c9b912b7e25>::operator()+0xc1 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\rpc\grpc_worker_cache.cc @ 38] 09 (Inline Function) ---------------- _pywrap_tensorflow!std::_Func_class::operator()+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 279]
0a (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoker_functor::_Call+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398] 0b (Inline Function) ---------------- _pywrap_tensorflow!std::invoke+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466]
0c (Inline Function) ---------------- _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Execute+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 240] 0d 0000000fe038fab0 00007ffb9721cab9 _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Run+0x85 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 247] 0e 0000000fe038faf0 00007ffbbf3bcab0 _pywrap_tensorflow!std::_Pad::_Call_func+0x9 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 210] 0f 0000000fe038fb20 00007ffbc1488364 ucrtbase!o__realloc_base+0x60 10 0000000fe038fb50 00007ffbc2b270d1 kernel32!BaseThreadInitThunk+0x14 11 0000000fe038fb80 00000000`00000000 ntdll!RtlUserThreadStart+0x21 [d:\rs1\minkernel\ntdll\rtlstrt.c @ 997]
Child-SP          RetAddr           Call Site
00 0000000fe057fb88 00007ffbc2b2cfb5 ntdll!ZwWaitForAlertByThreadId+0x14 [d:\rs1.obj.amd64fre\minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm @ 3731]
01 0000000fe057fb90 00007ffbbf018d28 ntdll!RtlSleepConditionVariableSRW+0xf5 [d:\rs1\minkernel\ntos\rtl\condition.c @ 1109]
02 0000000fe057fc10 00007ffbb6978368 KERNELBASE!SleepConditionVariableSRW+0x28 [d:\rs1\minkernel\kernelbase\synch.c @ 2020]
03 0000000fe057fc50 00007ffbb6978530 msvcp140!std::_Syserror_map+0x198
04 0000000fe057fc80 00007ffb971e89e7 msvcp140!Cnd_destroy+0x70
05 (Inline Function) ---------------- _pywrap_tensorflow!std::_Cnd_waitX+0xd [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 95] 06 (Inline Function) ---------------- _pywrap_tensorflow!std::condition_variable::wait+0xd [c:\program files (x86)\microsoft visual studio 14.0\vc\include\mutex @ 565]
07 0000000fe057fd00 00007ffb971e80f9 _pywrap_tensorflow!Eigen::EventCount::Park+0x67 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\eventcount.h @ 208]
08 0000000fe057fd50 00007ffb971e98a0 _pywrap_tensorflow!Eigen::EventCount::CommitWait+0xc9 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\eventcount.h @ 103]
09 0000000fe057fd90 00007ffb971e9bc3 _pywrap_tensorflow!Eigen::NonBlockingThreadPoolTempltensorflow::thread::EigenEnvironment::WaitForWork+0x180 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\nonblockingthreadpool.h @ 233]
0a 0000000fe057fdd0 00007ffb971e9f6e _pywrap_tensorflow!Eigen::NonBlockingThreadPoolTempltensorflow::thread::EigenEnvironment::WorkerLoop+0x293 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\nonblockingthreadpool.h @ 160]
0b (Inline Function) ---------------- _pywrap_tensorflow!std::_Func_class<void>::operator()+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 279] 0c (Inline Function) ---------------- _pywrap_tensorflow!tensorflow::thread::EigenEnvironment::CreateThread::__l2::<lambda_3e5df898eaffef9b52307b4af9ee50ab>::operator()+0x2c [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\lib\core\threadpool.cc @ 57]
0d (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoker_functor::_Call+0x2c [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398] 0e (Inline Function) ---------------- _pywrap_tensorflow!std::invoke+0x2c [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466]
0f (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoke_ret+0x2c [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1484] 10 0000000fe057fe40 00007ffb9721ce25 _pywrap_tensorflow!std::_Func_impl<<lambda_3e5df898eaffef9b52307b4af9ee50ab>,std::allocator<int>,void>::_Do_call+0x3e [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 212] 11 (Inline Function) ---------------- _pywrap_tensorflow!std::_Func_class::operator()+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 279]
12 (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoker_functor::_Call+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398] 13 (Inline Function) ---------------- _pywrap_tensorflow!std::invoke+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466]
14 (Inline Function) ---------------- _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Execute+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 240] 15 0000000fe057fe80 00007ffb9721cab9 _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Run+0x85 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 247] 16 0000000fe057fec0 00007ffbbf3bcab0 _pywrap_tensorflow!std::_Pad::_Call_func+0x9 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 210] 17 0000000fe057fef0 00007ffbc1488364 ucrtbase!o__realloc_base+0x60 18 0000000fe057ff20 00007ffbc2b270d1 kernel32!BaseThreadInitThunk+0x14 19 0000000fe057ff50 00000000`00000000 ntdll!RtlUserThreadStart+0x21 [d:\rs1\minkernel\ntdll\rtlstrt.c @ 997]
Child-SP          RetAddr           Call Site
00 0000000fe076f918 00007ffbc2b2d119 ntdll!ZwWaitForAlertByThreadId+0x14 [d:\rs1.obj.amd64fre\minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm @ 3731]
01 0000000fe076f920 00007ffbbf018d78 ntdll!RtlSleepConditionVariableCS+0xc9 [d:\rs1\minkernel\ntos\rtl\condition.c @ 927]
02 0000000fe076f990 00007ffb99585d6e KERNELBASE!SleepConditionVariableCS+0x28 [d:\rs1\minkernel\kernelbase\synch.c @ 1985]
03 0000000fe076f9c0 00007ffb995694a9 _pywrap_tensorflow!gpr_cv_wait+0x11e [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\support\sync_windows.c @ 103]
04 0000000fe076fa20 00007ffb99548fb2 _pywrap_tensorflow!grpc_pollset_work+0x179 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\iomgr\pollset_windows.c @ 182]
05 0000000fe076fad0 00007ffb99537cae _pywrap_tensorflow!grpc_completion_queue_next+0x1f2 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\surface\completion_queue.c @ 380]
06 0000000fe076fc10 00007ffb9750a8da _pywrap_tensorflow!grpc::CompletionQueue::AsyncNextInternal+0x3e [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\cpp\common\completion_queue.cc @ 56]
07 (Inline Function) ---------------- _pywrap_tensorflow!grpc::CompletionQueue::Next+0xe1 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\include\grpc++\impl\codegen\completion_queue.h @ 147] 08 0000000fe076fc60 00007ffb9721ce25 _pywrap_tensorflow!tensorflow::GrpcMasterService::HandleRPCsLoop+0x56a [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\rpc\grpc_master_service.cc @ 116] 09 (Inline Function) ---------------- _pywrap_tensorflow!std::_Func_class::operator()+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 279]
0a (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoker_functor::_Call+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398] 0b (Inline Function) ---------------- _pywrap_tensorflow!std::invoke+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466]
0c (Inline Function) ---------------- _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Execute+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 240] 0d 0000000fe076fd70 00007ffb9721cab9 _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Run+0x85 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 247] 0e 0000000fe076fdb0 00007ffbbf3bcab0 _pywrap_tensorflow!std::_Pad::_Call_func+0x9 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 210] 0f 0000000fe076fde0 00007ffbc1488364 ucrtbase!o__realloc_base+0x60 10 0000000fe076fe10 00007ffbc2b270d1 kernel32!BaseThreadInitThunk+0x14 11 0000000fe076fe40 00000000`00000000 ntdll!RtlUserThreadStart+0x21 [d:\rs1\minkernel\ntdll\rtlstrt.c @ 997]
Child-SP          RetAddr           Call Site
00 0000000fe095f8c8 00007ffbc2b2d119 ntdll!ZwWaitForAlertByThreadId+0x14 [d:\rs1.obj.amd64fre\minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm @ 3731]
01 0000000fe095f8d0 00007ffbbf018d78 ntdll!RtlSleepConditionVariableCS+0xc9 [d:\rs1\minkernel\ntos\rtl\condition.c @ 927]
02 0000000fe095f940 00007ffb99585d6e KERNELBASE!SleepConditionVariableCS+0x28 [d:\rs1\minkernel\kernelbase\synch.c @ 1985]
03 0000000fe095f970 00007ffb995694a9 _pywrap_tensorflow!gpr_cv_wait+0x11e [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\support\sync_windows.c @ 103]
04 0000000fe095f9d0 00007ffb99548fb2 _pywrap_tensorflow!grpc_pollset_work+0x179 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\iomgr\pollset_windows.c @ 182]
05 0000000fe095fa80 00007ffb99537cae _pywrap_tensorflow!grpc_completion_queue_next+0x1f2 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\surface\completion_queue.c @ 380]
06 0000000fe095fbc0 00007ffb9754189b _pywrap_tensorflow!grpc::CompletionQueue::AsyncNextInternal+0x3e [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\cpp\common\completion_queue.cc @ 56]
07 (Inline Function) ---------------- _pywrap_tensorflow!grpc::CompletionQueue::Next+0xed [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\include\grpc++\impl\codegen\completion_queue.h @ 147] 08 0000000fe095fc10 00007ffb9721ce25 _pywrap_tensorflow!tensorflow::anonymous namespace'::GrpcWorkerService::HandleRPCsLoop+0x6cb [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\rpc\grpc_worker_service.cc @ 145]
09 (Inline Function) ---------------- _pywrap_tensorflow!std::_Func_class<void>::operator()+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 279] 0a (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoker_functor::_Call+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398]
0b (Inline Function) ---------------- _pywrap_tensorflow!std::invoke+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466] 0c (Inline Function) ---------------- _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Execute+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 240]
0d 0000000fe095fd50 00007ffb9721cab9 _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Run+0x85 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 247]
0e 0000000fe095fd90 00007ffbbf3bcab0 _pywrap_tensorflow!std::_Pad::_Call_func+0x9 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 210]
0f 0000000fe095fdc0 00007ffbc1488364 ucrtbase!o__realloc_base+0x60
10 0000000fe095fdf0 00007ffbc2b270d1 kernel32!BaseThreadInitThunk+0x14
11 0000000fe095fe20<PHONE_NUMBER>000000 ntdll!RtlUserThreadStart+0x21 [d:\rs1\minkernel\ntdll\rtlstrt.c @ 997]
Child-SP          RetAddr           Call Site
00 0000000fe18de858 00007ffbbefc75ff ntdll!ZwWaitForSingleObject+0x14 [d:\rs1.obj.amd64fre\minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm @ 211]
01 0000000fe18de860 000000006ebde1bc KERNELBASE!WaitForSingleObjectEx+0x8f [d:\rs1\minkernel\kernelbase\synch.c @ 1328]
02 0000000fe18de900 000000006ebde56e python35!PyThread_init_thread+0x31c
03 0000000fe18de940 000000006ea87df7 python35!PyThread_acquire_lock_timed+0x6e
04 0000000fe18de970 000000006ea88099 python35!PyOS_SigintEvent+0x917
05 0000000fe18de9c0 000000006eb0e420 python35!PyOS_SigintEvent+0xbb9
06 0000000fe18de9f0 000000006eb89e20 python35!PyCFunction_Call+0x30
07 0000000fe18dea20 000000006eb86bc1 python35!PyEval_GetFuncDesc+0x3b0
08 0000000fe18dea80 000000006eb88a9e python35!PyEval_EvalFrameEx+0x3241
09 0000000fe18deb80 000000006eb8a070 python35!PyEval_EvalFrameEx+0x511e
0a 0000000fe18dec20 000000006eb89ea1 python35!PyEval_GetFuncDesc+0x600
0b 0000000fe18decc0 000000006eb86bc1 python35!PyEval_GetFuncDesc+0x431
0c 0000000fe18ded20 000000006eb88a9e python35!PyEval_EvalFrameEx+0x3241
0d 0000000fe18dee20 000000006eb8a070 python35!PyEval_EvalFrameEx+0x511e
0e 0000000fe18deec0 000000006eb89ea1 python35!PyEval_GetFuncDesc+0x600
0f 0000000fe18def60 000000006eb86bc1 python35!PyEval_GetFuncDesc+0x431
10 0000000fe18defc0 000000006eb88a9e python35!PyEval_EvalFrameEx+0x3241
11 0000000fe18df0c0 000000006eb8a070 python35!PyEval_EvalFrameEx+0x511e
12 0000000fe18df160 000000006eb89ea1 python35!PyEval_GetFuncDesc+0x600
13 0000000fe18df200 000000006eb86bc1 python35!PyEval_GetFuncDesc+0x431
14 0000000fe18df260 000000006eb88a9e python35!PyEval_EvalFrameEx+0x3241
15 0000000fe18df360 000000006eaf8751 python35!PyEval_EvalFrameEx+0x511e
16 0000000fe18df400 000000006eac6058 python35!PyFunction_SetAnnotations+0xb71
17 0000000fe18df4b0 000000006eb8a7cd python35!PyObject_Call+0x78
18 0000000fe18df4e0 000000006eb86c66 python35!PyEval_GetFuncDesc+0xd5d
19 0000000fe18df550 000000006eb89fe8 python35!PyEval_EvalFrameEx+0x32e6
1a 0000000fe18df650 000000006eb89ea1 python35!PyEval_GetFuncDesc+0x578
1b 0000000fe18df6f0 000000006eb86bc1 python35!PyEval_GetFuncDesc+0x431
1c 0000000fe18df750 000000006eb89fe8 python35!PyEval_EvalFrameEx+0x3241
1d 0000000fe18df850 000000006eb89ea1 python35!PyEval_GetFuncDesc+0x578
1e 0000000fe18df8f0 000000006eb86bc1 python35!PyEval_GetFuncDesc+0x431
1f 0000000fe18df950 000000006eb88a9e python35!PyEval_EvalFrameEx+0x3241
20 0000000fe18dfa50 000000006eaf8751 python35!PyEval_EvalFrameEx+0x511e
21 0000000fe18dfaf0 000000006eac6058 python35!PyFunction_SetAnnotations+0xb71
22 0000000fe18dfba0 000000006eadd82d python35!PyObject_Call+0x78
23 0000000fe18dfbd0 000000006eac6058 python35!PyMethod_New+0xa9d
24 0000000fe18dfc10 000000006eb899ec python35!PyObject_Call+0x78
25 0000000fe18dfc40 000000006ea89193 python35!PyEval_CallObjectWithKeywords+0xcc
26 0000000fe18dfc70 000000006ebde352 python35!PyOS_SigintEvent+0x1cb3
27 0000000fe18dfcb0 00007ffbbf3bcab0 python35!PyThread_init_thread+0x4b2
28 0000000fe18dfce0 00007ffbc1488364 ucrtbase!o__realloc_base+0x60
29 0000000fe18dfd10 00007ffbc2b270d1 kernel32!BaseThreadInitThunk+0x14
2a 0000000fe18dfd40<PHONE_NUMBER>000000 ntdll!RtlUserThreadStart+0x21 [d:\rs1\minkernel\ntdll\rtlstrt.c @ 997]
Child-SP          RetAddr           Call Site
00 0000000fe247f628 00007ffbc2b2cfb5 ntdll!ZwWaitForAlertByThreadId+0x14 [d:\rs1.obj.amd64fre\minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm @ 3731]
01 0000000fe247f630 00007ffbbf018d28 ntdll!RtlSleepConditionVariableSRW+0xf5 [d:\rs1\minkernel\ntos\rtl\condition.c @ 1109]
02 0000000fe247f6b0 00007ffbb6978368 KERNELBASE!SleepConditionVariableSRW+0x28 [d:\rs1\minkernel\kernelbase\synch.c @ 2020]
03 0000000fe247f6f0 00007ffbb6978530 msvcp140!std::_Syserror_map+0x198
04 0000000fe247f720 00007ffb971e89e7 msvcp140!Cnd_destroy+0x70
05 (Inline Function) ---------------- _pywrap_tensorflow!std::_Cnd_waitX+0xd [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 95] 06 (Inline Function) ---------------- _pywrap_tensorflow!std::condition_variable::wait+0xd [c:\program files (x86)\microsoft visual studio 14.0\vc\include\mutex @ 565]
07 0000000fe247f7a0 00007ffb971e80f9 _pywrap_tensorflow!Eigen::EventCount::Park+0x67 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\eventcount.h @ 208]
08 0000000fe247f7f0 00007ffb971e98a0 _pywrap_tensorflow!Eigen::EventCount::CommitWait+0xc9 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\eventcount.h @ 103]
09 0000000fe247f830 00007ffb971e9bc3 _pywrap_tensorflow!Eigen::NonBlockingThreadPoolTempltensorflow::thread::EigenEnvironment::WaitForWork+0x180 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\nonblockingthreadpool.h @ 233]
0a 0000000fe247f870 00007ffb971e9f6e _pywrap_tensorflow!Eigen::NonBlockingThreadPoolTempltensorflow::thread::EigenEnvironment::WorkerLoop+0x293 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\nonblockingthreadpool.h @ 160]
0b (Inline Function) ---------------- _pywrap_tensorflow!std::_Func_class<void>::operator()+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 279] 0c (Inline Function) ---------------- _pywrap_tensorflow!tensorflow::thread::EigenEnvironment::CreateThread::__l2::<lambda_3e5df898eaffef9b52307b4af9ee50ab>::operator()+0x2c [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\lib\core\threadpool.cc @ 57]
0d (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoker_functor::_Call+0x2c [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398] 0e (Inline Function) ---------------- _pywrap_tensorflow!std::invoke+0x2c [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466]
0f (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoke_ret+0x2c [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1484] 10 0000000fe247f8e0 00007ffb9721ce25 _pywrap_tensorflow!std::_Func_impl<<lambda_3e5df898eaffef9b52307b4af9ee50ab>,std::allocator<int>,void>::_Do_call+0x3e [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 212] 11 (Inline Function) ---------------- _pywrap_tensorflow!std::_Func_class::operator()+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 279]
12 (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoker_functor::_Call+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398] 13 (Inline Function) ---------------- _pywrap_tensorflow!std::invoke+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466]
14 (Inline Function) ---------------- _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Execute+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 240] 15 0000000fe247f920 00007ffb9721cab9 _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Run+0x85 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 247] 16 0000000fe247f960 00007ffbbf3bcab0 _pywrap_tensorflow!std::_Pad::_Call_func+0x9 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 210] 17 0000000fe247f990 00007ffbc1488364 ucrtbase!o__realloc_base+0x60 18 0000000fe247f9c0 00007ffbc2b270d1 kernel32!BaseThreadInitThunk+0x14 19 0000000fe247f9f0 00000000`00000000 ntdll!RtlUserThreadStart+0x21 [d:\rs1\minkernel\ntdll\rtlstrt.c @ 997]
Child-SP          RetAddr           Call Site
00 0000001c6cb7ec58 00007ffbc2b2cfb5 ntdll!ZwWaitForAlertByThreadId+0x14 [d:\rs1.obj.amd64fre\minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm @ 3731]
01 0000001c6cb7ec60 00007ffbbf018d28 ntdll!RtlSleepConditionVariableSRW+0xf5 [d:\rs1\minkernel\ntos\rtl\condition.c @ 1109]
02 0000001c6cb7ece0 00007ffbb6978368 KERNELBASE!SleepConditionVariableSRW+0x28 [d:\rs1\minkernel\kernelbase\synch.c @ 2020]
03 0000001c6cb7ed20 00007ffbb6978530 msvcp140!std::_Syserror_map+0x198
04 0000001c6cb7ed50 00007ffb973a6ecd msvcp140!Cnd_destroy+0x70
05 (Inline Function) ---------------- _pywrap_tensorflow!std::_Cnd_waitX+0xd [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 95] 06 (Inline Function) ---------------- _pywrap_tensorflow!std::condition_variable::wait+0xd [c:\program files (x86)\microsoft visual studio 14.0\vc\include\mutex @ 565]
07 0000001c6cb7edd0 00007ffb974f35e3 _pywrap_tensorflow!tensorflow::BlockingCounter::Wait+0x7d [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\lib\core\blocking_counter.h @ 53]
08 (Inline Function) ---------------- _pywrap_tensorflow!tensorflow::RunManyGraphs::Wait+0xc [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\master_session.cc @ 476] 09 0000001c6cb7ee20 00007ffb974f01c2 _pywrap_tensorflow!tensorflow::MasterSession::ReffedClientGraph::RunPartitions+0xb03 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\master_session.cc @ 622] 0a 0000001c6cb7f640 00007ffb974f29a3 _pywrap_tensorflow!tensorflow::MasterSession::DoRunWithLocalExecution+0x362 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\master_session.cc @ 1360] 0b 0000001c6cb7f880 00007ffb974dfabb _pywrap_tensorflow!tensorflow::MasterSession::Run+0xc3 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\master_session.cc @ 1161] 0c 0000001c6cb7f8f0 00007ffb9721ce25 _pywrap_tensorflow!<lambda_81a3b772d3ac2aa00a42b8da5cebc5eb>::operator()+0x3b [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\master.cc @ 325] 0d (Inline Function) ---------------- _pywrap_tensorflow!std::_Func_class::operator()+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 279]
0e (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoker_functor::_Call+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398] 0f (Inline Function) ---------------- _pywrap_tensorflow!std::invoke+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466]
10 (Inline Function) ---------------- _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Execute+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 240] 11 0000001c6cb7f950 00007ffb9721cab9 _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Run+0x85 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 247] 12 0000001c6cb7f990 00007ffbbf3bcab0 _pywrap_tensorflow!std::_Pad::_Call_func+0x9 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 210] 13 0000001c6cb7f9c0 00007ffbc1488364 ucrtbase!o__realloc_base+0x60 14 0000001c6cb7f9f0 00007ffbc2b270d1 kernel32!BaseThreadInitThunk+0x14 15 0000001c6cb7fa20 00000000`00000000 ntdll!RtlUserThreadStart+0x21 [d:\rs1\minkernel\ntdll\rtlstrt.c @ 997]

wow, I've never seen windows stack trace before, cc @mrry

form my experience, windows version is not very stable, i also ran into session run stuck when I enabled full tracing option with one custom operator built in, and setting timeout did not work, sometimes the first step could get stuck, but after i disabled full tracing everything is fine. but the script under linux is ok, no stuck happened.
and here is the stack:
Child-SP          RetAddr           Call Site
00 00000053c09bc8f8 00007ff9554423c3 ntdll!ZwRemoveIoCompletion+0xa [d:\blue.obj.amd64fre\minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm @ 238]
01 00000053c09bc900 00007ff90420fdd9 KERNELBASE!GetQueuedCompletionStatus+0x3f [d:\blue\minkernel\kernelbase\error.c @ 706]
02 00000053c09bc960 00007ff9041f93e6 _pywrap_tensorflow!grpc_iocp_work+0x69 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\iomgr\iocp_windows.c @ 87]
03 00000053c09bc9f0 00007ff9041d9466 _pywrap_tensorflow!grpc_pollset_work+0xb6 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\iomgr\pollset_windows.c @ 152]
04 00000053c09bcaa0 00007ff9041c7fa1 _pywrap_tensorflow!grpc_completion_queue_pluck+0x256 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\surface\completion_queue.c @ 510]
05 00000053c09bcbe0 00007ff9021a127b _pywrap_tensorflow!grpc::CoreCodegen::grpc_completion_queue_pluck+0x41 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\cpp\common\core_codegen.cc @ 66]
06 (Inline Function) ---------------- _pywrap_tensorflow!grpc::CompletionQueue::Pluck+0x53 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\include\grpc++\impl\codegen\completion_queue.h @ 207] 07 00000053c09bcc40 00007ff9021a5f96 _pywrap_tensorflow!grpc::BlockingUnaryCall<tensorflow::RunStepRequest,tensorflow::RunStepResponse>+0x32b [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\include\grpc++\impl\codegen\client_unary_call.h @ 72] 08 00000053c09bce60 00007ff9021a8af4 _pywrap_tensorflow!tensorflow::grpc::MasterService::Stub::RunStep+0x36 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\rpc\grpc_master_service_impl.cc @ 80] 09 00000053c09bceb0 00007ff9021c3fa4 _pywrap_tensorflow!tensorflow::GrpcRemoteMaster::RunStep+0xa4 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\rpc\grpc_remote_master.cc @ 60] 0a 00000053c09bd080 00007ff9021c387b _pywrap_tensorflow!tensorflow::GrpcSession::RunProto+0x124 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\rpc\grpc_session.cc @ 238] 0b 00000053c09bd0e0 00007ff901e4c0a0 _pywrap_tensorflow!tensorflow::GrpcSession::Run+0x4cb [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\rpc\grpc_session.cc @ 188] 0c 00000053c09bd410 00007ff901e58a7a _pywrap_tensorflow!TF_Run_Helper+0x1c0 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\c\c_api.cc @ 543] 0d 00000053c09bd5b0 00007ff901e2a509 _pywrap_tensorflow!TF_Run+0x1ba [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\c\c_api.cc @ 606] 0e 00000053c09bd680 00007ff901e29e1a _pywrap_tensorflow!tensorflow::TF_Run_wrapper_helper+0x6c9 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\python\client\tf_session_helper.cc @ 500] 0f 00000053c09bd950 00007ff901e607ac _pywrap_tensorflow!tensorflow::TF_Run_wrapper+0x4a [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\python\client\tf_session_helper.cc @ 550] 10 00000053c09bd9b0<PHONE_NUMBER>97e512 _pywrap_tensorflow!_wrap_TF_Run+0x40c [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\pywrap_tensorflow.cc @ 5862] 11 00000053c09bdbe0<PHONE_NUMBER>9f9e20 python35!PyCFunction_Call+0x122 12 00000053c09bdc10<PHONE_NUMBER>9f6bc1 python35!PyEval_GetFuncDesc+0x3b0 13 00000053c09bdc70<PHONE_NUMBER>9f8a9e python35!PyEval_EvalFrameEx+0x3241 14 00000053c09bdd70<PHONE_NUMBER>968751 python35!PyEval_EvalFrameEx+0x511e 15 00000053c09bde10<PHONE_NUMBER>936058 python35!PyFunction_SetAnnotations+0xb71 16 00000053c09bdec0<PHONE_NUMBER>9fa7cd python35!PyObject_Call+0x78 17 00000053c09bdef0<PHONE_NUMBER>9f6c66 python35!PyEval_GetFuncDesc+0xd5d 18 00000053c09bdf60<PHONE_NUMBER>9f8a9e python35!PyEval_EvalFrameEx+0x32e6 19 00000053c09be060<PHONE_NUMBER>9fa070 python35!PyEval_EvalFrameEx+0x511e 1a 00000053c09be100<PHONE_NUMBER>9f9ea1 python35!PyEval_GetFuncDesc+0x600 1b 00000053c09be1a0<PHONE_NUMBER>9f6bc1 python35!PyEval_GetFuncDesc+0x431 1c 00000053c09be200<PHONE_NUMBER>9f8a9e python35!PyEval_EvalFrameEx+0x3241 1d 00000053c09be300<PHONE_NUMBER>9fa070 python35!PyEval_EvalFrameEx+0x511e 1e 00000053c09be3a0<PHONE_NUMBER>9f9ea1 python35!PyEval_GetFuncDesc+0x600 1f 00000053c09be440<PHONE_NUMBER>9f6bc1 python35!PyEval_GetFuncDesc+0x431 20 00000053c09be4a0<PHONE_NUMBER>9f9fe8 python35!PyEval_EvalFrameEx+0x3241 21 00000053c09be5a0<PHONE_NUMBER>9f9ea1 python35!PyEval_GetFuncDesc+0x578 22 00000053c09be640<PHONE_NUMBER>9f6bc1 python35!PyEval_GetFuncDesc+0x431 23 00000053c09be6a0<PHONE_NUMBER>9f8a9e python35!PyEval_EvalFrameEx+0x3241 24 00000053c09be7a0<PHONE_NUMBER>9fa070 python35!PyEval_EvalFrameEx+0x511e 25 00000053c09be840<PHONE_NUMBER>9f9ea1 python35!PyEval_GetFuncDesc+0x600 26 00000053c09be8e0<PHONE_NUMBER>9f6bc1 python35!PyEval_GetFuncDesc+0x431 27 00000053c09be940<PHONE_NUMBER>9f8a9e python35!PyEval_EvalFrameEx+0x3241 28 00000053c09bea40<PHONE_NUMBER>968751 python35!PyEval_EvalFrameEx+0x511e 29 00000053c09beae0<PHONE_NUMBER>936058 python35!PyFunction_SetAnnotations+0xb71 2a 00000053c09beb90<PHONE_NUMBER>9fa7cd python35!PyObject_Call+0x78 2b 00000053c09bebc0<PHONE_NUMBER>9f6c66 python35!PyEval_GetFuncDesc+0xd5d 2c 00000053c09bec30<PHONE_NUMBER>9f9fe8 python35!PyEval_EvalFrameEx+0x32e6 2d 00000053c09bed30<PHONE_NUMBER>9f9ea1 python35!PyEval_GetFuncDesc+0x578 2e 00000053c09bedd0<PHONE_NUMBER>9f6bc1 python35!PyEval_GetFuncDesc+0x431 2f 00000053c09bee30<PHONE_NUMBER>9f9fe8 python35!PyEval_EvalFrameEx+0x3241 30 00000053c09bef30<PHONE_NUMBER>9f9ea1 python35!PyEval_GetFuncDesc+0x578 31 00000053c09befd0<PHONE_NUMBER>9f6bc1 python35!PyEval_GetFuncDesc+0x431 32 00000053c09bf030<PHONE_NUMBER>9f9fe8 python35!PyEval_EvalFrameEx+0x3241 33 00000053c09bf130<PHONE_NUMBER>9f9ea1 python35!PyEval_GetFuncDesc+0x578 34 00000053c09bf1d0<PHONE_NUMBER>9f6bc1 python35!PyEval_GetFuncDesc+0x431 35 00000053c09bf230<PHONE_NUMBER>9f8a9e python35!PyEval_EvalFrameEx+0x3241 36 00000053c09bf330<PHONE_NUMBER>9fa070 python35!PyEval_EvalFrameEx+0x511e 37 00000053c09bf3d0<PHONE_NUMBER>9f9ea1 python35!PyEval_GetFuncDesc+0x600 38 00000053c09bf470<PHONE_NUMBER>9f6bc1 python35!PyEval_GetFuncDesc+0x431 39 00000053c09bf4d0<PHONE_NUMBER>9f8a9e python35!PyEval_EvalFrameEx+0x3241 3a 00000053c09bf5d0<PHONE_NUMBER>a4423a python35!PyEval_EvalFrameEx+0x511e 3b 00000053c09bf670<PHONE_NUMBER>a43e85 python35!PyRun_FileExFlags+0x38a 3c 00000053c09bf6f0<PHONE_NUMBER>a42831 python35!PyRun_StringFlags+0xf5 3d 00000053c09bf750<PHONE_NUMBER>8cf119 python35!PyRun_SimpleStringFlags+0x41 3e 00000053c09bf790<PHONE_NUMBER>8cfbb3 python35!Py_hashtable_copy+0x70f9 3f 00000053c09bf7c0 000000001ca01264 python35!Py_Main+0x963 40 00000053c09bf8b0 00007ff9578f13d2 python+0x1264 41 00000053c09bf8f0 00007ff9581254e4 kernel32!BaseThreadInitThunk+0x22 [d:\9147\base\win32\client\thread.c @ 78] 42 00000053c09bf920 00000000`00000000 ntdll!RtlUserThreadStart+0x34 [d:\blue\minkernel\ntdll\rtlstrt.c @ 1026]
Child-SP          RetAddr           Call Site
00 00000053c61bf688 00007ff95818e65c ntdll!ZwWaitForAlertByThreadId+0xa [d:\blue.obj.amd64fre\minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm @ 3590]
01 00000053c61bf690 00007ff9554b8da8 ntdll!RtlSleepConditionVariableSRW+0xec [d:\blue\minkernel\ntdll\condition.c @ 1115]
02 00000053c61bf700 00007ff943158678 KERNELBASE!SleepConditionVariableSRW+0x28 [d:\blue\minkernel\kernelbase\synch.c @ 1904]
03 (Inline Function) ---------------- msvcp140!Concurrency::details::stl_condition_variable_win7::wait_for+0x14 [f:\dd\vctools\crt\crtw32\stdcpp\thr\primitives.h @ 216] 04 00000053c61bf740 00007ff943158840 msvcp140!Concurrency::details::stl_condition_variable_win7::wait+0x18 [f:\dd\vctools\crt\crtw32\stdcpp\thr\primitives.h @ 210] 05 00000053c61bf770 00007ff901e789e7 msvcp140!do_wait+0x4c [f:\dd\vctools\crt\crtw32\stdcpp\thr\cond.c @ 77] 06 (Inline Function) ---------------- _pywrap_tensorflow!std::_Cnd_waitX+0xd [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 95]
07 (Inline Function) ---------------- _pywrap_tensorflow!std::condition_variable::wait+0xd [c:\program files (x86)\microsoft visual studio 14.0\vc\include\mutex @ 565] 08 00000053c61bf7f0 00007ff901e780f9 _pywrap_tensorflow!Eigen::EventCount::Park+0x67 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\eventcount.h @ 208] 09 00000053c61bf840 00007ff901e798a0 _pywrap_tensorflow!Eigen::EventCount::CommitWait+0xc9 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\eventcount.h @ 103] 0a 00000053c61bf880 00007ff901e79bc3 _pywrap_tensorflow!Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork+0x180 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\nonblockingthreadpool.h @ 233] 0b 00000053c61bf8c0 00007ff901e79f6e _pywrap_tensorflow!Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop+0x293 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\nonblockingthreadpool.h @ 160] 0c (Inline Function) ---------------- _pywrap_tensorflow!std::_Func_class::operator()+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 279]
0d (Inline Function) ---------------- _pywrap_tensorflow!tensorflow::thread::EigenEnvironment::CreateThread::__l2::<lambda_3e5df898eaffef9b52307b4af9ee50ab>::operator()+0x2c [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\lib\core\threadpool.cc @ 57] 0e (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoker_functor::_Call+0x2c [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398]
0f (Inline Function) ---------------- _pywrap_tensorflow!std::invoke+0x2c [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466] 10 (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoke_ret+0x2c [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1484]
11 00000053c61bf930 00007ff901eace25 _pywrap_tensorflow!std::_Func_impl<<lambda_3e5df898eaffef9b52307b4af9ee50ab>,std::allocator,void>::_Do_call+0x3e [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 212]
12 (Inline Function) ---------------- _pywrap_tensorflow!std::_Func_class<void>::operator()+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 279] 13 (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoker_functor::_Call+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398]
14 (Inline Function) ---------------- _pywrap_tensorflow!std::invoke+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466] 15 (Inline Function) ---------------- _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Execute+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 240]
16 00000053c61bf970 00007ff901eacab9 _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Run+0x85 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 247]
17 00000053c61bf9b0 00007ff942ffbe1d _pywrap_tensorflow!std::_Pad::_Call_func+0x9 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 210]
18 (Inline Function) ---------------- ucrtbase!invoke_thread_procedure+0xe [d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 91] 19 00000053c61bf9e0 00007ff9578f13d2 ucrtbase!thread_start<unsigned int (__cdecl*)(void * __ptr64)>+0x5d [d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 115] 1a 00000053c61bfa10 00007ff9581254e4 kernel32!BaseThreadInitThunk+0x22 [d:\9147\base\win32\client\thread.c @ 78] 1b 00000053c61bfa40 00000000`00000000 ntdll!RtlUserThreadStart+0x34 [d:\blue\minkernel\ntdll\rtlstrt.c @ 1026]
Child-SP          RetAddr           Call Site
00 00000053c7e2f748 00007ff95817a192 ntdll!ZwWaitForAlertByThreadId+0xa [d:\blue.obj.amd64fre\minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm @ 3590]
01 00000053c7e2f750 00007ff955478178 ntdll!RtlSleepConditionVariableCS+0xc2 [d:\blue\minkernel\ntdll\condition.c @ 937]
02 00000053c7e2f7c0 00007ff904215d6e KERNELBASE!SleepConditionVariableCS+0x28 [d:\blue\minkernel\kernelbase\synch.c @ 1869]
03 00000053c7e2f7f0 00007ff9041f94a9 _pywrap_tensorflow!gpr_cv_wait+0x11e [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\support\sync_windows.c @ 103]
04 00000053c7e2f850 00007ff9041d8fb2 _pywrap_tensorflow!grpc_pollset_work+0x179 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\iomgr\pollset_windows.c @ 182]
05 00000053c7e2f900 00007ff9041c7cae _pywrap_tensorflow!grpc_completion_queue_next+0x1f2 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\surface\completion_queue.c @ 380]
06 00000053c7e2fa40 00007ff9021c59e1 _pywrap_tensorflow!grpc::CompletionQueue::AsyncNextInternal+0x3e [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\cpp\common\completion_queue.cc @ 56]
07 (Inline Function) ---------------- _pywrap_tensorflow!grpc::CompletionQueue::Next+0xa5 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\include\grpc++\impl\codegen\completion_queue.h @ 147] 08 00000053c7e2fa90 00007ff901eace25 _pywrap_tensorflow!<lambda_ea64bb373946421891242c9b912b7e25>::operator()+0xc1 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\rpc\grpc_worker_cache.cc @ 38] 09 (Inline Function) ---------------- _pywrap_tensorflow!std::_Func_class::operator()+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 279]
0a (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoker_functor::_Call+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398] 0b (Inline Function) ---------------- _pywrap_tensorflow!std::invoke+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466]
0c (Inline Function) ---------------- _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Execute+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 240] 0d 00000053c7e2fb00 00007ff901eacab9 _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Run+0x85 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 247] 0e 00000053c7e2fb40 00007ff942ffbe1d _pywrap_tensorflow!std::_Pad::_Call_func+0x9 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 210] 0f (Inline Function) ---------------- ucrtbase!invoke_thread_procedure+0xe [d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 91]
10 00000053c7e2fb70 00007ff9578f13d2 ucrtbase!thread_start<unsigned int (__cdecl*)(void * __ptr64)>+0x5d [d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 115]
11 00000053c7e2fba0 00007ff9581254e4 kernel32!BaseThreadInitThunk+0x22 [d:\9147\base\win32\client\thread.c @ 78]
12 00000053c7e2fbd0<PHONE_NUMBER>000000 ntdll!RtlUserThreadStart+0x34 [d:\blue\minkernel\ntdll\rtlstrt.c @ 1026]
Child-SP          RetAddr           Call Site
00 00000053c801f958 00007ff95818e65c ntdll!ZwWaitForAlertByThreadId+0xa [d:\blue.obj.amd64fre\minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm @ 3590]
01 00000053c801f960 00007ff9554b8da8 ntdll!RtlSleepConditionVariableSRW+0xec [d:\blue\minkernel\ntdll\condition.c @ 1115]
02 00000053c801f9d0 00007ff943158678 KERNELBASE!SleepConditionVariableSRW+0x28 [d:\blue\minkernel\kernelbase\synch.c @ 1904]
03 (Inline Function) ---------------- msvcp140!Concurrency::details::stl_condition_variable_win7::wait_for+0x14 [f:\dd\vctools\crt\crtw32\stdcpp\thr\primitives.h @ 216] 04 00000053c801fa10 00007ff943158840 msvcp140!Concurrency::details::stl_condition_variable_win7::wait+0x18 [f:\dd\vctools\crt\crtw32\stdcpp\thr\primitives.h @ 210] 05 00000053c801fa40 00007ff901e789e7 msvcp140!do_wait+0x4c [f:\dd\vctools\crt\crtw32\stdcpp\thr\cond.c @ 77] 06 (Inline Function) ---------------- _pywrap_tensorflow!std::_Cnd_waitX+0xd [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 95]
07 (Inline Function) ---------------- _pywrap_tensorflow!std::condition_variable::wait+0xd [c:\program files (x86)\microsoft visual studio 14.0\vc\include\mutex @ 565] 08 00000053c801fac0 00007ff901e780f9 _pywrap_tensorflow!Eigen::EventCount::Park+0x67 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\eventcount.h @ 208] 09 00000053c801fb10 00007ff901e798a0 _pywrap_tensorflow!Eigen::EventCount::CommitWait+0xc9 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\eventcount.h @ 103] 0a 00000053c801fb50 00007ff901e79bc3 _pywrap_tensorflow!Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork+0x180 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\nonblockingthreadpool.h @ 233] 0b 00000053c801fb90 00007ff901e79f6e _pywrap_tensorflow!Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop+0x293 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\nonblockingthreadpool.h @ 160] 0c (Inline Function) ---------------- _pywrap_tensorflow!std::_Func_class::operator()+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 279]
0d (Inline Function) ---------------- _pywrap_tensorflow!tensorflow::thread::EigenEnvironment::CreateThread::__l2::<lambda_3e5df898eaffef9b52307b4af9ee50ab>::operator()+0x2c [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\lib\core\threadpool.cc @ 57] 0e (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoker_functor::_Call+0x2c [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398]
0f (Inline Function) ---------------- _pywrap_tensorflow!std::invoke+0x2c [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466] 10 (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoke_ret+0x2c [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1484]
11 00000053c801fc00 00007ff901eace25 _pywrap_tensorflow!std::_Func_impl<<lambda_3e5df898eaffef9b52307b4af9ee50ab>,std::allocator,void>::_Do_call+0x3e [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 212]
12 (Inline Function) ---------------- _pywrap_tensorflow!std::_Func_class<void>::operator()+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 279] 13 (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoker_functor::_Call+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398]
14 (Inline Function) ---------------- _pywrap_tensorflow!std::invoke+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466] 15 (Inline Function) ---------------- _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Execute+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 240]
16 00000053c801fc40 00007ff901eacab9 _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Run+0x85 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 247]
17 00000053c801fc80 00007ff942ffbe1d _pywrap_tensorflow!std::_Pad::_Call_func+0x9 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 210]
18 (Inline Function) ---------------- ucrtbase!invoke_thread_procedure+0xe [d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 91] 19 00000053c801fcb0 00007ff9578f13d2 ucrtbase!thread_start<unsigned int (__cdecl*)(void * __ptr64)>+0x5d [d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 115] 1a 00000053c801fce0 00007ff9581254e4 kernel32!BaseThreadInitThunk+0x22 [d:\9147\base\win32\client\thread.c @ 78] 1b 00000053c801fd10 00000000`00000000 ntdll!RtlUserThreadStart+0x34 [d:\blue\minkernel\ntdll\rtlstrt.c @ 1026]
Child-SP          RetAddr           Call Site
00 00000053c820f3e8 00007ff95817a192 ntdll!ZwWaitForAlertByThreadId+0xa [d:\blue.obj.amd64fre\minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm @ 3590]
01 00000053c820f3f0 00007ff955478178 ntdll!RtlSleepConditionVariableCS+0xc2 [d:\blue\minkernel\ntdll\condition.c @ 937]
02 00000053c820f460 00007ff904215d6e KERNELBASE!SleepConditionVariableCS+0x28 [d:\blue\minkernel\kernelbase\synch.c @ 1869]
03 00000053c820f490 00007ff9041f94a9 _pywrap_tensorflow!gpr_cv_wait+0x11e [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\support\sync_windows.c @ 103]
04 00000053c820f4f0 00007ff9041d8fb2 _pywrap_tensorflow!grpc_pollset_work+0x179 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\iomgr\pollset_windows.c @ 182]
05 00000053c820f5a0 00007ff9041c7cae _pywrap_tensorflow!grpc_completion_queue_next+0x1f2 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\surface\completion_queue.c @ 380]
06 00000053c820f6e0 00007ff90219a8da _pywrap_tensorflow!grpc::CompletionQueue::AsyncNextInternal+0x3e [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\cpp\common\completion_queue.cc @ 56]
07 (Inline Function) ---------------- _pywrap_tensorflow!grpc::CompletionQueue::Next+0xe1 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\include\grpc++\impl\codegen\completion_queue.h @ 147] 08 00000053c820f730 00007ff901eace25 _pywrap_tensorflow!tensorflow::GrpcMasterService::HandleRPCsLoop+0x56a [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\rpc\grpc_master_service.cc @ 116] 09 (Inline Function) ---------------- _pywrap_tensorflow!std::_Func_class::operator()+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 279]
0a (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoker_functor::_Call+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398] 0b (Inline Function) ---------------- _pywrap_tensorflow!std::invoke+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466]
0c (Inline Function) ---------------- _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Execute+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 240] 0d 00000053c820f840 00007ff901eacab9 _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Run+0x85 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 247] 0e 00000053c820f880 00007ff942ffbe1d _pywrap_tensorflow!std::_Pad::_Call_func+0x9 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 210] 0f (Inline Function) ---------------- ucrtbase!invoke_thread_procedure+0xe [d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 91]
10 00000053c820f8b0 00007ff9578f13d2 ucrtbase!thread_start<unsigned int (__cdecl*)(void * __ptr64)>+0x5d [d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 115]
11 00000053c820f8e0 00007ff9581254e4 kernel32!BaseThreadInitThunk+0x22 [d:\9147\base\win32\client\thread.c @ 78]
12 00000053c820f910<PHONE_NUMBER>000000 ntdll!RtlUserThreadStart+0x34 [d:\blue\minkernel\ntdll\rtlstrt.c @ 1026]
Child-SP          RetAddr           Call Site
00 00000053c83ff9e8 00007ff95817a192 ntdll!ZwWaitForAlertByThreadId+0xa [d:\blue.obj.amd64fre\minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm @ 3590]
01 00000053c83ff9f0 00007ff955478178 ntdll!RtlSleepConditionVariableCS+0xc2 [d:\blue\minkernel\ntdll\condition.c @ 937]
02 00000053c83ffa60 00007ff904215d6e KERNELBASE!SleepConditionVariableCS+0x28 [d:\blue\minkernel\kernelbase\synch.c @ 1869]
03 00000053c83ffa90 00007ff9041f94a9 _pywrap_tensorflow!gpr_cv_wait+0x11e [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\support\sync_windows.c @ 103]
04 00000053c83ffaf0 00007ff9041d8fb2 _pywrap_tensorflow!grpc_pollset_work+0x179 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\iomgr\pollset_windows.c @ 182]
05 00000053c83ffba0 00007ff9041c7cae _pywrap_tensorflow!grpc_completion_queue_next+0x1f2 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\core\lib\surface\completion_queue.c @ 380]
06 00000053c83ffce0 00007ff9021d189b _pywrap_tensorflow!grpc::CompletionQueue::AsyncNextInternal+0x3e [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\src\cpp\common\completion_queue.cc @ 56]
07 (Inline Function) ---------------- _pywrap_tensorflow!grpc::CompletionQueue::Next+0xed [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\grpc\src\grpc\include\grpc++\impl\codegen\completion_queue.h @ 147] 08 00000053c83ffd30 00007ff901eace25 _pywrap_tensorflow!tensorflow::anonymous namespace'::GrpcWorkerService::HandleRPCsLoop+0x6cb [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\rpc\grpc_worker_service.cc @ 145]
09 (Inline Function) ---------------- _pywrap_tensorflow!std::_Func_class<void>::operator()+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 279] 0a (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoker_functor::_Call+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398]
0b (Inline Function) ---------------- _pywrap_tensorflow!std::invoke+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466] 0c (Inline Function) ---------------- _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Execute+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 240]
0d 00000053c83ffe70 00007ff901eacab9 _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Run+0x85 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 247]
0e 00000053c83ffeb0 00007ff942ffbe1d _pywrap_tensorflow!std::_Pad::_Call_func+0x9 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 210]
0f (Inline Function) ---------------- ucrtbase!invoke_thread_procedure+0xe [d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 91] 10 00000053c83ffee0 00007ff9578f13d2 ucrtbase!thread_start<unsigned int (__cdecl*)(void * __ptr64)>+0x5d [d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 115] 11 00000053c83fff10 00007ff9581254e4 kernel32!BaseThreadInitThunk+0x22 [d:\9147\base\win32\client\thread.c @ 78] 12 00000053c83fff40 00000000`00000000 ntdll!RtlUserThreadStart+0x34 [d:\blue\minkernel\ntdll\rtlstrt.c @ 1026]
Child-SP          RetAddr           Call Site
00 00000053c8e5e408 00007ff955441118 ntdll!ZwWaitForSingleObject+0xa [d:\blue.obj.amd64fre\minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm @ 198]
01 00000053c8e5e410<PHONE_NUMBER>a4e1bc KERNELBASE!WaitForSingleObjectEx+0x94 [d:\blue\minkernel\kernelbase\synch.c @ 1301]
02 00000053c8e5e4b0<PHONE_NUMBER>a4e56e python35!PyThread_init_thread+0x31c
03 00000053c8e5e4f0<PHONE_NUMBER>8f7df7 python35!PyThread_acquire_lock_timed+0x6e
04 00000053c8e5e520<PHONE_NUMBER>8f8099 python35!PyOS_SigintEvent+0x917
05 00000053c8e5e570<PHONE_NUMBER>97e420 python35!PyOS_SigintEvent+0xbb9
06 00000053c8e5e5a0<PHONE_NUMBER>9f9e20 python35!PyCFunction_Call+0x30
07 00000053c8e5e5d0<PHONE_NUMBER>9f6bc1 python35!PyEval_GetFuncDesc+0x3b0
08 00000053c8e5e630<PHONE_NUMBER>9f8a9e python35!PyEval_EvalFrameEx+0x3241
09 00000053c8e5e730<PHONE_NUMBER>9fa070 python35!PyEval_EvalFrameEx+0x511e
0a 00000053c8e5e7d0<PHONE_NUMBER>9f9ea1 python35!PyEval_GetFuncDesc+0x600
0b 00000053c8e5e870<PHONE_NUMBER>9f6bc1 python35!PyEval_GetFuncDesc+0x431
0c 00000053c8e5e8d0<PHONE_NUMBER>9f8a9e python35!PyEval_EvalFrameEx+0x3241
0d 00000053c8e5e9d0<PHONE_NUMBER>9fa070 python35!PyEval_EvalFrameEx+0x511e
0e 00000053c8e5ea70<PHONE_NUMBER>9f9ea1 python35!PyEval_GetFuncDesc+0x600
0f 00000053c8e5eb10<PHONE_NUMBER>9f6bc1 python35!PyEval_GetFuncDesc+0x431
10 00000053c8e5eb70<PHONE_NUMBER>9f8a9e python35!PyEval_EvalFrameEx+0x3241
11 00000053c8e5ec70<PHONE_NUMBER>9fa070 python35!PyEval_EvalFrameEx+0x511e
12 00000053c8e5ed10<PHONE_NUMBER>9f9ea1 python35!PyEval_GetFuncDesc+0x600
13 00000053c8e5edb0<PHONE_NUMBER>9f6bc1 python35!PyEval_GetFuncDesc+0x431
14 00000053c8e5ee10<PHONE_NUMBER>9f8a9e python35!PyEval_EvalFrameEx+0x3241
15 00000053c8e5ef10<PHONE_NUMBER>968751 python35!PyEval_EvalFrameEx+0x511e
16 00000053c8e5efb0<PHONE_NUMBER>936058 python35!PyFunction_SetAnnotations+0xb71
17 00000053c8e5f060<PHONE_NUMBER>9fa7cd python35!PyObject_Call+0x78
18 00000053c8e5f090<PHONE_NUMBER>9f6c66 python35!PyEval_GetFuncDesc+0xd5d
19 00000053c8e5f100<PHONE_NUMBER>9f9fe8 python35!PyEval_EvalFrameEx+0x32e6
1a 00000053c8e5f200<PHONE_NUMBER>9f9ea1 python35!PyEval_GetFuncDesc+0x578
1b 00000053c8e5f2a0<PHONE_NUMBER>9f6bc1 python35!PyEval_GetFuncDesc+0x431
1c 00000053c8e5f300<PHONE_NUMBER>9f9fe8 python35!PyEval_EvalFrameEx+0x3241
1d 00000053c8e5f400<PHONE_NUMBER>9f9ea1 python35!PyEval_GetFuncDesc+0x578
1e 00000053c8e5f4a0<PHONE_NUMBER>9f6bc1 python35!PyEval_GetFuncDesc+0x431
1f 00000053c8e5f500<PHONE_NUMBER>9f8a9e python35!PyEval_EvalFrameEx+0x3241
20 00000053c8e5f600<PHONE_NUMBER>968751 python35!PyEval_EvalFrameEx+0x511e
21 00000053c8e5f6a0<PHONE_NUMBER>936058 python35!PyFunction_SetAnnotations+0xb71
22 00000053c8e5f750<PHONE_NUMBER>94d82d python35!PyObject_Call+0x78
23 00000053c8e5f780<PHONE_NUMBER>936058 python35!PyMethod_New+0xa9d
24 00000053c8e5f7c0<PHONE_NUMBER>9f99ec python35!PyObject_Call+0x78
25 00000053c8e5f7f0<PHONE_NUMBER>8f9193 python35!PyEval_CallObjectWithKeywords+0xcc
26 00000053c8e5f820<PHONE_NUMBER>a4e352 python35!PyOS_SigintEvent+0x1cb3
27 00000053c8e5f860 00007ff942ffbe1d python35!PyThread_init_thread+0x4b2
28 (Inline Function) ---------------- ucrtbase!invoke_thread_procedure+0xe [d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 91] 29 00000053c8e5f890 00007ff9578f13d2 ucrtbase!thread_start<unsigned int (__cdecl*)(void * __ptr64)>+0x5d [d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 115] 2a 00000053c8e5f8c0 00007ff9581254e4 kernel32!BaseThreadInitThunk+0x22 [d:\9147\base\win32\client\thread.c @ 78] 2b 00000053c8e5f8f0 00000000`00000000 ntdll!RtlUserThreadStart+0x34 [d:\blue\minkernel\ntdll\rtlstrt.c @ 1026]
Child-SP          RetAddr           Call Site
00 00000053c942f688 00007ff95818e65c ntdll!ZwWaitForAlertByThreadId+0xa [d:\blue.obj.amd64fre\minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm @ 3590]
01 00000053c942f690 00007ff9554b8da8 ntdll!RtlSleepConditionVariableSRW+0xec [d:\blue\minkernel\ntdll\condition.c @ 1115]
02 00000053c942f700 00007ff943158678 KERNELBASE!SleepConditionVariableSRW+0x28 [d:\blue\minkernel\kernelbase\synch.c @ 1904]
03 (Inline Function) ---------------- msvcp140!Concurrency::details::stl_condition_variable_win7::wait_for+0x14 [f:\dd\vctools\crt\crtw32\stdcpp\thr\primitives.h @ 216] 04 00000053c942f740 00007ff943158840 msvcp140!Concurrency::details::stl_condition_variable_win7::wait+0x18 [f:\dd\vctools\crt\crtw32\stdcpp\thr\primitives.h @ 210] 05 00000053c942f770 00007ff901e789e7 msvcp140!do_wait+0x4c [f:\dd\vctools\crt\crtw32\stdcpp\thr\cond.c @ 77] 06 (Inline Function) ---------------- _pywrap_tensorflow!std::_Cnd_waitX+0xd [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 95]
07 (Inline Function) ---------------- _pywrap_tensorflow!std::condition_variable::wait+0xd [c:\program files (x86)\microsoft visual studio 14.0\vc\include\mutex @ 565] 08 00000053c942f7f0 00007ff901e780f9 _pywrap_tensorflow!Eigen::EventCount::Park+0x67 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\eventcount.h @ 208] 09 00000053c942f840 00007ff901e798a0 _pywrap_tensorflow!Eigen::EventCount::CommitWait+0xc9 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\eventcount.h @ 103] 0a 00000053c942f880 00007ff901e79bc3 _pywrap_tensorflow!Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WaitForWork+0x180 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\nonblockingthreadpool.h @ 233] 0b 00000053c942f8c0 00007ff901e79f6e _pywrap_tensorflow!Eigen::NonBlockingThreadPoolTempl<tensorflow::thread::EigenEnvironment>::WorkerLoop+0x293 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\contrib\cmake\build\external\eigen_archive\unsupported\eigen\cxx11\src\threadpool\nonblockingthreadpool.h @ 160] 0c (Inline Function) ---------------- _pywrap_tensorflow!std::_Func_class::operator()+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 279]
0d (Inline Function) ---------------- _pywrap_tensorflow!tensorflow::thread::EigenEnvironment::CreateThread::__l2::<lambda_3e5df898eaffef9b52307b4af9ee50ab>::operator()+0x2c [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\lib\core\threadpool.cc @ 57] 0e (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoker_functor::_Call+0x2c [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398]
0f (Inline Function) ---------------- _pywrap_tensorflow!std::invoke+0x2c [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466] 10 (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoke_ret+0x2c [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1484]
11 00000053c942f930 00007ff901eace25 _pywrap_tensorflow!std::_Func_impl<<lambda_3e5df898eaffef9b52307b4af9ee50ab>,std::allocator,void>::_Do_call+0x3e [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 212]
12 (Inline Function) ---------------- _pywrap_tensorflow!std::_Func_class<void>::operator()+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 279] 13 (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoker_functor::_Call+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398]
14 (Inline Function) ---------------- _pywrap_tensorflow!std::invoke+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466] 15 (Inline Function) ---------------- _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Execute+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 240]
16 00000053c942f970 00007ff901eacab9 _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Run+0x85 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 247]
17 00000053c942f9b0 00007ff942ffbe1d _pywrap_tensorflow!std::_Pad::_Call_func+0x9 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 210]
18 (Inline Function) ---------------- ucrtbase!invoke_thread_procedure+0xe [d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 91] 19 00000053c942f9e0 00007ff9578f13d2 ucrtbase!thread_start<unsigned int (__cdecl*)(void * __ptr64)>+0x5d [d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 115] 1a 00000053c942fa10 00007ff9581254e4 kernel32!BaseThreadInitThunk+0x22 [d:\9147\base\win32\client\thread.c @ 78] 1b 00000053c942fa40 00000000`00000000 ntdll!RtlUserThreadStart+0x34 [d:\blue\minkernel\ntdll\rtlstrt.c @ 1026]
Child-SP          RetAddr           Call Site
00 00000053c8c6f5b8 00007ff95818e65c ntdll!ZwWaitForAlertByThreadId+0xa [d:\blue.obj.amd64fre\minkernel\ntdll\daytona\objfre\amd64\usrstubs.asm @ 3590]
01 00000053c8c6f5c0 00007ff9554b8da8 ntdll!RtlSleepConditionVariableSRW+0xec [d:\blue\minkernel\ntdll\condition.c @ 1115]
02 00000053c8c6f630 00007ff943158678 KERNELBASE!SleepConditionVariableSRW+0x28 [d:\blue\minkernel\kernelbase\synch.c @ 1904]
03 (Inline Function) ---------------- msvcp140!Concurrency::details::stl_condition_variable_win7::wait_for+0x14 [f:\dd\vctools\crt\crtw32\stdcpp\thr\primitives.h @ 216] 04 00000053c8c6f670 00007ff943158840 msvcp140!Concurrency::details::stl_condition_variable_win7::wait+0x18 [f:\dd\vctools\crt\crtw32\stdcpp\thr\primitives.h @ 210] 05 00000053c8c6f6a0 00007ff9021827be msvcp140!do_wait+0x4c [f:\dd\vctools\crt\crtw32\stdcpp\thr\cond.c @ 77] 06 (Inline Function) ---------------- _pywrap_tensorflow!std::_Cnd_waitX+0xe [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 95]
07 (Inline Function) ---------------- _pywrap_tensorflow!std::condition_variable::wait+0xe [c:\program files (x86)\microsoft visual studio 14.0\vc\include\mutex @ 565] 08 (Inline Function) ---------------- _pywrap_tensorflow!tensorflow::BlockingCounter::Wait+0x6e [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\lib\core\blocking_counter.h @ 53]
09 00000053c8c6f720 00007ff902181c62 _pywrap_tensorflow!tensorflow::MasterSession::ReffedClientGraph::RetrieveLogs+0x1fe [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\master_session.cc @ 168]
0a 00000053c8c6f950 00007ff902180280 _pywrap_tensorflow!tensorflow::MasterSession::ReffedClientGraph::ProcessStats+0x62 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\master_session.cc @ 741]
0b 00000053c8c6f9f0 00007ff9021829a3 _pywrap_tensorflow!tensorflow::MasterSession::DoRunWithLocalExecution+0x420 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\master_session.cc @ 1370]
0c 00000053c8c6fc30 00007ff90216fabb _pywrap_tensorflow!tensorflow::MasterSession::Run+0xc3 [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\master_session.cc @ 1161]
0d 00000053c8c6fca0 00007ff901eace25 _pywrap_tensorflow!<lambda_81a3b772d3ac2aa00a42b8da5cebc5eb>::operator()+0x3b [d:\tensorflow\tensorscripts\src\tensorflow\tensorflow\core\distributed_runtime\master.cc @ 325]
0e (Inline Function) ---------------- _pywrap_tensorflow!std::_Func_class<void>::operator()+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\functional @ 279] 0f (Inline Function) ---------------- _pywrap_tensorflow!std::_Invoker_functor::_Call+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1398]
10 (Inline Function) ---------------- _pywrap_tensorflow!std::invoke+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\type_traits @ 1466] 11 (Inline Function) ---------------- _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Execute+0x16 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 240]
12 00000053c8c6fd00 00007ff901eacab9 _pywrap_tensorflow!std::_LaunchPad<std::unique_ptr<std::tuple<std::function<void __cdecl(void)> >,std::default_delete<std::tuple<std::function<void __cdecl(void)> > > > >::_Run+0x85 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 247]
13 00000053c8c6fd40 00007ff942ffbe1d _pywrap_tensorflow!std::_Pad::_Call_func+0x9 [c:\program files (x86)\microsoft visual studio 14.0\vc\include\thr\xthread @ 210]
14 (Inline Function) ---------------- ucrtbase!invoke_thread_procedure+0xe [d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 91] 15 00000053c8c6fd70 00007ff9578f13d2 ucrtbase!thread_start<unsigned int (__cdecl*)(void * __ptr64)>+0x5d [d:\th\minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 115] 16 00000053c8c6fda0 00007ff9581254e4 kernel32!BaseThreadInitThunk+0x22 [d:\9147\base\win32\client\thread.c @ 78] 17 00000053c8c6fdd0 00000000`00000000 ntdll!RtlUserThreadStart+0x34 [d:\blue\minkernel\ntdll\rtlstrt.c @ 1026]

Something weird is that many workers ran into session timeout just before ending. For example, one cluster has 800 worker processes, after 127 workers successfully ended training and no timeout error occurred, all other workers started to ran into timeout and no worker could successfully finish even one step. After master worker finished reading data and started to save checkpoint, it will hang forever and master worker memory usage is about 2G.
Something important information:

when building up cluster, each worker will only see all parameter servers and is not aware of all other workers, this could avoiding establishing too many tcp connections between workers
all non-master workers will wait for exit signal variable using the same session for training when they finished processing their data.

My guess is parameter server is something wrong and can not response any more, so training will timeout and saver will hang forever without any timeout provided. Also 127 is very interesting, could be any 128 limiting?

@mrry Looks like all the threads in that stack trace are waiting, so presumably the problem is elsewhere.  Do you have recommendations for what extra information to get?

@girving @mrry  I changed my script to write checkpoint to hdfs with sharded=True, and found one of the parameter server failed to output checkpoint, so I am pretty sure worker timeout and saver stuck were caused by this one parameter server.here is the stack:
stack.txt

@girving @mrry hey guys, are you taking care of this issue? if you have no time, please give some advice and i will try my best to debug it.

@fesun Thanks for sending over the thread stacks. Unfortunately, there doesn't seem to be anything informative in the parameter server stack... all threads appear to be blocked waiting for more work (or in tf.Server.join()).
The most important observation seems to be that creating a tf.train.Saver with sharded=True is a more stable configuration than sharded=False, especially when the total parameter size is large. You mentioned that you saw a failure when writing with sharded=True to HDFS, so it's possible that there's a bug in the HDFS layer, and it might be worth instrumenting that code to see if an unhandled failure is occurring. Does "failed to output checkpoint" mean "saver.save() returned with no exception and yet it didn't write a file it should have"?
[As an aside, it looks like I might have found a bug in SaveV2, which doesn't check the error status from writer.Add(). This could potentially lead to a silent failure if the underlying write fails. @concretevitamin, can you take a look at this and confirm?]
Beyond that, I don't think we know enough about your program to know where the problem might lie. There are a lot of details—surrounding the setup of the parameters and the saver, the coordination between the workers, and the use of timeouts—that are unclear to me. Can you create a minimal example that exhibits the same problems and share the code for that? With some code in black-and-white, we should be able to provide more help with tracking down the problem.

@mrry thanks for your responses.
First, I used shared=True and wrote checkpoint to hdfs, and found only one parameter server had no output without any error occurred, and as you said, the ps stacks are all waiting for more work. Also before saving checkpoint, all workers were timeout even with 100 seconds timeout. I am pretty sure that one parameter server can't response any more.
Second, each worker is not aware of other workers, they only connect to ps. Each step used the same timeout(2 seconds)
Third, when the total worker count is small like 200, it works fine with or without shard=True, but when the worker count is 800 or even 400, cluster will hang with or without shard=True when the training is almost done. I am using windows version and even I provided code you may not reproduce the issue using very small cluster. But I will try to minimize it.

First, I used shared=True and wrote checkpoint to hdfs, and found only one parameter server had no output without any error occurred, and as you said, the ps stacks are all waiting for more work.

Is it always the same PS task that fails to write a checkpoint? Does it fail on the first attempt to write a checkpoint, or does it fail non-deterministically at some point during training?

Also before saving checkpoint, all workers were timeout even with 100 seconds timeout.

What were the workers doing that timed out?

I am pretty sure that one parameter server can't response any more.

How did you confirm this? Can you try running a step on that PS task, e.g. to fetch a single variable that's held on that task?

Second, each worker is not aware of other workers, they only connect to ps.

Which mechanism do you use for this? (By default, all workers know about all PS tasks and all other worker tasks when you set up the tf.train.ClusterSpec.)

Each step used the same timeout(2 seconds)

Does this include steps that read from a queue-based input pipeline? Or steps that save a checkpoint? It's possible that, if the queues are mostly empty, this timeout could be leaving behind a lot of cancelled attempts in the queue data structures (which are more efficient in the case where timeouts are rare).

Third, when the total worker count is small like 200, it works fine with or without shard=True, but when the worker count is 800 or even 400, cluster will hang with or without shard=True when the training is almost done.

This is strange, because I'd expect the reliability of tf.train.Saver(sharded=True).save() to be a function of the number of PS tasks, and not affected by the number of workers.
(Out of curiosity, why are you using multiple workers per machine?)

I am using windows version and even I provided code you may not reproduce the issue using very small cluster. But I will try to minimize it.

This is true... I don't have easy access to a dedicated cluster of hundreds of Windows machines :). Hopefully we can work with you to figure out the problem and then patch it.

@mrry I am quite interested in the TensorFlow cluster scale in Google. What is the largest number of TensorFlow worker process have you Google guys used for model training? And what is the upper bound of QPS can a parameter server handle? Your experience may help our deciding machine count.

@shishaochen

Google uses stubby while open-source version uses grpc which still has some obvious performance problems (https://github.com/tensorflow/tensorflow/issues/6116). I've been told that Google plans to transition to gRPC internally, so that may eliminate the gap eventually.

When I trained networks in Google and outside I have not seen communication (minus the gRPC issues) being the main bottleneck. The reason is that distributed training tends to be done using parameter server architecture which scales badly for reasons unrelated to hardware. You can actually hurt your accuracy by increasing hardware resources.

For asynchronous updates, imagine having 50 replicas updating parameters asynchronously. This means each replica sees parameters which are stale by 49 steps. As you increase number of workers, the staleness increases even if network communication is instant. One ImageNet experiment I've seen attained best accuracy for 50 replicas, and then accuracy went down after that. Synchronous training does better for same replica count, but exhibits similar diminishing returns as you increase number of replicas.
You may have better scaling using model parallelism, but this means using larger model, and larger models take longer to train, which is too long given that some of the smaller models already get trained for 6 months. For the largest image models/data-sizes, I would say a few hundred GPU machines is an upper bound on the amount of resources you can efficiently utilize using parameter server.

@yaroslavvb Much thanks for your detailed explanation. As there are many idle machines without GPU, we want to leverage them for distributed training with CPU only.

@mrry

Second, each worker is not aware of other workers, they only connect to ps.
Which mechanism do you use for this? (By default, all workers know about all PS tasks and all other worker tasks when you set up the tf.train.ClusterSpec.)

A: only provide all ps servers and itself when starting worker server:
cluster = tf.train.ClusterSpec({"ps": ps_hosts, "worker": {task_index: worker_hosts[task_index]}}), also tensorflow provide device_filters when constructing ConfigProto:
tf.ConfigProto(device_filters=["/job:ps", "/job:worker/task:%d" %  LAGS.task_index])

Does this include steps that read from a queue-based input pipeline? Or steps that save a checkpoint? It's possible that, if the queues are mostly empty, this timeout could be leaving behind a lot of cancelled attempts in the queue data structures (which are more efficient in the case where timeouts are rare).

A: the only queue I was using is file name queue, and here is the data reading code:
filename_queuetf.train.string_input_producer([data_file_name], num_epochs=FLAGS.iteration)
_, batch_example = tf.TFRecordReader().read_up_to(filename_queue, batch_size)
my_data = user_ops.my_own_parsing_operator(batch_example)

So I am leveraging tfrecord to read data from file, parse it using my own parser after getting raw sample data. I don't think it's queue issue.

(Out of curiosity, why are you using multiple workers per machine?)

A: My neural network is very simple, it's not CPU bottleneck, increasing worker count will better utilize resources and increase throughput. I also tried python multi-thread version by using the same session and different session each thread, but didn't work well, multi-thread can not scale out well.
I will reproduce this issue and dump tcp packet to see if I can find something. Will update later.

@mrry
I changed the code in tensorflow/core/distributed_runtime/rpc/grpc_worker_service.cc:Line124 from:
    // number of each request type.
    for (int i = 0; i < 1000; ++i) {
      EnqueueRecvTensorRequestRaw();
    }
    for (int i = 0; i < 100; ++i) {
      ENQUEUE_REQUEST(RunGraph, true);
    }
    for (int i = 0; i < 100; ++i) {
      ENQUEUE_REQUEST(CleanupGraph, false);
    }

to
    for (int i = 0; i < 5000; ++i) {
      EnqueueRecvTensorRequestRaw();
    }
    for (int i = 0; i < 2000; ++i) {
      ENQUEUE_REQUEST(RunGraph, true);
    }
    for (int i = 0; i < 2000; ++i) {
      ENQUEUE_REQUEST(CleanupGraph, false);
    }

This change solved the hang issue. Now with total 400 workers, no timeout error occurred and writing checkpoint finished successfully, I verified it by four running.
I don't understand why this could solve this problem, is there any other performance issues when the cluster is large?
here is the skeleton of worker code:
build graph and create session using tf supervisor
......
try:
    while not sv.should_stop():
        sess.run()
except tf.errors.OutOfRangeError:
    print("End of file")

# increase finish_count variable by one
sess.run(increase_finish_count_op)

if is_master_worker:
    # wait for all workers finish
    while True:
        if sess.run(finish_count_variable) < total_worker_count:
            time.sleep(30)
        else:
            break
    # write checkpoint to hdfs
    saver.save()
    # write exit_signal variables to notify all other workers to exit
    sess.run(increase_exit_signal_op)
else:
    while True:
        if sess.run(exit_signal_variable) != 0:
            time.sleep(30)
        else:
            break

I've been following the discussion and the fix of @fesun is exactly what I guessed when I read this issue from the beginning. Frankly speaking the current gRPC implementation has some serious issues that makes it more like an ad-hoc solution for small number of tasks instead of a design for production-ready large-scale training.
My hunch is that, since all Recv ops work asynchronously in data-plane, it is not really compatible with the synchronous nature of RPC, which is designed for control-plane with the following assumptions: 1) message size for both request and reply should be small 2) requests should either be timed-out and retried or returned with normal replies fast. Tensor transport in data plane, however, violates both assumptions.
Of course the original TF inside Google with RDMA as out-of-band data plane does not have this issue. RDMA is not universally available; but on the second thought, should we offer TCP socket (instead of gRPC) as an out-of-band alternative solution, instead of transferring those large tensors by waiting for upstream computation and/or copying huge gRPC messages and eventually drain all the resources available for other RPC handler? All I know is that changing 100 to 2000 should not be the final solution to this problem; we need some redesign with the current remote Recv ops.

@fesun @mrry great findings!  In response to @mrry's previous hunch on the SaveV2 issue, we recently pushed a patch to the master branch that OP_REQUIRES_OK the writer.Add(Slice) calls, so hopefully the checkpoint writing path has become more robust (or easier to debug).

@fesun Thanks for digging into the ENQUEUE_REQUEST() issue. This seems like a bug that only occurs at higher pending request counts (e.g. when we have a larger cluster of workers, all contacting the same PS task), but it doesn't seem to trigger deterministically (e.g. when the number of enqueued requests drops to zero).
Perhaps somebody on the gRPC team has a hunch.... @ctiller, you were super-helpful when we were writing the original async code for TensorFlow on gRPC, and I recall you mentioning that there could be a negative impact on performance if the number of enqueued requests for a method ever dropped to zero. Can you think of a reason why a request might hang indefinitely in this regime? (It would be great to find a simple repro that doesn't involve running 800 processes....) Perhaps I made some fundamentally incorrect assumption about how the async code works?

@mrry If changing the three constants to very small numbers, maybe it's possible to reproduce the issue with dozens of workers, I may try it when I get some time.

@mrry After increasing request object counts, there is performance decay along with running time, at first each step only takes 0.3 second, after 4~5 hours, each step takes 1 second. There must be some severe bug or design issues.

@fesun Thanks for looking into this further. Have you been able to reproduce this with some simple code that runs on a single machine? If so, we can try to reproduce it locally and take a look.
Otherwise, it would be great if you could try to attach some profiling tools such as pprof to see if (i) there is some slower path being taken later in the execution, which would show up in a sample-based profile, or (ii) there is a memory leak that would show up in a heap profile of the running process.

@fesun any update? Are you still experiencing this problem or can we close the issue?

@skye My workaround works for me on v1.0.1. But I think the issue is still there, I was told the workaround posted here didn't work for latest v1.2.

@fesun Maybe a small reproducing script would help us to identify the problem. As you mentioned, if changing the three constants to very small numbers in ENQUEUE_REQUEST could reproduce the issue in a smaller scale (2~3 nodes), I am glad to run a customized build locally and see if I could reproduce that. Problem is I don't have a Windows box, so I am not sure if it works.

@byronyi Sadly I can't reproduce it using small cluster.

You mentioned the neural net involved in this case is simple. What is the largest size of tensors a worker need to fetch from PS, and how many tensors are transmitted in a round?
gRPC is implemented in IOCP on Windows and it's beyond my knowledge if it only happens with a large number of concurrent connection and/or requests per second. But I may have some idea if this issue occurs when transmitting large tensors.

@byronyi The tensor size is not large, smaller than 1M, what I want to highlight is the time when tensorflow hangs. It happens only when some workers finished training and waiting for others to finish using while loop by checking one finish_count variable. The phenomenon is that sess.run() didn't return within 20 seconds timeout when some workers finished without timeout exception, at last I will call my custom saving operator(master worker call sess.run([my_saving]) without timeout setting) and all the parameter server wrote models successfully, but sess.run() never returned. Increasing those three magic numbers do solve my problem temporary, but my scenario is this: I have dozens of files to process, I start parameter servers once, restart workers for each file, when I increase magic number to 3000, with 500 workers, tensorflow hangs after about 6-7 restart, increasing to 5000, hangs after about 10 restart. Seems that each worker will occupy one request object and not reusable. I believe this grpc implementation definitely has some wrong and probably platform independent.

Could you elaborate on the way how you restart the workers? Is it repeated call to session.run, or you restart the whole worker process? I remember there are some issues with current design such that a clean restart is not always possible within the same process' lifetime (there're some global resources that can't be released without exit of the process).

Terminate all worker processes and create them again. @byronyi If you want to know more details, please contact me through email, maybe we can have a quick talk, so you can know exactly what I am doing.

There's a decent chance that this hanging bug was fixed by the upgrade to a more recent version of gRPC, which addresses a race condition (leading to potential deadlock) in grpc::CompletionQueue. I'm closing this in the hope that things are now working at HEAD and in the nightly builds; please reopen if you can reproduce with an up-to-date version of TensorFlow.