Assertion failure:  in lowlevel_dma.cc:2775 - Assertion `dist.find(dst_mem) != dist.end()' failed.

When running tester_io on 4 nodes:
setenv GASNET_BACKTRACE 1
${HOME}/local/bin/gasnetrun_ibv -n 4  ./tester_io -n 125000000 -s 125 -r 3 -ll:cpu 4 -ll:dma 3
I get the following assertion failure on the current DMA branch
tester_io: /users/gshipman/local/src/legion-2/runtime//lowlevel_dma.cc:2775: void LegionRuntime::LowLevel::find_shortest_path(LegionRuntime::LowLevel::Memory, LegionRuntime::LowLevel::Memory, std::vector<Realm::Memory, std::allocatorRealm::Memory >&): Assertion dist.find(dst_mem) != dist.end()' failed. *** Caught a fatal signal: SIGABRT(6) on node 1/4 tester_io: /users/gshipman/local/src/legion-2/runtime//activemsg.cc:374: void SrcDataPool::release_srcptr(void*): Assertion it != in_use.end()' failed.
*** Caught a fatal signal: SIGABRT(6) on node 0/4
tester_io: /users/gshipman/local/src/legion-2/runtime//lowlevel_dma.cc:2775: void LegionRuntime::LowLevel::find_shortest_path(LegionRuntime::LowLevel::Memory, LegionRuntime::LowLevel::Memory, std::vector<Realm::Memory, std::allocatorRealm::Memory >&): Assertion `dist.find(dst_mem) != dist.end()' failed.
*** Caught a fatal signal: SIGABRT(6) on node 2/4
[0] /usr/bin/gstack 65423
[0] Thread 14 (Thread 0x2ab91208e700 (LWP 65424)):
[0] #0  0x00002ab90d26f373 in select () from /lib64/libc.so.6
[0] #1  0x00002ab90b30365f in service_thread_start () from /usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[0] #2  0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0
[0] #3  0x00002ab90d2768fd in clone () from /lib64/libc.so.6
[0] Thread 13 (Thread 0x2ab9128c3700 (LWP 65425)):
[0] #0  0x00002ab90d26d0d3 in poll () from /lib64/libc.so.6
[0] #1  0x00002ab90b301ef0 in btl_openib_async_thread () from /usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[0] #2  0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0
[0] #3  0x00002ab90d2768fd in clone () from /lib64/libc.so.6
[0] Thread 12 (Thread 0x2ab913b17700 (LWP 65427)):
[0] #0  0x00002ab90d25d287 in sched_yield () from /lib64/libc.so.6
[0] #1  0x0000000000d3afbe in gasneti_bt_gstack ()
[0] #2  0x0000000000d3c0ff in gasneti_print_backtrace ()
[0] #3  0x0000000000d9fb4e in gasneti_defaultSignalHandler ()
[0] #4
[0] #5  0x00002ab90d1c0625 in raise () from /lib64/libc.so.6
[0] #6  0x00002ab90d1c1e05 in abort () from /lib64/libc.so.6
[0] #7  0x00002ab90d1b974e in __assert_fail_base () from /lib64/libc.so.6
[0] #8  0x00002ab90d1b9810 in __assert_fail () from /lib64/libc.so.6
[0] #9  0x00000000009aef25 in SrcDataPool::release_srcptr(void*) ()
[0] #10 0x00000000009b1813 in SrcDataPool::release_srcptr_handler(void*, int, int) ()
[0] #11 0x0000000000d24cfc in gasnetc_rcv_reap ()
[0] #12 0x0000000000d26a8f in gasnetc_RequestGeneric ()
[0] #13 0x0000000000d159f8 in gasnetc_AMRequestLongAsyncM ()
[0] #14 0x00000000009b657f in ActiveMessageEndpoint::send_long(OutgoingMessage*, void*) ()
[0] #15 0x00000000009b31ba in ActiveMessageEndpoint::push_messages(int, bool) ()
[0] #16 0x00000000009b6e7d in EndpointManager::push_messages(int, bool) ()
[0] #17 0x00000000009b0c2f in do_some_polling() ()
[0] #18 0x00000000009b0d2c in gasnet_poll_thread_loop(void*) ()
[0] #19 0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0
[0] #20 0x00002ab90d2768fd in clone () from /lib64/libc.so.6
[0] Thread 11 (Thread 0x2ab913d18700 (LWP 65428)):
[0] #0  0x00002ab90abfd5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[0] #1  0x00000000009afb74 in IncomingMessageManager::get_messages(int&, bool) ()
[0] #2  0x00000000009afcfd in Realm::HandlerThread::thread_main() ()
[0] #3  0x0000000000a0a1a9 in Realm::PreemptableThread::thread_entry(void*) ()
[0] #4  0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0
[0] #5  0x00002ab90d2768fd in clone () from /lib64/libc.so.6
[0] Thread 10 (Thread 0x2ab933f1c700 (LWP 65429)):
[0] #0  0x00002ab90abfd5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[0] #1  0x000000000098f75c in LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread*, bool) ()
[0] #2  0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop() ()
[0] #3  0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void*) ()
[0] #4  0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0
[0] #5  0x00002ab90d2768fd in clone () from /lib64/libc.so.6
[0] Thread 9 (Thread 0x2ab93411d700 (LWP 65430)):
[0] #0  0x00002ab90abfd5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[0] #1  0x000000000098f75c in LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread*, bool) ()
[0] #2  0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop() ()
[0] #3  0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void*) ()
[0] #4  0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0
[0] #5  0x00002ab90d2768fd in clone () from /lib64/libc.so.6
[0] Thread 8 (Thread 0x2ab93431e700 (LWP 65431)):
[0] #0  0x00002ab90abfd5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[0] #1  0x000000000098f75c in LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread*, bool) ()
[0] #2  0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop() ()
[0] #3  0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void*) ()
[0] #4  0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0
[0] #5  0x00002ab90d2768fd in clone () from /lib64/libc.so.6
[0] Thread 7 (Thread 0x2ab93451f700 (LWP 65432)):
[0] #0  0x00002ab90abfd5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[0] #1  0x000000000098f75c in LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread*, bool) ()
[0] #2  0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop() ()
[0] #3  0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void*) ()
[0] #4  0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0
[0] #5  0x00002ab90d2768fd in clone () from /lib64/libc.so.6
[0] Thread 6 (Thread 0x2ab934720700 (LWP 65433)):
[0] #0  0x00002ab90ac00264 in __lll_lock_wait () from /lib64/libpthread.so.0
[0] #1  0x00002ab90abfb508 in _L_lock_854 () from /lib64/libpthread.so.0
[0] #2  0x00002ab90abfb3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
[0] #3  0x00000000009b2872 in SrcDataPool::Lock::Lock(SrcDataPool&) ()
[0] #4  0x00000000009b0103 in OutgoingMessage::reserve_srcdata() ()
[0] #5  0x00000000009b3389 in ActiveMessageEndpoint::enqueue_message(OutgoingMessage*, bool) ()
[0] #6  0x00000000009b6fcb in EndpointManager::enqueue_message(unsigned short, OutgoingMessage*, bool) ()
[0] #7  0x00000000009b152b in enqueue_message(unsigned short, int, void const*, unsigned long, void const*, unsigned long, int, void*) ()
[0] #8  0x0000000000a0d363 in ActiveMessageMediumNoReply<142, Realm::SpawnTaskMessage::RequestArgs, &(Realm::SpawnTaskMessage::handle_request(Realm::SpawnTaskMessage::RequestArgs, void const*, unsigned long))>::request(unsigned short, Realm::SpawnTaskMessage::RequestArgs&, void const*, unsigned long, int, void*) ()
[0] #9  0x0000000000a0803c in Realm::SpawnTaskMessage::send_request(unsigned short, Realm::Processor, unsigned int, void const*, unsigned long, Realm::ProfilingRequestSet const*, Realm::Event, Realm::Event, int) ()
[0] #10 0x0000000000a08359 in Realm::RemoteProcessor::spawn_task(unsigned int, void const*, unsigned long, Realm::Event, Realm::Event, int) ()
[0] #11 0x0000000000a07051 in Realm::Processor::spawn(unsigned int, void const*, unsigned long, Realm::Event, int) const ()
[0] #12 0x0000000000c74c72 in LegionRuntime::HighLevel::Runtime::issue_runtime_meta_task(void const*, unsigned long, LegionRuntime::HighLevel::HLRTaskID, LegionRuntime::HighLevel::Operation*, Realm::Event, int, Realm::Processor) ()
[0] #13 0x0000000000c5693d in LegionRuntime::HighLevel::MessageManager::send_message(bool) ()
[0] #14 0x0000000000c56840 in LegionRuntime::HighLevel::MessageManager::package_message(LegionRuntime::HighLevel::Serializer&, LegionRuntime::HighLevel::MessageManager::MessageKind, bool) ()
[0] #15 0x0000000000c55b64 in LegionRuntime::HighLevel::MessageManager::send_add_distributed_remote(LegionRuntime::HighLevel::Serializer&, bool) ()
[0] #16 0x0000000000c70966 in LegionRuntime::HighLevel::Runtime::send_add_distributed_remote(unsigned int, LegionRuntime::HighLevel::Serializer&) ()
[0] #17 0x0000000000ce991c in LegionRuntime::HighLevel::DistributedCollectable::send_remote_reference(unsigned int, unsigned int) ()
[0] #18 0x0000000000b41809 in LegionRuntime::HighLevel::RegionTreeForest::send_remote_references(std::map<LegionRuntime::HighLevel::LogicalView*, LegionRuntime::HighLevel::AVXTLBitMask<512u>, std::lessLegionRuntime::HighLevel::LogicalView*, LegionRuntime::HighLevel::AlignedAllocator<std::pair<LegionRuntime::HighLevel::LogicalView* const, LegionRuntime::HighLevel::AVXTLBitMask<512u> > > > const&, std::set<LegionRuntime::HighLevel::PhysicalManager*, std::lessLegionRuntime::HighLevel::PhysicalManager*, std::allocatorLegionRuntime::HighLevel::PhysicalManager* > const&, unsigned int) ()
[0] #19 0x0000000000ae1433 in LegionRuntime::HighLevel::IndexTask::send_remote_state(unsigned int, std::vector<unsigned int, std::allocator > const&, std::vector<unsigned int, std::allocator > const&, std::vector<unsigned int, std::allocator > const&) ()
[0] #20 0x0000000000ae175b in LegionRuntime::HighLevel::IndexTask::handle_slice_request(LegionRuntime::HighLevel::Runtime*, LegionRuntime::HighLevel::Deserializer&, unsigned int) ()
[0] #21 0x0000000000c7235f in LegionRuntime::HighLevel::Runtime::handle_slice_request(LegionRuntime::HighLevel::Deserializer&, unsigned int) ()
[0] #22 0x0000000000c57294 in LegionRuntime::HighLevel::MessageManager::handle_messages(unsigned int, char const*, unsigned long) ()
[0] #23 0x0000000000c569ff in LegionRuntime::HighLevel::MessageManager::process_message(void const*, unsigned long) ()
[0] #24 0x0000000000c72a5d in LegionRuntime::HighLevel::Runtime::process_message_task(void const*, unsigned long) ()
[0] #25 0x0000000000c7e154 in LegionRuntime::HighLevel::Runtime::high_level_runtime_task(void const*, unsigned long, Realm::Processor) ()
[0] #26 0x0000000000a09fe8 in Realm::PreemptableThread::run_task(Realm::Task*, Realm::Processor) ()
[0] #27 0x0000000000a0a420 in Realm::GreenletTask::run(void*) ()
[0] #28 0x0000000000a50100 in greenlet::_run(void*) ()
[0] #29 0x0000000000a4f85c in _greenlet_start(void*) ()
[0] #30 0x0000000000000000 in ?? ()
[0] Thread 5 (Thread 0x2ab934921700 (LWP 65434)):
[0] #0  0x00002ab90abfd5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[0] #1  0x00000000009c6bca in GASNetCondVar::wait() ()
[0] #2  0x0000000000a0b257 in Realm::GreenletProcessor::execute_task() ()
[0] #3  0x0000000000a0a539 in Realm::GreenletThread::thread_main() ()
[0] #4  0x0000000000a0a1a9 in Realm::PreemptableThread::thread_entry(void*) ()
[0] #5  0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0
[0] #6  0x00002ab90d2768fd in clone () from /lib64/libc.so.6
[0] Thread 4 (Thread 0x2ab934b22700 (LWP 65435)):
[0] #0  0x00002ab90abfd5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[0] #1  0x00000000009c6bca in GASNetCondVar::wait() ()
[0] #2  0x0000000000a0b257 in Realm::GreenletProcessor::execute_task() ()
[0] #3  0x0000000000a0a539 in Realm::GreenletThread::thread_main() ()
[0] #4  0x0000000000a0a1a9 in Realm::PreemptableThread::thread_entry(void*) ()
[0] #5  0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0
[0] #6  0x00002ab90d2768fd in clone () from /lib64/libc.so.6
[0] Thread 3 (Thread 0x2ab935000700 (LWP 65436)):
[0] #0  0x00002ab90abfd5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[0] #1  0x00000000009c6bca in GASNetCondVar::wait() ()
[0] #2  0x0000000000a0b257 in Realm::GreenletProcessor::execute_task() ()
[0] #3  0x0000000000a0a539 in Realm::GreenletThread::thread_main() ()
[0] #4  0x0000000000a0a1a9 in Realm::PreemptableThread::thread_entry(void*) ()
[0] #5  0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0
[0] #6  0x00002ab90d2768fd in clone () from /lib64/libc.so.6
[0] Thread 2 (Thread 0x2ab935201700 (LWP 65437)):
[0] #0  0x00002ab90abfd5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[0] #1  0x00000000009c6bca in GASNetCondVar::wait() ()
[0] #2  0x0000000000a0b257 in Realm::GreenletProcessor::execute_task() ()
[0] #3  0x0000000000a0a539 in Realm::GreenletThread::thread_main() ()
[0] #4  0x0000000000a0a1a9 in Realm::PreemptableThread::thread_entry(void*) ()
[0] #5  0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0
[0] #6  0x00002ab90d2768fd in clone () from /lib64/libc.so.6
[0] Thread 1 (Thread 0x2ab911c5d1e0 (LWP 65423)):
[0] #0  0x00002ab90d23aa3d in nanosleep () from /lib64/libc.so.6
[0] #1  0x00002ab90d23a8b0 in sleep () from /lib64/libc.so.6
[0] #2  0x0000000000a412c7 in Realm::RuntimeImpl::run(unsigned int, Realm::Runtime::RunStyle, void const*, unsigned long, bool) ()
[0] #3  0x0000000000a3e00c in Realm::Runtime::run(unsigned int, Realm::Runtime::RunStyle, void const*, unsigned long, bool) ()
[0] #4  0x0000000000c7be7f in LegionRuntime::HighLevel::Runtime::start(int, char**, bool) ()
[0] #5  0x0000000000a623cf in LegionRuntime::HighLevel::HighLevelRuntime::start(int, char**, bool) ()
[0] #6  0x000000000097b33b in main ()
[2] /usr/bin/gstack 98078
[2] Thread 14 (Thread 0x2b35a9d6c700 (LWP 98079)):
[2] #0  0x00002b35a4f4d373 in select () from /lib64/libc.so.6
[2] #1  0x00002b35a2fe165f in service_thread_start () from /usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[2] #2  0x00002b35a28d79d1 in start_thread () from /lib64/libpthread.so.0
[2] #3  0x00002b35a4f548fd in clone () from /lib64/libc.so.6
[2] Thread 13 (Thread 0x2b35aa5a1700 (LWP 98080)):
[2] #0  0x00002b35a4f4b0d3 in poll () from /lib64/libc.so.6
[2] #1  0x00002b35a2fdfef0 in btl_openib_async_thread () from /usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[2] #2  0x00002b35a28d79d1 in start_thread () from /lib64/libpthread.so.0
[2] #3  0x00002b35a4f548fd in clone () from /lib64/libc.so.6
[2] Thread 12 (Thread 0x2b35ab816700 (LWP 98082)):
[2] #0  0x00002b35a4f3b287 in sched_yield () from /lib64/libc.so.6
[2] #1  0x0000000000d26931 in gasnetc_RequestGeneric ()
[2] #2  0x0000000000d15d72 in gasnetc_AMRequestShortM ()
[2] #3  0x00000000009b3d7f in ActiveMessageEndpoint::send_short(OutgoingMessage*) ()
[2] #4  0x00000000009b2f1e in ActiveMessageEndpoint::push_messages(int, bool) ()
[2] #5  0x00000000009b6e7d in EndpointManager::push_messages(int, bool) ()
[2] #6  0x00000000009b0c2f in do_some_polling() ()
[2] #7  0x00000000009b0d2c in gasnet_poll_thread_loop(void*) ()
[2] #8  0x00002b35a28d79d1 in start_thread () from /lib64/libpthread.so.0
[2] #9  0x00002b35a4f548fd in clone () from /lib64/libc.so.6
[2] Thread 11 (Thread 0x2b35aba17700 (LWP 98083)):
[2] #0  0x00002b35a28db5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[2] #1  0x00000000009afb74 in IncomingMessageManager::get_messages(int&, bool) ()
[2] #2  0x00000000009afcfd in Realm::HandlerThread::thread_main() ()
[2] #3  0x0000000000a0a1a9 in Realm::PreemptableThread::thread_entry(void*) ()
[2] #4  0x00002b35a28d79d1 in start_thread () from /lib64/libpthread.so.0
[2] #5  0x00002b35a4f548fd in clone () from /lib64/libc.so.6
[2] Thread 10 (Thread 0x2b35cbe00700 (LWP 98084)):
[2] #0  0x00002b35a28db5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[2] #1  0x000000000098f75c in LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread*, bool) ()
[2] #2  0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop() ()
[2] #3  0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void*) ()
[2] #4  0x00002b35a28d79d1 in start_thread () from /lib64/libpthread.so.0
[2] #5  0x00002b35a4f548fd in clone () from /lib64/libc.so.6
[2] Thread 9 (Thread 0x2b35cc001700 (LWP 98085)):
[2] #0  0x00002b35a28db5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[2] #1  0x000000000098f75c in LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread*, bool) ()
[2] #2  0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop() ()
[2] #3  0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void*) ()
[2] #4  0x00002b35a28d79d1 in start_thread () from /lib64/libpthread.so.0
[2] #5  0x00002b35a4f548fd in clone () from /lib64/libc.so.6
[2] Thread 8 (Thread 0x2b35cc202700 (LWP 98086)):
[2] #0  0x00002b35a28db5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[2] #1  0x000000000098f75c in LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread*, bool) ()
[2] #2  0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop() ()
[2] #3  0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void*) ()
[2] #4  0x00002b35a28d79d1 in start_thread () from /lib64/libpthread.so.0
[2] #5  0x00002b35a4f548fd in clone () from /lib64/libc.so.6
[2] Thread 7 (Thread 0x2b35cc403700 (LWP 98087)):
[2] #0  0x00002b35a28db5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[2] #1  0x000000000098f75c in LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread*, bool) ()
[2] #2  0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop() ()
[2] #3  0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void*) ()
[2] #4  0x00002b35a28d79d1 in start_thread () from /lib64/libpthread.so.0
[2] #5  0x00002b35a4f548fd in clone () from /lib64/libc.so.6
[2] Thread 6 (Thread 0x2b35cc604700 (LWP 98088)):
[2] #0  0x00002b35a4f3b287 in sched_yield () from /lib64/libc.so.6
[2] #1  0x0000000000d3afbe in gasneti_bt_gstack ()
[2] #2  0x0000000000d3c0ff in gasneti_print_backtrace ()
[2] #3  0x0000000000d9fb4e in gasneti_defaultSignalHandler ()
[2] #4
[2] #5  0x00002b35a4e9e625 in raise () from /lib64/libc.so.6
[2] #6  0x00002b35a4e9fe05 in abort () from /lib64/libc.so.6
[2] #7  0x00002b35a4e9774e in __assert_fail_base () from /lib64/libc.so.6
[2] #8  0x00002b35a4e97810 in __assert_fail () from /lib64/libc.so.6
[2] #9  0x00000000009c0383 in LegionRuntime::LowLevel::find_shortest_path(Realm::Memory, Realm::Memory, std::vector<Realm::Memory, std::allocatorRealm::Memory >&) ()
[2] #10 0x00000000009cfc39 in void LegionRuntime::LowLevel::CopyRequest::perform_new_dma<3u>(Realm::Memory, Realm::Memory) ()
[2] #11 0x00000000009c0619 in LegionRuntime::LowLevel::CopyRequest::perform_dma() ()
[2] #12 0x00000000009bec95 in LegionRuntime::LowLevel::CopyRequest::check_readiness(bool, LegionRuntime::LowLevel::DmaRequestQueue*) ()
[2] #13 0x00000000009c5251 in Realm::Domain::copy(std::vector<Realm::Domain::CopySrcDstField, std::allocatorRealm::Domain::CopySrcDstField > const&, std::vector<Realm::Domain::CopySrcDstField, std::allocatorRealm::Domain::CopySrcDstField > const&, Realm::ProfilingRequestSet const&, Realm::Event, int, bool) const ()
[2] #14 0x00000000009c4acd in Realm::Domain::copy(std::vector<Realm::Domain::CopySrcDstField, std::allocatorRealm::Domain::CopySrcDstField > const&, std::vector<Realm::Domain::CopySrcDstField, std::allocatorRealm::Domain::CopySrcDstField > const&, Realm::Event, int, bool) const ()
[2] #15 0x0000000000b46163 in LegionRuntime::HighLevel::RegionTreeForest::issue_copy(Realm::Domain const&, LegionRuntime::HighLevel::Operation*, std::vector<Realm::Domain::CopySrcDstField, std::allocatorRealm::Domain::CopySrcDstField > const&, std::vector<Realm::Domain::CopySrcDstField, std::allocatorRealm::Domain::CopySrcDstField > const&, Realm::Event) ()
[2] #16 0x0000000000b3fdfc in LegionRuntime::HighLevel::RegionTreeForest::copy_across(LegionRuntime::HighLevel::Operation*, LegionRuntime::HighLevel::RegionTreeContext, LegionRuntime::HighLevel::RegionTreeContext, LegionRuntime::HighLevel::RegionRequirement const&, LegionRuntime::HighLevel::RegionRequirement const&, LegionRuntime::HighLevel::InstanceRef const&, LegionRuntime::HighLevel::InstanceRef const&, Realm::Event) ()
[2] #17 0x0000000000a84b4f in LegionRuntime::HighLevel::CopyOp::trigger_execution() ()
[2] #18 0x0000000000c7e448 in LegionRuntime::HighLevel::Runtime::high_level_runtime_task(void const*, unsigned long, Realm::Processor) ()
[2] #19 0x0000000000a09fe8 in Realm::PreemptableThread::run_task(Realm::Task*, Realm::Processor) ()
[2] #20 0x0000000000a0a420 in Realm::GreenletTask::run(void*) ()
[2] #21 0x0000000000a50100 in greenlet::_run(void*) ()
[2] #22 0x0000000000a4f85c in _greenlet_start(void*) ()
[2] #23 0x0000000000000000 in ?? ()
[2] Thread 5 (Thread 0x2b35cc805700 (LWP 98089)):
[2] #0  0x00002b35a28db5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[2] #1  0x00000000009c6bca in GASNetCondVar::wait() ()
[2] #2  0x0000000000a0b257 in Realm::GreenletProcessor::execute_task() ()
[2] #3  0x0000000000a0a539 in Realm::GreenletThread::thread_main() ()
[2] #4  0x0000000000a0a1a9 in Realm::PreemptableThread::thread_entry(void*) ()
[2] #5  0x00002b35a28d79d1 in start_thread () from /lib64/libpthread.so.0
[2] #6  0x00002b35a4f548fd in clone () from /lib64/libc.so.6
[2] Thread 4 (Thread 0x2b35cca06700 (LWP 98090)):
[2] #0  0x00002b35a28db5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[2] #1  0x00000000009c6bca in GASNetCondVar::wait() ()
[2] #2  0x0000000000a0b257 in Realm::GreenletProcessor::execute_task() ()
[2] #3  0x0000000000a0a539 in Realm::GreenletThread::thread_main() ()
[2] #4  0x0000000000a0a1a9 in Realm::PreemptableThread::thread_entry(void*) ()
[2] #5  0x00002b35a28d79d1 in start_thread () from /lib64/libpthread.so.0
[2] #6  0x00002b35a4f548fd in clone () from /lib64/libc.so.6
[2] Thread 3 (Thread 0x2b35ccc07700 (LWP 98091)):
[2] #0  0x00002b35a28db5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[2] #1  0x00000000009c6bca in GASNetCondVar::wait() ()
[2] #2  0x0000000000a0b257 in Realm::GreenletProcessor::execute_task() ()
[2] #3  0x0000000000a0a539 in Realm::GreenletThread::thread_main() ()
[2] #4  0x0000000000a0a1a9 in Realm::PreemptableThread::thread_entry(void*) ()
[2] #5  0x00002b35a28d79d1 in start_thread () from /lib64/libpthread.so.0
[2] #6  0x00002b35a4f548fd in clone () from /lib64/libc.so.6
[2] Thread 2 (Thread 0x2b35cce08700 (LWP 98092)):
[2] #0  0x00002b35a28db5bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[2] #1  0x00000000009c6bca in GASNetCondVar::wait() ()
[2] #2  0x0000000000a0b257 in Realm::GreenletProcessor::execute_task() ()
[2] #3  0x0000000000a0a539 in Realm::GreenletThread::thread_main() ()
[2] #4  0x0000000000a0a1a9 in Realm::PreemptableThread::thread_entry(void*) ()
[2] #5  0x00002b35a28d79d1 in start_thread () from /lib64/libpthread.so.0
[2] #6  0x00002b35a4f548fd in clone () from /lib64/libc.so.6
[2] Thread 1 (Thread 0x2b35a993b1e0 (LWP 98078)):
[2] #0  0x00002b35a4f18a3d in nanosleep () from /lib64/libc.so.6
[2] #1  0x00002b35a4f188b0 in sleep () from /lib64/libc.so.6
[2] #2  0x0000000000a412c7 in Realm::RuntimeImpl::run(unsigned int, Realm::Runtime::RunStyle, void const*, unsigned long, bool) ()
[2] #3  0x0000000000a3e00c in Realm::Runtime::run(unsigned int, Realm::Runtime::RunStyle, void const*, unsigned long, bool) ()
[2] #4  0x0000000000c7be7f in LegionRuntime::HighLevel::Runtime::start(int, char**, bool) ()
[2] #5  0x0000000000a623cf in LegionRuntime::HighLevel::HighLevelRuntime::start(int, char**, bool) ()
[2] #6  0x000000000097b33b in main ()
[1] /usr/bin/gstack 71570
[1] Thread 14 (Thread 0x2ac20e682700 (LWP 71571)):
[1] #0  0x00002ac209863373 in select () from /lib64/libc.so.6
[1] #1  0x00002ac2078f765f in service_thread_start () from /usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[1] #2  0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0
[1] #3  0x00002ac20986a8fd in clone () from /lib64/libc.so.6
[1] Thread 13 (Thread 0x2ac20eeb7700 (LWP 71572)):
[1] #0  0x00002ac2098610d3 in poll () from /lib64/libc.so.6
[1] #1  0x00002ac2078f5ef0 in btl_openib_async_thread () from /usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[1] #2  0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0
[1] #3  0x00002ac20986a8fd in clone () from /lib64/libc.so.6
[1] Thread 12 (Thread 0x2ac21010b700 (LWP 71574)):
[1] #0  0x00002ac2071f2380 in pthread_spin_lock () from /lib64/libpthread.so.0
[1] #1  0x00002ac20e6a517f in ?? () from /usr/lib64/libipathverbs-rdmav2.so
[1] #2  0x0000000000d24a41 in gasnetc_rcv_reap ()
[1] #3  0x0000000000d25209 in gasnetc_AMPoll ()
[1] #4  0x00000000009b0c34 in do_some_polling() ()
[1] #5  0x00000000009b0d2c in gasnet_poll_thread_loop(void*) ()
[1] #6  0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0
[1] #7  0x00002ac20986a8fd in clone () from /lib64/libc.so.6
[1] Thread 11 (Thread 0x2ac21030c700 (LWP 71575)):
[1] #0  0x00002ac2071f15bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[1] #1  0x00000000009afb74 in IncomingMessageManager::get_messages(int&, bool) ()
[1] #2  0x00000000009afcfd in Realm::HandlerThread::thread_main() ()
[1] #3  0x0000000000a0a1a9 in Realm::PreemptableThread::thread_entry(void*) ()
[1] #4  0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0
[1] #5  0x00002ac20986a8fd in clone () from /lib64/libc.so.6
[1] Thread 10 (Thread 0x2ac230510700 (LWP 71576)):
[1] #0  0x00002ac2071f15bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[1] #1  0x000000000098f75c in LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread*, bool) ()
[1] #2  0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop() ()
[1] #3  0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void*) ()
[1] #4  0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0
[1] #5  0x00002ac20986a8fd in clone () from /lib64/libc.so.6
[1] Thread 9 (Thread 0x2ac230711700 (LWP 71577)):
[1] #0  0x00002ac2071f15bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[1] #1  0x000000000098f75c in LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread*, bool) ()
[1] #2  0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop() ()
[1] #3  0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void*) ()
[1] #4  0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0
[1] #5  0x00002ac20986a8fd in clone () from /lib64/libc.so.6
[1] Thread 8 (Thread 0x2ac230912700 (LWP 71578)):
[1] #0  0x00002ac2071f15bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[1] #1  0x000000000098f75c in LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread*, bool) ()
[1] #2  0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop() ()
[1] #3  0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void*) ()
[1] #4  0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0
[1] #5  0x00002ac20986a8fd in clone () from /lib64/libc.so.6
[1] Thread 7 (Thread 0x2ac230b13700 (LWP 71579)):
[1] #0  0x00002ac2071f15bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[1] #1  0x000000000098f75c in LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread*, bool) ()
[1] #2  0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop() ()
[1] #3  0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void*) ()
[1] #4  0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0
[1] #5  0x00002ac20986a8fd in clone () from /lib64/libc.so.6
[1] Thread 6 (Thread 0x2ac230d14700 (LWP 71580)):
[1] #0  0x00002ac209851287 in sched_yield () from /lib64/libc.so.6
[1] #1  0x0000000000d3afbe in gasneti_bt_gstack ()
[1] #2  0x0000000000d3c0ff in gasneti_print_backtrace ()
[1] #3  0x0000000000d9fb4e in gasneti_defaultSignalHandler ()
[1] #4
[1] #5  0x00002ac2097b4625 in raise () from /lib64/libc.so.6
[1] #6  0x00002ac2097b5e05 in abort () from /lib64/libc.so.6
[1] #7  0x00002ac2097ad74e in __assert_fail_base () from /lib64/libc.so.6
[1] #8  0x00002ac2097ad810 in __assert_fail () from /lib64/libc.so.6
[1] #9  0x00000000009c0383 in LegionRuntime::LowLevel::find_shortest_path(Realm::Memory, Realm::Memory, std::vector<Realm::Memory, std::allocatorRealm::Memory >&) ()
[1] #10 0x00000000009cfc39 in void LegionRuntime::LowLevel::CopyRequest::perform_new_dma<3u>(Realm::Memory, Realm::Memory) ()
[1] #11 0x00000000009c0619 in LegionRuntime::LowLevel::CopyRequest::perform_dma() ()
[1] #12 0x00000000009bec95 in LegionRuntime::LowLevel::CopyRequest::check_readiness(bool, LegionRuntime::LowLevel::DmaRequestQueue*) ()
[1] #13 0x00000000009c5251 in Realm::Domain::copy(std::vector<Realm::Domain::CopySrcDstField, std::allocatorRealm::Domain::CopySrcDstField > const&, std::vector<Realm::Domain::CopySrcDstField, std::allocatorRealm::Domain::CopySrcDstField > const&, Realm::ProfilingRequestSet const&, Realm::Event, int, bool) const ()
[1] #14 0x00000000009c4acd in Realm::Domain::copy(std::vector<Realm::Domain::CopySrcDstField, std::allocatorRealm::Domain::CopySrcDstField > const&, std::vector<Realm::Domain::CopySrcDstField, std::allocatorRealm::Domain::CopySrcDstField > const&, Realm::Event, int, bool) const ()
[1] #15 0x0000000000b46163 in LegionRuntime::HighLevel::RegionTreeForest::issue_copy(Realm::Domain const&, LegionRuntime::HighLevel::Operation*, std::vector<Realm::Domain::CopySrcDstField, std::allocatorRealm::Domain::CopySrcDstField > const&, std::vector<Realm::Domain::CopySrcDstField, std::allocatorRealm::Domain::CopySrcDstField > const&, Realm::Event) ()
[1] #16 0x0000000000b3fdfc in LegionRuntime::HighLevel::RegionTreeForest::copy_across(LegionRuntime::HighLevel::Operation*, LegionRuntime::HighLevel::RegionTreeContext, LegionRuntime::HighLevel::RegionTreeContext, LegionRuntime::HighLevel::RegionRequirement const&, LegionRuntime::HighLevel::RegionRequirement const&, LegionRuntime::HighLevel::InstanceRef const&, LegionRuntime::HighLevel::InstanceRef const&, Realm::Event) ()
[1] #17 0x0000000000a84b4f in LegionRuntime::HighLevel::CopyOp::trigger_execution() ()
[1] #18 0x0000000000c7e448 in LegionRuntime::HighLevel::Runtime::high_level_runtime_task(void const*, unsigned long, Realm::Processor) ()
[1] #19 0x0000000000a09fe8 in Realm::PreemptableThread::run_task(Realm::Task*, Realm::Processor) ()
[1] #20 0x0000000000a0a420 in Realm::GreenletTask::run(void*) ()
[1] #21 0x0000000000a50100 in greenlet::_run(void*) ()
[1] #22 0x0000000000a4f85c in _greenlet_start(void*) ()
[1] #23 0x0000000000000000 in ?? ()
[1] Thread 5 (Thread 0x2ac230f15700 (LWP 71581)):
[1] #0  0x00002ac2071f15bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[1] #1  0x00000000009c6bca in GASNetCondVar::wait() ()
[1] #2  0x0000000000a0b257 in Realm::GreenletProcessor::execute_task() ()
[1] #3  0x0000000000a0a539 in Realm::GreenletThread::thread_main() ()
[1] #4  0x0000000000a0a1a9 in Realm::PreemptableThread::thread_entry(void*) ()
[1] #5  0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0
[1] #6  0x00002ac20986a8fd in clone () from /lib64/libc.so.6
[1] Thread 4 (Thread 0x2ac231300700 (LWP 71582)):
[1] #0  0x00002ac2071f15bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[1] #1  0x00000000009c6bca in GASNetCondVar::wait() ()
[1] #2  0x0000000000a0b257 in Realm::GreenletProcessor::execute_task() ()
[1] #3  0x0000000000a0a539 in Realm::GreenletThread::thread_main() ()
[1] #4  0x0000000000a0a1a9 in Realm::PreemptableThread::thread_entry(void*) ()
[1] #5  0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0
[1] #6  0x00002ac20986a8fd in clone () from /lib64/libc.so.6
[1] Thread 3 (Thread 0x2ac231501700 (LWP 71583)):
[1] #0  0x00002ac2071f15bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[1] #1  0x00000000009c6bca in GASNetCondVar::wait() ()
[1] #2  0x0000000000a0b257 in Realm::GreenletProcessor::execute_task() ()
[1] #3  0x0000000000a0a539 in Realm::GreenletThread::thread_main() ()
[1] #4  0x0000000000a0a1a9 in Realm::PreemptableThread::thread_entry(void*) ()
[1] #5  0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0
[1] #6  0x00002ac20986a8fd in clone () from /lib64/libc.so.6
[1] Thread 2 (Thread 0x2ac231702700 (LWP 71584)):
[1] #0  0x00002ac2071f15bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[1] #1  0x00000000009c6bca in GASNetCondVar::wait() ()
[1] #2  0x0000000000a0b257 in Realm::GreenletProcessor::execute_task() ()
[1] #3  0x0000000000a0a539 in Realm::GreenletThread::thread_main() ()
[1] #4  0x0000000000a0a1a9 in Realm::PreemptableThread::thread_entry(void*) ()
[1] #5  0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0
[1] #6  0x00002ac20986a8fd in clone () from /lib64/libc.so.6
[1] Thread 1 (Thread 0x2ac20e2511e0 (LWP 71570)):
[1] #0  0x00002ac20982ea3d in nanosleep () from /lib64/libc.so.6
[1] #1  0x00002ac20982e8b0 in sleep () from /lib64/libc.so.6
[1] #2  0x0000000000a412c7 in Realm::RuntimeImpl::run(unsigned int, Realm::Runtime::RunStyle, void const*, unsigned long, bool) ()
[1] #3  0x0000000000a3e00c in Realm::Runtime::run(unsigned int, Realm::Runtime::RunStyle, void const*, unsigned long, bool) ()
[1] #4  0x0000000000c7be7f in LegionRuntime::HighLevel::Runtime::start(int, char**, bool) ()
[1] #5  0x0000000000a623cf in LegionRuntime::HighLevel::HighLevelRuntime::start(int, char**, bool) ()
[1] #6  0x000000000097b33b in main ()
mpirun noticed that process rank 1 with PID 71570 on node ml137 exited on signal 6 (Aborted).
*** Caught a signal: SIGTERM(15) on node 3/4

Galen,
Could you please add the following flags and rerun the test.
-ll:rsize 1024 -ll:ahandlers 3
The first is because you need to preallocate some registered men in order
to perform inter node data xfer. The second is because there we currently
create remote XferDes in active message handler thread, therefore add more
active message threads could avoid some dependency issues.
On Tuesday, September 1, 2015, gshipman<EMAIL_ADDRESS>wrote:

When running tester_io on 4 nodes:
setenv GASNET_BACKTRACE 1
${HOME}/local/bin/gasnetrun_ibv -n 4 ./tester_io -n 125000000 -s 125 -r 3
-ll:cpu 4 -ll:dma 3
I get the following assertion failure on the current DMA branch
tester_io:
/users/gshipman/local/src/legion-2/runtime//lowlevel_dma.cc:2775: void
LegionRuntime::LowLevel::find_shortest_path(LegionRuntime::LowLevel::Memory,
LegionRuntime::LowLevel::Memory, std::vector >&): Assertion dist.find(dst_mem)
!= dist.end()' failed.
*** Caught a fatal signal: SIGABRT(6) on node 1/4
tester_io: /users/gshipman/local/src/legion-2/runtime//activemsg.cc:374:
void SrcDataPool::release_srcptr(void*): Assertionit != in_use.end()'
failed.
*** Caught a fatal signal: SIGABRT(6) on node 0/4
tester_io:
/users/gshipman/local/src/legion-2/runtime//lowlevel_dma.cc:2775: void
LegionRuntime::LowLevel::find_shortest_path(LegionRuntime::LowLevel::Memory,
LegionRuntime::LowLevel::Memory, std::vector >&): Assertion
`dist.find(dst_mem) != dist.end()' failed.
*** Caught a fatal signal: SIGABRT(6) on node 2/4
[0] /usr/bin/gstack 65423
[0] Thread 14 (Thread 0x2ab91208e700 (LWP 65424)):
[0] #0 0x00002ab90d26f373 in select () from /lib64/libc.so.6
[0] #1 https://github.com/StanfordLegion/legion/issues/1
0x00002ab90b30365f in service_thread_start () from
/usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[0] #2 https://github.com/StanfordLegion/legion/pull/2
0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0
[0] #3 https://github.com/StanfordLegion/legion/pull/3
0x00002ab90d2768fd in clone () from /lib64/libc.so.6
[0] Thread 13 (Thread 0x2ab9128c3700 (LWP 65425)):
[0] #0 0x00002ab90d26d0d3 in poll () from /lib64/libc.so.6
[0] #1 https://github.com/StanfordLegion/legion/issues/1
0x00002ab90b301ef0 in btl_openib_async_thread () from
/usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[0] #2 https://github.com/StanfordLegion/legion/pull/2
0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0
[0] #3 https://github.com/StanfordLegion/legion/pull/3
0x00002ab90d2768fd in clone () from /lib64/libc.so.6
[0] Thread 12 (Thread 0x2ab913b17700 (LWP 65427)):
[0] #0 0x00002ab90d25d287 in sched_yield () from /lib64/libc.so.6
[0] #1 https://github.com/StanfordLegion/legion/issues/1
0x0000000000d3afbe in gasneti_bt_gstack ()
[0] #2 https://github.com/StanfordLegion/legion/pull/2
0x0000000000d3c0ff in gasneti_print_backtrace ()
[0] #3 https://github.com/StanfordLegion/legion/pull/3
0x0000000000d9fb4e in gasneti_defaultSignalHandler ()
[0] #4 https://github.com/StanfordLegion/legion/issues/4
[0] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002ab90d1c0625 in raise () from /lib64/libc.so.6
[0] #6 https://github.com/StanfordLegion/legion/issues/6
0x00002ab90d1c1e05 in abort () from /lib64/libc.so.6
[0] #7 https://github.com/StanfordLegion/legion/pull/7
0x00002ab90d1b974e in __assert_fail_base () from /lib64/libc.so.6
[0] #8 https://github.com/StanfordLegion/legion/pull/8
0x00002ab90d1b9810 in __assert_fail () from /lib64/libc.so.6
[0] #9 https://github.com/StanfordLegion/legion/issues/9
0x00000000009aef25 in SrcDataPool::release_srcptr(void
) () [0] #10 https://github.com/StanfordLegion/legion/pull/10
0x00000000009b1813 in SrcDataPool::release_srcptr_handler(void, int,
int) ()
[0] #11 https://github.com/StanfordLegion/legion/pull/11
0x0000000000d24cfc in gasnetc_rcv_reap ()
[0] #12 https://github.com/StanfordLegion/legion/pull/12
0x0000000000d26a8f in gasnetc_RequestGeneric ()
[0] #13 https://github.com/StanfordLegion/legion/pull/13
0x0000000000d159f8 in gasnetc_AMRequestLongAsyncM ()
[0] #14 https://github.com/StanfordLegion/legion/pull/14
0x00000000009b657f in ActiveMessageEndpoint::send_long(OutgoingMessage*,
void*) ()
[0] #15 https://github.com/StanfordLegion/legion/pull/15
0x00000000009b31ba in ActiveMessageEndpoint::push_messages(int, bool) ()
[0] #16 https://github.com/StanfordLegion/legion/issues/16
0x00000000009b6e7d in EndpointManager::push_messages(int, bool) ()
[0] #17 https://github.com/StanfordLegion/legion/issues/17
0x00000000009b0c2f in do_some_polling() ()
[0] #18 https://github.com/StanfordLegion/legion/issues/18
0x00000000009b0d2c in gasnet_poll_thread_loop(void
) () [0] #19 https://github.com/StanfordLegion/legion/issues/19
0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0 [0] #20
https://github.com/StanfordLegion/legion/pull/20 0x00002ab90d2768fd in
clone () from /lib64/libc.so.6 [0] Thread 11 (Thread 0x2ab913d18700 (LWP
65428)): [0] #0 0x00002ab90abfd5bc in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0 [0] #1
https://github.com/StanfordLegion/legion/issues/1 0x00000000009afb74 in
IncomingMessageManager::get_messages(int&, bool) () [0] #2
https://github.com/StanfordLegion/legion/pull/2 0x00000000009afcfd in
Realm::HandlerThread::thread_main() () [0] #3
https://github.com/StanfordLegion/legion/pull/3 0x0000000000a0a1a9 in
Realm::PreemptableThread::thread_entry(void) ()
[0] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0
[0] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002ab90d2768fd in clone () from /lib64/libc.so.6
[0] Thread 10 (Thread 0x2ab933f1c700 (LWP 65429)):
[0] #0 0x00002ab90abfd5bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[0] #1 https://github.com/StanfordLegion/legion/issues/1
0x000000000098f75c in
LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread
, bool) () [0] #2 https://github.com/StanfordLegion/legion/pull/2
0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop()
() [0] #3 https://github.com/StanfordLegion/legion/pull/3
0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void) ()
[0] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0
[0] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002ab90d2768fd in clone () from /lib64/libc.so.6
[0] Thread 9 (Thread 0x2ab93411d700 (LWP 65430)):
[0] #0 0x00002ab90abfd5bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[0] #1 https://github.com/StanfordLegion/legion/issues/1
0x000000000098f75c in
LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread
, bool) () [0] #2 https://github.com/StanfordLegion/legion/pull/2
0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop()
() [0] #3 https://github.com/StanfordLegion/legion/pull/3
0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void) ()
[0] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0
[0] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002ab90d2768fd in clone () from /lib64/libc.so.6
[0] Thread 8 (Thread 0x2ab93431e700 (LWP 65431)):
[0] #0 0x00002ab90abfd5bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[0] #1 https://github.com/StanfordLegion/legion/issues/1
0x000000000098f75c in
LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread
, bool) () [0] #2 https://github.com/StanfordLegion/legion/pull/2
0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop()
() [0] #3 https://github.com/StanfordLegion/legion/pull/3
0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void) ()
[0] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0
[0] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002ab90d2768fd in clone () from /lib64/libc.so.6
[0] Thread 7 (Thread 0x2ab93451f700 (LWP 65432)):
[0] #0 0x00002ab90abfd5bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[0] #1 https://github.com/StanfordLegion/legion/issues/1
0x000000000098f75c in
LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread
, bool) () [0] #2 https://github.com/StanfordLegion/legion/pull/2
0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop()
() [0] #3 https://github.com/StanfordLegion/legion/pull/3
0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void) ()
[0] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0
[0] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002ab90d2768fd in clone () from /lib64/libc.so.6
[0] Thread 6 (Thread 0x2ab934720700 (LWP 65433)):
[0] #0 0x00002ab90ac00264 in __lll_lock_wait () from /lib64/libpthread.so.0
[0] #1 https://github.com/StanfordLegion/legion/issues/1
0x00002ab90abfb508 in _L_lock_854 () from /lib64/libpthread.so.0
[0] #2 https://github.com/StanfordLegion/legion/pull/2
0x00002ab90abfb3d7 in pthread_mutex_lock () from /lib64/libpthread.so.0
[0] #3 https://github.com/StanfordLegion/legion/pull/3
0x00000000009b2872 in SrcDataPool::Lock::Lock(SrcDataPool&) ()
[0] #4 https://github.com/StanfordLegion/legion/issues/4
0x00000000009b0103 in OutgoingMessage::reserve_srcdata() ()
[0] #5 https://github.com/StanfordLegion/legion/pull/5
0x00000000009b3389 in ActiveMessageEndpoint::enqueue_message(OutgoingMessage
, bool) () [0] #6 https://github.com/StanfordLegion/legion/issues/6
0x00000000009b6fcb in EndpointManager::enqueue_message(unsigned short,
OutgoingMessage, bool) ()
[0] #7 https://github.com/StanfordLegion/legion/pull/7
0x00000000009b152b in enqueue_message(unsigned short, int, void const*,
unsigned long, void const*, unsigned long, int, void
) () [0] #8 https://github.com/StanfordLegion/legion/pull/8
0x0000000000a0d363 in ActiveMessageMediumNoReply<142,
Realm::SpawnTaskMessage::RequestArgs,
&(Realm::SpawnTaskMessage::handle_request(Realm::SpawnTaskMessage::RequestArgs,
void const, unsigned long))>::request(unsigned short,
Realm::SpawnTaskMessage::RequestArgs&, void const*, unsigned long, int,
void*) ()
[0] #9 https://github.com/StanfordLegion/legion/issues/9
0x0000000000a0803c in Realm::SpawnTaskMessage::send_request(unsigned short,
Realm::Processor, unsigned int, void const*, unsigned long,
Realm::ProfilingRequestSet const*, Realm::Event, Realm::Event, int) ()
[0] #10 https://github.com/StanfordLegion/legion/pull/10
0x0000000000a08359 in Realm::RemoteProcessor::spawn_task(unsigned int, void
const
, unsigned long, Realm::Event, Realm::Event, int) () [0] #11
https://github.com/StanfordLegion/legion/pull/11 0x0000000000a07051 in
Realm::Processor::spawn(unsigned int, void const, unsigned long,
Realm::Event, int) const ()
[0] #12 https://github.com/StanfordLegion/legion/pull/12
0x0000000000c74c72 in
LegionRuntime::HighLevel::Runtime::issue_runtime_meta_task(void const*,
unsigned long, LegionRuntime::HighLevel::HLRTaskID,
LegionRuntime::HighLevel::Operation*, Realm::Event, int,
Realm::Processor) ()
[0] #13 https://github.com/StanfordLegion/legion/pull/13
0x0000000000c5693d in
LegionRuntime::HighLevel::MessageManager::send_message(bool) ()
[0] #14 https://github.com/StanfordLegion/legion/pull/14
0x0000000000c56840 in
LegionRuntime::HighLevel::MessageManager::package_message(LegionRuntime::HighLevel::Serializer&,
LegionRuntime::HighLevel::MessageManager::MessageKind, bool) ()
[0] #15 https://github.com/StanfordLegion/legion/pull/15
0x0000000000c55b64 in
LegionRuntime::HighLevel::MessageManager::send_add_distributed_remote(LegionRuntime::HighLevel::Serializer&,
bool) ()
[0] #16 https://github.com/StanfordLegion/legion/issues/16
0x0000000000c70966 in
LegionRuntime::HighLevel::Runtime::send_add_distributed_remote(unsigned
int, LegionRuntime::HighLevel::Serializer&) ()
[0] #17 https://github.com/StanfordLegion/legion/issues/17
0x0000000000ce991c in
LegionRuntime::HighLevel::DistributedCollectable::send_remote_reference(unsigned
int, unsigned int) ()
[0] #18 https://github.com/StanfordLegion/legion/issues/18
0x0000000000b41809 in
LegionRuntime::HighLevel::RegionTreeForest::send_remote_references(std::map,
std::lessLegionRuntime::HighLevel::LogicalView*,
LegionRuntime::HighLevel::AlignedAllocator > > > const&, std::set,
std::allocatorLegionRuntime::HighLevel::PhysicalManager* > const&, unsigned
int) ()
[0] #19 https://github.com/StanfordLegion/legion/issues/19
0x0000000000ae1433 in
LegionRuntime::HighLevel::IndexTask::send_remote_state(unsigned int,
std::vector > const&, std::vector > const&, std::vector > const&) ()
[0] #20 https://github.com/StanfordLegion/legion/pull/20
0x0000000000ae175b in
LegionRuntime::HighLevel::IndexTask::handle_slice_request(LegionRuntime::HighLevel::Runtime
, LegionRuntime::HighLevel::Deserializer&, unsigned int) () [0] #21
https://github.com/StanfordLegion/legion/pull/21 0x0000000000c7235f in
LegionRuntime::HighLevel::Runtime::handle_slice_request(LegionRuntime::HighLevel::Deserializer&,
unsigned int) () [0] #22 https://github.com/StanfordLegion/legion/pull/22
0x0000000000c57294 in
LegionRuntime::HighLevel::MessageManager::handle_messages(unsigned int,
char const, unsigned long) ()
[0] #23 https://github.com/StanfordLegion/legion/pull/23
0x0000000000c569ff in
LegionRuntime::HighLevel::MessageManager::process_message(void const
, unsigned long) () [0] #24
https://github.com/StanfordLegion/legion/issues/24 0x0000000000c72a5d in
LegionRuntime::HighLevel::Runtime::process_message_task(void const,
unsigned long) ()
[0] #25 https://github.com/StanfordLegion/legion/pull/25
0x0000000000c7e154 in
LegionRuntime::HighLevel::Runtime::high_level_runtime_task(void const
, unsigned long, Realm::Processor) () [0] #26
https://github.com/StanfordLegion/legion/pull/26 0x0000000000a09fe8 in
Realm::PreemptableThread::run_task(Realm::Task, Realm::Processor) ()
[0] #27 https://github.com/StanfordLegion/legion/issues/27
0x0000000000a0a420 in Realm::GreenletTask::run(void
) () [0] #28 https://github.com/StanfordLegion/legion/pull/28
0x0000000000a50100 in greenlet::_run(void) ()
[0] #29 https://github.com/StanfordLegion/legion/pull/29
0x0000000000a4f85c in _greenlet_start(void
) () [0] #30 https://github.com/StanfordLegion/legion/pull/30
0x0000000000000000 in ?? () [0] Thread 5 (Thread 0x2ab934921700 (LWP
65434)): [0] #0 0x00002ab90abfd5bc in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0 [0] #1
https://github.com/StanfordLegion/legion/issues/1 0x00000000009c6bca in
GASNetCondVar::wait() () [0] #2
https://github.com/StanfordLegion/legion/pull/2 0x0000000000a0b257 in
Realm::GreenletProcessor::execute_task() () [0] #3
https://github.com/StanfordLegion/legion/pull/3 0x0000000000a0a539 in
Realm::GreenletThread::thread_main() () [0] #4
https://github.com/StanfordLegion/legion/issues/4 0x0000000000a0a1a9 in
Realm::PreemptableThread::thread_entry(void) ()
[0] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0
[0] #6 https://github.com/StanfordLegion/legion/issues/6
0x00002ab90d2768fd in clone () from /lib64/libc.so.6
[0] Thread 4 (Thread 0x2ab934b22700 (LWP 65435)):
[0] #0 0x00002ab90abfd5bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[0] #1 https://github.com/StanfordLegion/legion/issues/1
0x00000000009c6bca in GASNetCondVar::wait() ()
[0] #2 https://github.com/StanfordLegion/legion/pull/2
0x0000000000a0b257 in Realm::GreenletProcessor::execute_task() ()
[0] #3 https://github.com/StanfordLegion/legion/pull/3
0x0000000000a0a539 in Realm::GreenletThread::thread_main() ()
[0] #4 https://github.com/StanfordLegion/legion/issues/4
0x0000000000a0a1a9 in Realm::PreemptableThread::thread_entry(void
) () [0] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0 [0] #6
https://github.com/StanfordLegion/legion/issues/6 0x00002ab90d2768fd in
clone () from /lib64/libc.so.6 [0] Thread 3 (Thread 0x2ab935000700 (LWP
65436)): [0] #0 0x00002ab90abfd5bc in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0 [0] #1
https://github.com/StanfordLegion/legion/issues/1 0x00000000009c6bca in
GASNetCondVar::wait() () [0] #2
https://github.com/StanfordLegion/legion/pull/2 0x0000000000a0b257 in
Realm::GreenletProcessor::execute_task() () [0] #3
https://github.com/StanfordLegion/legion/pull/3 0x0000000000a0a539 in
Realm::GreenletThread::thread_main() () [0] #4
https://github.com/StanfordLegion/legion/issues/4 0x0000000000a0a1a9 in
Realm::PreemptableThread::thread_entry(void) ()
[0] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0
[0] #6 https://github.com/StanfordLegion/legion/issues/6
0x00002ab90d2768fd in clone () from /lib64/libc.so.6
[0] Thread 2 (Thread 0x2ab935201700 (LWP 65437)):
[0] #0 0x00002ab90abfd5bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[0] #1 https://github.com/StanfordLegion/legion/issues/1
0x00000000009c6bca in GASNetCondVar::wait() ()
[0] #2 https://github.com/StanfordLegion/legion/pull/2
0x0000000000a0b257 in Realm::GreenletProcessor::execute_task() ()
[0] #3 https://github.com/StanfordLegion/legion/pull/3
0x0000000000a0a539 in Realm::GreenletThread::thread_main() ()
[0] #4 https://github.com/StanfordLegion/legion/issues/4
0x0000000000a0a1a9 in Realm::PreemptableThread::thread_entry(void
) () [0] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002ab90abf99d1 in start_thread () from /lib64/libpthread.so.0 [0] #6
https://github.com/StanfordLegion/legion/issues/6 0x00002ab90d2768fd in
clone () from /lib64/libc.so.6 [0] Thread 1 (Thread 0x2ab911c5d1e0 (LWP
65423)): [0] #0 0x00002ab90d23aa3d in nanosleep () from /lib64/libc.so.6
[0] #1 https://github.com/StanfordLegion/legion/issues/1
0x00002ab90d23a8b0 in sleep () from /lib64/libc.so.6 [0] #2
https://github.com/StanfordLegion/legion/pull/2 0x0000000000a412c7 in
Realm::RuntimeImpl::run(unsigned int, Realm::Runtime::RunStyle, void const,
unsigned long, bool) ()
[0] #3 https://github.com/StanfordLegion/legion/pull/3
0x0000000000a3e00c in Realm::Runtime::run(unsigned int,
Realm::Runtime::RunStyle, void const
, unsigned long, bool) () [0] #4
https://github.com/StanfordLegion/legion/issues/4 0x0000000000c7be7f in
LegionRuntime::HighLevel::Runtime::start(int, char
, bool) () [0] #5 https://github.com/StanfordLegion/legion/pull/5
0x0000000000a623cf in
LegionRuntime::HighLevel::HighLevelRuntime::start(int, char
, bool) () [0] #6 https://github.com/StanfordLegion/legion/issues/6
0x000000000097b33b in main () [2] /usr/bin/gstack 98078 [2] Thread 14
(Thread 0x2b35a9d6c700 (LWP 98079)): [2] #0 0x00002b35a4f4d373 in select ()
from /lib64/libc.so.6 [2] #1
https://github.com/StanfordLegion/legion/issues/1 0x00002b35a2fe165f in
service_thread_start () from
/usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[2] #2 https://github.com/StanfordLegion/legion/pull/2 0x00002b35a28d79d1
in start_thread () from /lib64/libpthread.so.0 [2] #3
https://github.com/StanfordLegion/legion/pull/3 0x00002b35a4f548fd in
clone () from /lib64/libc.so.6 [2] Thread 13 (Thread 0x2b35aa5a1700 (LWP
98080)): [2] #0 0x00002b35a4f4b0d3 in poll () from /lib64/libc.so.6 [2] #1
https://github.com/StanfordLegion/legion/issues/1 0x00002b35a2fdfef0 in
btl_openib_async_thread () from
/usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[2] #2 https://github.com/StanfordLegion/legion/pull/2 0x00002b35a28d79d1
in start_thread () from /lib64/libpthread.so.0 [2] #3
https://github.com/StanfordLegion/legion/pull/3 0x00002b35a4f548fd in
clone () from /lib64/libc.so.6 [2] Thread 12 (Thread 0x2b35ab816700 (LWP
98082)): [2] #0 0x00002b35a4f3b287 in sched_yield () from /lib64/libc.so.6
[2] #1 https://github.com/StanfordLegion/legion/issues/1
0x0000000000d26931 in gasnetc_RequestGeneric () [2] #2
https://github.com/StanfordLegion/legion/pull/2 0x0000000000d15d72 in
gasnetc_AMRequestShortM () [2] #3
https://github.com/StanfordLegion/legion/pull/3 0x00000000009b3d7f in
ActiveMessageEndpoint::send_short(OutgoingMessage) ()
[2] #4 https://github.com/StanfordLegion/legion/issues/4
0x00000000009b2f1e in ActiveMessageEndpoint::push_messages(int, bool) ()
[2] #5 https://github.com/StanfordLegion/legion/pull/5
0x00000000009b6e7d in EndpointManager::push_messages(int, bool) ()
[2] #6 https://github.com/StanfordLegion/legion/issues/6
0x00000000009b0c2f in do_some_polling() ()
[2] #7 https://github.com/StanfordLegion/legion/pull/7
0x00000000009b0d2c in gasnet_poll_thread_loop(void
) () [2] #8 https://github.com/StanfordLegion/legion/pull/8
0x00002b35a28d79d1 in start_thread () from /lib64/libpthread.so.0 [2] #9
https://github.com/StanfordLegion/legion/issues/9 0x00002b35a4f548fd in
clone () from /lib64/libc.so.6 [2] Thread 11 (Thread 0x2b35aba17700 (LWP
98083)): [2] #0 0x00002b35a28db5bc in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0 [2] #1
https://github.com/StanfordLegion/legion/issues/1 0x00000000009afb74 in
IncomingMessageManager::get_messages(int&, bool) () [2] #2
https://github.com/StanfordLegion/legion/pull/2 0x00000000009afcfd in
Realm::HandlerThread::thread_main() () [2] #3
https://github.com/StanfordLegion/legion/pull/3 0x0000000000a0a1a9 in
Realm::PreemptableThread::thread_entry(void) ()
[2] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002b35a28d79d1 in start_thread () from /lib64/libpthread.so.0
[2] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002b35a4f548fd in clone () from /lib64/libc.so.6
[2] Thread 10 (Thread 0x2b35cbe00700 (LWP 98084)):
[2] #0 0x00002b35a28db5bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[2] #1 https://github.com/StanfordLegion/legion/issues/1
0x000000000098f75c in
LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread
, bool) () [2] #2 https://github.com/StanfordLegion/legion/pull/2
0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop()
() [2] #3 https://github.com/StanfordLegion/legion/pull/3
0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void) ()
[2] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002b35a28d79d1 in start_thread () from /lib64/libpthread.so.0
[2] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002b35a4f548fd in clone () from /lib64/libc.so.6
[2] Thread 9 (Thread 0x2b35cc001700 (LWP 98085)):
[2] #0 0x00002b35a28db5bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[2] #1 https://github.com/StanfordLegion/legion/issues/1
0x000000000098f75c in
LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread
, bool) () [2] #2 https://github.com/StanfordLegion/legion/pull/2
0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop()
() [2] #3 https://github.com/StanfordLegion/legion/pull/3
0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void) ()
[2] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002b35a28d79d1 in start_thread () from /lib64/libpthread.so.0
[2] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002b35a4f548fd in clone () from /lib64/libc.so.6
[2] Thread 8 (Thread 0x2b35cc202700 (LWP 98086)):
[2] #0 0x00002b35a28db5bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[2] #1 https://github.com/StanfordLegion/legion/issues/1
0x000000000098f75c in
LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread
, bool) () [2] #2 https://github.com/StanfordLegion/legion/pull/2
0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop()
() [2] #3 https://github.com/StanfordLegion/legion/pull/3
0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void) ()
[2] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002b35a28d79d1 in start_thread () from /lib64/libpthread.so.0
[2] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002b35a4f548fd in clone () from /lib64/libc.so.6
[2] Thread 7 (Thread 0x2b35cc403700 (LWP 98087)):
[2] #0 0x00002b35a28db5bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[2] #1 https://github.com/StanfordLegion/legion/issues/1
0x000000000098f75c in
LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread
, bool) () [2] #2 https://github.com/StanfordLegion/legion/pull/2
0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop()
() [2] #3 https://github.com/StanfordLegion/legion/pull/3
0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void) ()
[2] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002b35a28d79d1 in start_thread () from /lib64/libpthread.so.0
[2] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002b35a4f548fd in clone () from /lib64/libc.so.6
[2] Thread 6 (Thread 0x2b35cc604700 (LWP 98088)):
[2] #0 0x00002b35a4f3b287 in sched_yield () from /lib64/libc.so.6
[2] #1 https://github.com/StanfordLegion/legion/issues/1
0x0000000000d3afbe in gasneti_bt_gstack ()
[2] #2 https://github.com/StanfordLegion/legion/pull/2
0x0000000000d3c0ff in gasneti_print_backtrace ()
[2] #3 https://github.com/StanfordLegion/legion/pull/3
0x0000000000d9fb4e in gasneti_defaultSignalHandler ()
[2] #4 https://github.com/StanfordLegion/legion/issues/4
[2] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002b35a4e9e625 in raise () from /lib64/libc.so.6
[2] #6 https://github.com/StanfordLegion/legion/issues/6
0x00002b35a4e9fe05 in abort () from /lib64/libc.so.6
[2] #7 https://github.com/StanfordLegion/legion/pull/7
0x00002b35a4e9774e in __assert_fail_base () from /lib64/libc.so.6
[2] #8 https://github.com/StanfordLegion/legion/pull/8
0x00002b35a4e97810 in __assert_fail () from /lib64/libc.so.6
[2] #9 https://github.com/StanfordLegion/legion/issues/9
0x00000000009c0383 in
LegionRuntime::LowLevel::find_shortest_path(Realm::Memory, Realm::Memory,
std::vector >&) ()
[2] #10 https://github.com/StanfordLegion/legion/pull/10
0x00000000009cfc39 in void
LegionRuntime::LowLevel::CopyRequest::perform_new_dma<3u>(Realm::Memory,
Realm::Memory) ()
[2] #11 https://github.com/StanfordLegion/legion/pull/11
0x00000000009c0619 in LegionRuntime::LowLevel::CopyRequest::perform_dma() ()
[2] #12 https://github.com/StanfordLegion/legion/pull/12
0x00000000009bec95 in
LegionRuntime::LowLevel::CopyRequest::check_readiness(bool,
LegionRuntime::LowLevel::DmaRequestQueue
*) () [2] #13 https://github.com/StanfordLegion/legion/pull/13
0x00000000009c5251 in Realm::Domain::copy(std::vector > const&, std::vector

const&, Realm::ProfilingRequestSet const&, Realm::Event, int, bool) const
() [2] #14 https://github.com/StanfordLegion/legion/pull/14
0x00000000009c4acd in Realm::Domain::copy(std::vector > const&, std::vector
const&, Realm::Event, int, bool) const () [2] #15
https://github.com/StanfordLegion/legion/pull/15 0x0000000000b46163 in
LegionRuntime::HighLevel::RegionTreeForest::issue_copy(Realm::Domain
const&, LegionRuntime::HighLevel::Operation*, std::vector > const&,
std::vector > const&, Realm::Event) ()
[2] #16 https://github.com/StanfordLegion/legion/issues/16
0x0000000000b3fdfc in
LegionRuntime::HighLevel::RegionTreeForest::copy_across(LegionRuntime::HighLevel::Operation

, LegionRuntime::HighLevel::RegionTreeContext,
LegionRuntime::HighLevel::RegionTreeContext,
LegionRuntime::HighLevel::RegionRequirement const&,
LegionRuntime::HighLevel::RegionRequirement const&,
LegionRuntime::HighLevel::InstanceRef const&,
LegionRuntime::HighLevel::InstanceRef const&, Realm::Event) () [2] #17
https://github.com/StanfordLegion/legion/issues/17 0x0000000000a84b4f in
LegionRuntime::HighLevel::CopyOp::trigger_execution() () [2] #18
https://github.com/StanfordLegion/legion/issues/18 0x0000000000c7e448 in
LegionRuntime::HighLevel::Runtime::high_level_runtime_task(void const,
unsigned long, Realm::Processor) ()
[2] #19 https://github.com/StanfordLegion/legion/issues/19
0x0000000000a09fe8 in Realm::PreemptableThread::run_task(Realm::Task
, Realm::Processor) () [2] #20
https://github.com/StanfordLegion/legion/pull/20 0x0000000000a0a420 in
Realm::GreenletTask::run(void) ()
[2] #21 https://github.com/StanfordLegion/legion/pull/21
0x0000000000a50100 in greenlet::_run(void
) () [2] #22 https://github.com/StanfordLegion/legion/pull/22
0x0000000000a4f85c in _greenlet_start(void) ()
[2] #23 https://github.com/StanfordLegion/legion/pull/23
0x0000000000000000 in ?? ()
[2] Thread 5 (Thread 0x2b35cc805700 (LWP 98089)):
[2] #0 0x00002b35a28db5bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[2] #1 https://github.com/StanfordLegion/legion/issues/1
0x00000000009c6bca in GASNetCondVar::wait() ()
[2] #2 https://github.com/StanfordLegion/legion/pull/2
0x0000000000a0b257 in Realm::GreenletProcessor::execute_task() ()
[2] #3 https://github.com/StanfordLegion/legion/pull/3
0x0000000000a0a539 in Realm::GreenletThread::thread_main() ()
[2] #4 https://github.com/StanfordLegion/legion/issues/4
0x0000000000a0a1a9 in Realm::PreemptableThread::thread_entry(void
) () [2] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002b35a28d79d1 in start_thread () from /lib64/libpthread.so.0 [2] #6
https://github.com/StanfordLegion/legion/issues/6 0x00002b35a4f548fd in
clone () from /lib64/libc.so.6 [2] Thread 4 (Thread 0x2b35cca06700 (LWP
98090)): [2] #0 0x00002b35a28db5bc in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0 [2] #1
https://github.com/StanfordLegion/legion/issues/1 0x00000000009c6bca in
GASNetCondVar::wait() () [2] #2
https://github.com/StanfordLegion/legion/pull/2 0x0000000000a0b257 in
Realm::GreenletProcessor::execute_task() () [2] #3
https://github.com/StanfordLegion/legion/pull/3 0x0000000000a0a539 in
Realm::GreenletThread::thread_main() () [2] #4
https://github.com/StanfordLegion/legion/issues/4 0x0000000000a0a1a9 in
Realm::PreemptableThread::thread_entry(void) ()
[2] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002b35a28d79d1 in start_thread () from /lib64/libpthread.so.0
[2] #6 https://github.com/StanfordLegion/legion/issues/6
0x00002b35a4f548fd in clone () from /lib64/libc.so.6
[2] Thread 3 (Thread 0x2b35ccc07700 (LWP 98091)):
[2] #0 0x00002b35a28db5bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[2] #1 https://github.com/StanfordLegion/legion/issues/1
0x00000000009c6bca in GASNetCondVar::wait() ()
[2] #2 https://github.com/StanfordLegion/legion/pull/2
0x0000000000a0b257 in Realm::GreenletProcessor::execute_task() ()
[2] #3 https://github.com/StanfordLegion/legion/pull/3
0x0000000000a0a539 in Realm::GreenletThread::thread_main() ()
[2] #4 https://github.com/StanfordLegion/legion/issues/4
0x0000000000a0a1a9 in Realm::PreemptableThread::thread_entry(void
) () [2] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002b35a28d79d1 in start_thread () from /lib64/libpthread.so.0 [2] #6
https://github.com/StanfordLegion/legion/issues/6 0x00002b35a4f548fd in
clone () from /lib64/libc.so.6 [2] Thread 2 (Thread 0x2b35cce08700 (LWP
98092)): [2] #0 0x00002b35a28db5bc in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0 [2] #1
https://github.com/StanfordLegion/legion/issues/1 0x00000000009c6bca in
GASNetCondVar::wait() () [2] #2
https://github.com/StanfordLegion/legion/pull/2 0x0000000000a0b257 in
Realm::GreenletProcessor::execute_task() () [2] #3
https://github.com/StanfordLegion/legion/pull/3 0x0000000000a0a539 in
Realm::GreenletThread::thread_main() () [2] #4
https://github.com/StanfordLegion/legion/issues/4 0x0000000000a0a1a9 in
Realm::PreemptableThread::thread_entry(void) ()
[2] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002b35a28d79d1 in start_thread () from /lib64/libpthread.so.0
[2] #6 https://github.com/StanfordLegion/legion/issues/6
0x00002b35a4f548fd in clone () from /lib64/libc.so.6
[2] Thread 1 (Thread 0x2b35a993b1e0 (LWP 98078)):
[2] #0 0x00002b35a4f18a3d in nanosleep () from /lib64/libc.so.6
[2] #1 https://github.com/StanfordLegion/legion/issues/1
0x00002b35a4f188b0 in sleep () from /lib64/libc.so.6
[2] #2 https://github.com/StanfordLegion/legion/pull/2
0x0000000000a412c7 in Realm::RuntimeImpl::run(unsigned int,
Realm::Runtime::RunStyle, void const
, unsigned long, bool) () [2] #3
https://github.com/StanfordLegion/legion/pull/3 0x0000000000a3e00c in
Realm::Runtime::run(unsigned int, Realm::Runtime::RunStyle, void const,
unsigned long, bool) ()
[2] #4 https://github.com/StanfordLegion/legion/issues/4
0x0000000000c7be7f in LegionRuntime::HighLevel::Runtime::start(int, char
, bool) () [2] #5 https://github.com/StanfordLegion/legion/pull/5
0x0000000000a623cf in
LegionRuntime::HighLevel::HighLevelRuntime::start(int, char, bool) ()
[2] #6 https://github.com/StanfordLegion/legion/issues/6
0x000000000097b33b in main ()
[1] /usr/bin/gstack 71570
[1] Thread 14 (Thread 0x2ac20e682700 (LWP 71571)):
[1] #0 0x00002ac209863373 in select () from /lib64/libc.so.6
[1] #1 https://github.com/StanfordLegion/legion/issues/1
0x00002ac2078f765f in service_thread_start () from
/usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[1] #2 https://github.com/StanfordLegion/legion/pull/2
0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0
[1] #3 https://github.com/StanfordLegion/legion/pull/3
0x00002ac20986a8fd in clone () from /lib64/libc.so.6
[1] Thread 13 (Thread 0x2ac20eeb7700 (LWP 71572)):
[1] #0 0x00002ac2098610d3 in poll () from /lib64/libc.so.6
[1] #1 https://github.com/StanfordLegion/legion/issues/1
0x00002ac2078f5ef0 in btl_openib_async_thread () from
/usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[1] #2 https://github.com/StanfordLegion/legion/pull/2
0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0
[1] #3 https://github.com/StanfordLegion/legion/pull/3
0x00002ac20986a8fd in clone () from /lib64/libc.so.6
[1] Thread 12 (Thread 0x2ac21010b700 (LWP 71574)):
[1] #0 0x00002ac2071f2380 in pthread_spin_lock () from
/lib64/libpthread.so.0
[1] #1 https://github.com/StanfordLegion/legion/issues/1
0x00002ac20e6a517f in ?? () from /usr/lib64/libipathverbs-rdmav2.so
[1] #2 https://github.com/StanfordLegion/legion/pull/2
0x0000000000d24a41 in gasnetc_rcv_reap ()
[1] #3 https://github.com/StanfordLegion/legion/pull/3
0x0000000000d25209 in gasnetc_AMPoll ()
[1] #4 https://github.com/StanfordLegion/legion/issues/4
0x00000000009b0c34 in do_some_polling() ()
[1] #5 https://github.com/StanfordLegion/legion/pull/5
0x00000000009b0d2c in gasnet_poll_thread_loop(void
) () [1] #6 https://github.com/StanfordLegion/legion/issues/6
0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0 [1] #7
https://github.com/StanfordLegion/legion/pull/7 0x00002ac20986a8fd in
clone () from /lib64/libc.so.6 [1] Thread 11 (Thread 0x2ac21030c700 (LWP
71575)): [1] #0 0x00002ac2071f15bc in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0 [1] #1
https://github.com/StanfordLegion/legion/issues/1 0x00000000009afb74 in
IncomingMessageManager::get_messages(int&, bool) () [1] #2
https://github.com/StanfordLegion/legion/pull/2 0x00000000009afcfd in
Realm::HandlerThread::thread_main() () [1] #3
https://github.com/StanfordLegion/legion/pull/3 0x0000000000a0a1a9 in
Realm::PreemptableThread::thread_entry(void) ()
[1] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0
[1] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002ac20986a8fd in clone () from /lib64/libc.so.6
[1] Thread 10 (Thread 0x2ac230510700 (LWP 71576)):
[1] #0 0x00002ac2071f15bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[1] #1 https://github.com/StanfordLegion/legion/issues/1
0x000000000098f75c in
LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread
, bool) () [1] #2 https://github.com/StanfordLegion/legion/pull/2
0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop()
() [1] #3 https://github.com/StanfordLegion/legion/pull/3
0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void) ()
[1] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0
[1] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002ac20986a8fd in clone () from /lib64/libc.so.6
[1] Thread 9 (Thread 0x2ac230711700 (LWP 71577)):
[1] #0 0x00002ac2071f15bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[1] #1 https://github.com/StanfordLegion/legion/issues/1
0x000000000098f75c in
LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread
, bool) () [1] #2 https://github.com/StanfordLegion/legion/pull/2
0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop()
() [1] #3 https://github.com/StanfordLegion/legion/pull/3
0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void) ()
[1] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0
[1] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002ac20986a8fd in clone () from /lib64/libc.so.6
[1] Thread 8 (Thread 0x2ac230912700 (LWP 71578)):
[1] #0 0x00002ac2071f15bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[1] #1 https://github.com/StanfordLegion/legion/issues/1
0x000000000098f75c in
LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread
, bool) () [1] #2 https://github.com/StanfordLegion/legion/pull/2
0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop()
() [1] #3 https://github.com/StanfordLegion/legion/pull/3
0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void) ()
[1] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0
[1] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002ac20986a8fd in clone () from /lib64/libc.so.6
[1] Thread 7 (Thread 0x2ac230b13700 (LWP 71579)):
[1] #0 0x00002ac2071f15bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[1] #1 https://github.com/StanfordLegion/legion/issues/1
0x000000000098f75c in
LegionRuntime::LowLevel::XferDesQueue::dequeue_xferDes(LegionRuntime::LowLevel::DMAThread
, bool) () [1] #2 https://github.com/StanfordLegion/legion/pull/2
0x000000000098c5f9 in LegionRuntime::LowLevel::DMAThread::dma_thread_loop()
() [1] #3 https://github.com/StanfordLegion/legion/pull/3
0x00000000009c7792 in LegionRuntime::LowLevel::DMAThread::start(void) ()
[1] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0
[1] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002ac20986a8fd in clone () from /lib64/libc.so.6
[1] Thread 6 (Thread 0x2ac230d14700 (LWP 71580)):
[1] #0 0x00002ac209851287 in sched_yield () from /lib64/libc.so.6
[1] #1 https://github.com/StanfordLegion/legion/issues/1
0x0000000000d3afbe in gasneti_bt_gstack ()
[1] #2 https://github.com/StanfordLegion/legion/pull/2
0x0000000000d3c0ff in gasneti_print_backtrace ()
[1] #3 https://github.com/StanfordLegion/legion/pull/3
0x0000000000d9fb4e in gasneti_defaultSignalHandler ()
[1] #4 https://github.com/StanfordLegion/legion/issues/4
[1] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002ac2097b4625 in raise () from /lib64/libc.so.6
[1] #6 https://github.com/StanfordLegion/legion/issues/6
0x00002ac2097b5e05 in abort () from /lib64/libc.so.6
[1] #7 https://github.com/StanfordLegion/legion/pull/7
0x00002ac2097ad74e in __assert_fail_base () from /lib64/libc.so.6
[1] #8 https://github.com/StanfordLegion/legion/pull/8
0x00002ac2097ad810 in __assert_fail () from /lib64/libc.so.6
[1] #9 https://github.com/StanfordLegion/legion/issues/9
0x00000000009c0383 in
LegionRuntime::LowLevel::find_shortest_path(Realm::Memory, Realm::Memory,
std::vector >&) ()
[1] #10 https://github.com/StanfordLegion/legion/pull/10
0x00000000009cfc39 in void
LegionRuntime::LowLevel::CopyRequest::perform_new_dma<3u>(Realm::Memory,
Realm::Memory) ()
[1] #11 https://github.com/StanfordLegion/legion/pull/11
0x00000000009c0619 in LegionRuntime::LowLevel::CopyRequest::perform_dma() ()
[1] #12 https://github.com/StanfordLegion/legion/pull/12
0x00000000009bec95 in
LegionRuntime::LowLevel::CopyRequest::check_readiness(bool,
LegionRuntime::LowLevel::DmaRequestQueue
*) () [1] #13 https://github.com/StanfordLegion/legion/pull/13
0x00000000009c5251 in Realm::Domain::copy(std::vector > const&, std::vector

const&, Realm::ProfilingRequestSet const&, Realm::Event, int, bool) const
() [1] #14 https://github.com/StanfordLegion/legion/pull/14
0x00000000009c4acd in Realm::Domain::copy(std::vector > const&, std::vector
const&, Realm::Event, int, bool) const () [1] #15
https://github.com/StanfordLegion/legion/pull/15 0x0000000000b46163 in
LegionRuntime::HighLevel::RegionTreeForest::issue_copy(Realm::Domain
const&, LegionRuntime::HighLevel::Operation*, std::vector > const&,
std::vector > const&, Realm::Event) ()
[1] #16 https://github.com/StanfordLegion/legion/issues/16
0x0000000000b3fdfc in
LegionRuntime::HighLevel::RegionTreeForest::copy_across(LegionRuntime::HighLevel::Operation

, LegionRuntime::HighLevel::RegionTreeContext,
LegionRuntime::HighLevel::RegionTreeContext,
LegionRuntime::HighLevel::RegionRequirement const&,
LegionRuntime::HighLevel::RegionRequirement const&,
LegionRuntime::HighLevel::InstanceRef const&,
LegionRuntime::HighLevel::InstanceRef const&, Realm::Event) () [1] #17
https://github.com/StanfordLegion/legion/issues/17 0x0000000000a84b4f in
LegionRuntime::HighLevel::CopyOp::trigger_execution() () [1] #18
https://github.com/StanfordLegion/legion/issues/18 0x0000000000c7e448 in
LegionRuntime::HighLevel::Runtime::high_level_runtime_task(void const,
unsigned long, Realm::Processor) ()
[1] #19 https://github.com/StanfordLegion/legion/issues/19
0x0000000000a09fe8 in Realm::PreemptableThread::run_task(Realm::Task
, Realm::Processor) () [1] #20
https://github.com/StanfordLegion/legion/pull/20 0x0000000000a0a420 in
Realm::GreenletTask::run(void) ()
[1] #21 https://github.com/StanfordLegion/legion/pull/21
0x0000000000a50100 in greenlet::_run(void
) () [1] #22 https://github.com/StanfordLegion/legion/pull/22
0x0000000000a4f85c in _greenlet_start(void) ()
[1] #23 https://github.com/StanfordLegion/legion/pull/23
0x0000000000000000 in ?? ()
[1] Thread 5 (Thread 0x2ac230f15700 (LWP 71581)):
[1] #0 0x00002ac2071f15bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[1] #1 https://github.com/StanfordLegion/legion/issues/1
0x00000000009c6bca in GASNetCondVar::wait() ()
[1] #2 https://github.com/StanfordLegion/legion/pull/2
0x0000000000a0b257 in Realm::GreenletProcessor::execute_task() ()
[1] #3 https://github.com/StanfordLegion/legion/pull/3
0x0000000000a0a539 in Realm::GreenletThread::thread_main() ()
[1] #4 https://github.com/StanfordLegion/legion/issues/4
0x0000000000a0a1a9 in Realm::PreemptableThread::thread_entry(void
) () [1] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0 [1] #6
https://github.com/StanfordLegion/legion/issues/6 0x00002ac20986a8fd in
clone () from /lib64/libc.so.6 [1] Thread 4 (Thread 0x2ac231300700 (LWP
71582)): [1] #0 0x00002ac2071f15bc in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0 [1] #1
https://github.com/StanfordLegion/legion/issues/1 0x00000000009c6bca in
GASNetCondVar::wait() () [1] #2
https://github.com/StanfordLegion/legion/pull/2 0x0000000000a0b257 in
Realm::GreenletProcessor::execute_task() () [1] #3
https://github.com/StanfordLegion/legion/pull/3 0x0000000000a0a539 in
Realm::GreenletThread::thread_main() () [1] #4
https://github.com/StanfordLegion/legion/issues/4 0x0000000000a0a1a9 in
Realm::PreemptableThread::thread_entry(void) ()
[1] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0
[1] #6 https://github.com/StanfordLegion/legion/issues/6
0x00002ac20986a8fd in clone () from /lib64/libc.so.6
[1] Thread 3 (Thread 0x2ac231501700 (LWP 71583)):
[1] #0 0x00002ac2071f15bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[1] #1 https://github.com/StanfordLegion/legion/issues/1
0x00000000009c6bca in GASNetCondVar::wait() ()
[1] #2 https://github.com/StanfordLegion/legion/pull/2
0x0000000000a0b257 in Realm::GreenletProcessor::execute_task() ()
[1] #3 https://github.com/StanfordLegion/legion/pull/3
0x0000000000a0a539 in Realm::GreenletThread::thread_main() ()
[1] #4 https://github.com/StanfordLegion/legion/issues/4
0x0000000000a0a1a9 in Realm::PreemptableThread::thread_entry(void
) () [1] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0 [1] #6
https://github.com/StanfordLegion/legion/issues/6 0x00002ac20986a8fd in
clone () from /lib64/libc.so.6 [1] Thread 2 (Thread 0x2ac231702700 (LWP
71584)): [1] #0 0x00002ac2071f15bc in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0 [1] #1
https://github.com/StanfordLegion/legion/issues/1 0x00000000009c6bca in
GASNetCondVar::wait() () [1] #2
https://github.com/StanfordLegion/legion/pull/2 0x0000000000a0b257 in
Realm::GreenletProcessor::execute_task() () [1] #3
https://github.com/StanfordLegion/legion/pull/3 0x0000000000a0a539 in
Realm::GreenletThread::thread_main() () [1] #4
https://github.com/StanfordLegion/legion/issues/4 0x0000000000a0a1a9 in
Realm::PreemptableThread::thread_entry(void) ()
[1] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002ac2071ed9d1 in start_thread () from /lib64/libpthread.so.0
[1] #6 https://github.com/StanfordLegion/legion/issues/6
0x00002ac20986a8fd in clone () from /lib64/libc.so.6
[1] Thread 1 (Thread 0x2ac20e2511e0 (LWP 71570)):
[1] #0 0x00002ac20982ea3d in nanosleep () from /lib64/libc.so.6
[1] #1 https://github.com/StanfordLegion/legion/issues/1
0x00002ac20982e8b0 in sleep () from /lib64/libc.so.6
[1] #2 https://github.com/StanfordLegion/legion/pull/2
0x0000000000a412c7 in Realm::RuntimeImpl::run(unsigned int,
Realm::Runtime::RunStyle, void const
, unsigned long, bool) () [1] #3
https://github.com/StanfordLegion/legion/pull/3 0x0000000000a3e00c in
Realm::Runtime::run(unsigned int, Realm::Runtime::RunStyle, void const,
unsigned long, bool) ()
[1] #4 https://github.com/StanfordLegion/legion/issues/4
0x0000000000c7be7f in LegionRuntime::HighLevel::Runtime::start(int, char
, bool) () [1] #5 https://github.com/StanfordLegion/legion/pull/5
0x0000000000a623cf in
LegionRuntime::HighLevel::HighLevelRuntime::start(int, char, bool) ()
[1] #6 https://github.com/StanfordLegion/legion/issues/6
0x000000000097b33b in main () mpirun noticed that process rank 1 with PID
71570 on node ml137 exited on signal 6 (Aborted).
*** Caught a signal: SIGTERM(15) on node 3/4
—
Reply to this email directly or view it on GitHub
https://github.com/StanfordLegion/legion/issues/51.

--
Sincerely,
Zhihao

Different failure:
Reported: 4 (out of 4) daemons - 4 (out of 4) procs
Pinned Memory Usage: GASNET=256, RMEM=1024, LMB=8, SDP=64, total=1352
tester_io: /users/gshipman/local/src/legion-2/runtime//activemsg.cc:717: void enqueue_incoming(gasnet_node_t, IncomingMessage*): Assertion incoming_message_manager != 0' failed. *** Caught a fatal signal: SIGABRT(6) on node 2/4 tester_io: /users/gshipman/local/src/legion-2/runtime//activemsg.cc:717: void enqueue_incoming(gasnet_node_t, IncomingMessage*): Assertion incoming_message_manager != 0' failed.
*** Caught a fatal signal: SIGABRT(6) on node 3/4
tester_io: /users/gshipman/local/src/legion-2/runtime//activemsg.cc:717: void enqueue_incoming(gasnet_node_t, IncomingMessage*): Assertion `incoming_message_manager != 0' failed.
*** Caught a fatal signal: SIGABRT(6) on node 1/4
[3] /usr/bin/gstack 30952
[3] Thread 7 (Thread 0x2b9416c00700 (LWP 30953)):
[3] #0  0x00002b9411de3373 in select () from /lib64/libc.so.6
[3] #1  0x00002b940fe7765f in service_thread_start () from /usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[3] #2  0x00002b940f76d9d1 in start_thread () from /lib64/libpthread.so.0
[3] #3  0x00002b9411dea8fd in clone () from /lib64/libc.so.6
[3] Thread 6 (Thread 0x2b9417433700 (LWP 30954)):
[3] #0  0x00002b9411de10d3 in poll () from /lib64/libc.so.6
[3] #1  0x00002b940fe75ef0 in btl_openib_async_thread () from /usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[3] #2  0x00002b940f76d9d1 in start_thread () from /lib64/libpthread.so.0
[3] #3  0x00002b9411dea8fd in clone () from /lib64/libc.so.6
[3] Thread 5 (Thread 0x2b9418792700 (LWP 30955)):
[3] #0  0x00002b9411dd1287 in sched_yield () from /lib64/libc.so.6
[3] #1  0x0000000000d31e7e in gasneti_bt_gstack ()
[3] #2  0x0000000000d32fbf in gasneti_print_backtrace ()
[3] #3  0x0000000000d96a0e in gasneti_defaultSignalHandler ()
[3] #4
[3] #5  0x00002b9411d34625 in raise () from /lib64/libc.so.6
[3] #6  0x00002b9411d35e05 in abort () from /lib64/libc.so.6
[3] #7  0x00002b9411d2d74e in __assert_fail_base () from /lib64/libc.so.6
[3] #8  0x00002b9411d2d810 in __assert_fail () from /lib64/libc.so.6
[3] #9  0x00000000009af384 in enqueue_incoming(unsigned short, IncomingMessage*) ()
[3] #10 0x0000000000a41b35 in MessageRawArgs<Realm::NodeAnnounceMessage::RequestArgs, 141, &(void dummy_short_handlerRealm::NodeAnnounceMessage::RequestArgs(Realm::NodeAnnounceMessage::RequestArgs)), &(Realm::NodeAnnounceMessage::handle_request(Realm::NodeAnnounceMessage::RequestArgs, void const*, unsigned long)), 8>::handler_medium(void*, void*, unsigned long, int, int, int, int, int, int, int, int) ()
[3] #11 0x0000000000d16898 in gasnetc_processPacket ()
[3] #12 0x0000000000d1bb51 in gasnetc_rcv_reap ()
[3] #13 0x0000000000d1c0c9 in gasnetc_AMPoll ()
[3] #14 0x00000000009b02fd in do_some_polling() ()
[3] #15 0x00000000009b03f5 in gasnet_poll_thread_loop(void*) ()
[3] #16 0x00002b940f76d9d1 in start_thread () from /lib64/libpthread.so.0
[3] #17 0x00002b9411dea8fd in clone () from /lib64/libc.so.6
[3] Thread 4 (Thread 0x2b9418996700 (LWP 30957)):
[3] #0  0x00002b940f7715bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[3] #1  0x00000000009af246 in IncomingMessageManager::get_messages(int&, bool) ()
[3] #2  0x00000000009af3cf in Realm::HandlerThread::thread_main() ()
[3] #3  0x0000000000a07d6f in Realm::PreemptableThread::thread_entry(void*) ()
[3] #4  0x00002b940f76d9d1 in start_thread () from /lib64/libpthread.so.0
[3] #5  0x00002b9411dea8fd in clone () from /lib64/libc.so.6
[3] Thread 3 (Thread 0x2b9418b97700 (LWP 30958)):
[3] #0  0x00002b940f7715bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[3] #1  0x00000000009af246 in IncomingMessageManager::get_messages(int&, bool) ()
[3] #2  0x00000000009af3cf in Realm::HandlerThread::thread_main() ()
[3] #3  0x0000000000a07d6f in Realm::PreemptableThread::thread_entry(void*) ()
[3] #4  0x00002b940f76d9d1 in start_thread () from /lib64/libpthread.so.0
[3] #5  0x00002b9411dea8fd in clone () from /lib64/libc.so.6
[3] Thread 2 (Thread 0x2b9418d98700 (LWP 30959)):
[3] #0  0x00002b940f7715bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[3] #1  0x00000000009af246 in IncomingMessageManager::get_messages(int&, bool) ()
[3] #2  0x00000000009af3cf in Realm::HandlerThread::thread_main() ()
[3] #3  0x0000000000a07d6f in Realm::PreemptableThread::thread_entry(void*) ()
[3] #4  0x00002b940f76d9d1 in start_thread () from /lib64/libpthread.so.0
[3] #5  0x00002b9411dea8fd in clone () from /lib64/libc.so.6
[3] Thread 1 (Thread 0x2b94167d11e0 (LWP 30952)):
[3] #0  0x0000000000d34840 in gasneti_AMPSHMPoll ()
[3] #1  0x0000000000d1c117 in gasnetc_AMPoll ()
[3] #2  0x00000000009b02fd in do_some_polling() ()
[3] #3  0x0000000000a36de9 in Realm::NodeAnnounceMessage::await_all_announcements() ()
[3] #4  0x0000000000a3e34b in Realm::RuntimeImpl::init(int*, char***) ()
[3] #5  0x0000000000a3b5ae in Realm::Runtime::init(int*, char***) ()
[3] #6  0x0000000000c72b8c in LegionRuntime::HighLevel::Runtime::start(int, char**, bool) ()
[3] #7  0x0000000000a5f507 in LegionRuntime::HighLevel::HighLevelRuntime::start(int, char**, bool) ()
[3] #8  0x000000000097ae6b in main ()
[2] /usr/bin/gstack 24412
[2] Thread 7 (Thread 0x2acd2f865700 (LWP 24413)):
[2] #0  0x00002acd2aa48373 in select () from /lib64/libc.so.6
[2] #1  0x00002acd28adc65f in service_thread_start () from /usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[2] #2  0x00002acd283d29d1 in start_thread () from /lib64/libpthread.so.0
[2] #3  0x00002acd2aa4f8fd in clone () from /lib64/libc.so.6
[2] Thread 6 (Thread 0x2acd30098700 (LWP 24414)):
[2] #0  0x00002acd2aa460d3 in poll () from /lib64/libc.so.6
[2] #1  0x00002acd28adaef0 in btl_openib_async_thread () from /usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[2] #2  0x00002acd283d29d1 in start_thread () from /lib64/libpthread.so.0
[2] #3  0x00002acd2aa4f8fd in clone () from /lib64/libc.so.6
[2] Thread 5 (Thread 0x2acd313f7700 (LWP 24415)):
[2] #0  0x00002acd2aa36287 in sched_yield () from /lib64/libc.so.6
[2] #1  0x0000000000d31e7e in gasneti_bt_gstack ()
[2] #2  0x0000000000d32fbf in gasneti_print_backtrace ()
[2] #3  0x0000000000d96a0e in gasneti_defaultSignalHandler ()
[2] #4
[2] #5  0x00002acd2a999625 in raise () from /lib64/libc.so.6
[2] #6  0x00002acd2a99ae05 in abort () from /lib64/libc.so.6
[2] #7  0x00002acd2a99274e in __assert_fail_base () from /lib64/libc.so.6
[2] #8  0x00002acd2a992810 in __assert_fail () from /lib64/libc.so.6
[2] #9  0x00000000009af384 in enqueue_incoming(unsigned short, IncomingMessage*) ()
[2] #10 0x0000000000a41b35 in MessageRawArgs<Realm::NodeAnnounceMessage::RequestArgs, 141, &(void dummy_short_handlerRealm::NodeAnnounceMessage::RequestArgs(Realm::NodeAnnounceMessage::RequestArgs)), &(Realm::NodeAnnounceMessage::handle_request(Realm::NodeAnnounceMessage::RequestArgs, void const*, unsigned long)), 8>::handler_medium(void*, void*, unsigned long, int, int, int, int, int, int, int, int) ()
[2] #11 0x0000000000d16898 in gasnetc_processPacket ()
[2] #12 0x0000000000d1bb51 in gasnetc_rcv_reap ()
[2] #13 0x0000000000d1c0c9 in gasnetc_AMPoll ()
[2] #14 0x00000000009b02fd in do_some_polling() ()
[2] #15 0x00000000009b03f5 in gasnet_poll_thread_loop(void*) ()
[2] #16 0x00002acd283d29d1 in start_thread () from /lib64/libpthread.so.0
[2] #17 0x00002acd2aa4f8fd in clone () from /lib64/libc.so.6
[2] Thread 4 (Thread 0x2acd315fb700 (LWP 24417)):
[2] #0  0x00002acd283d65bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[2] #1  0x00000000009af246 in IncomingMessageManager::get_messages(int&, bool) ()
[2] #2  0x00000000009af3cf in Realm::HandlerThread::thread_main() ()
[2] #3  0x0000000000a07d6f in Realm::PreemptableThread::thread_entry(void*) ()
[2] #4  0x00002acd283d29d1 in start_thread () from /lib64/libpthread.so.0
[2] #5  0x00002acd2aa4f8fd in clone () from /lib64/libc.so.6
[2] Thread 3 (Thread 0x2acd317fc700 (LWP 24418)):
[2] #0  0x00002acd283d65bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[2] #1  0x00000000009af246 in IncomingMessageManager::get_messages(int&, bool) ()
[2] #2  0x00000000009af3cf in Realm::HandlerThread::thread_main() ()
[2] #3  0x0000000000a07d6f in Realm::PreemptableThread::thread_entry(void*) ()
[2] #4  0x00002acd283d29d1 in start_thread () from /lib64/libpthread.so.0
[2] #5  0x00002acd2aa4f8fd in clone () from /lib64/libc.so.6
[2] Thread 2 (Thread 0x2acd319fd700 (LWP 24419)):
[2] #0  0x00002acd283d65bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[2] #1  0x00000000009af246 in IncomingMessageManager::get_messages(int&, bool) ()
[2] #2  0x00000000009af3cf in Realm::HandlerThread::thread_main() ()
[2] #3  0x0000000000a07d6f in Realm::PreemptableThread::thread_entry(void*) ()
[2] #4  0x00002acd283d29d1 in start_thread () from /lib64/libpthread.so.0
[2] #5  0x00002acd2aa4f8fd in clone () from /lib64/libc.so.6
[2] Thread 1 (Thread 0x2acd2f4361e0 (LWP 24412)):
[2] #0  0x00000000009b02fd in do_some_polling() ()
[2] #1  0x0000000000a36de9 in Realm::NodeAnnounceMessage::await_all_announcements() ()
[2] #2  0x0000000000a3e34b in Realm::RuntimeImpl::init(int*, char***) ()
[2] #3  0x0000000000a3b5ae in Realm::Runtime::init(int*, char***) ()
[2] #4  0x0000000000c72b8c in LegionRuntime::HighLevel::Runtime::start(int, char**, bool) ()
[2] #5  0x0000000000a5f507 in LegionRuntime::HighLevel::HighLevelRuntime::start(int, char**, bool) ()
[2] #6  0x000000000097ae6b in main ()
[1] /usr/bin/gstack 5359
[1] Thread 7 (Thread 0x2b1d7a3bf700 (LWP 5360)):
[1] #0  0x00002b1d755a2373 in select () from /lib64/libc.so.6
[1] #1  0x00002b1d7363665f in service_thread_start () from /usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[1] #2  0x00002b1d72f2c9d1 in start_thread () from /lib64/libpthread.so.0
[1] #3  0x00002b1d755a98fd in clone () from /lib64/libc.so.6
[1] Thread 6 (Thread 0x2b1d7abf2700 (LWP 5361)):
[1] #0  0x00002b1d755a00d3 in poll () from /lib64/libc.so.6
[1] #1  0x00002b1d73634ef0 in btl_openib_async_thread () from /usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[1] #2  0x00002b1d72f2c9d1 in start_thread () from /lib64/libpthread.so.0
[1] #3  0x00002b1d755a98fd in clone () from /lib64/libc.so.6
[1] Thread 5 (Thread 0x2b1d7bf51700 (LWP 5362)):
[1] #0  0x00002b1d75590287 in sched_yield () from /lib64/libc.so.6
[1] #1  0x0000000000d31e7e in gasneti_bt_gstack ()
[1] #2  0x0000000000d32fbf in gasneti_print_backtrace ()
[1] #3  0x0000000000d96a0e in gasneti_defaultSignalHandler ()
[1] #4
[1] #5  0x00002b1d754f3625 in raise () from /lib64/libc.so.6
[1] #6  0x00002b1d754f4e05 in abort () from /lib64/libc.so.6
[1] #7  0x00002b1d754ec74e in __assert_fail_base () from /lib64/libc.so.6
[1] #8  0x00002b1d754ec810 in __assert_fail () from /lib64/libc.so.6
[1] #9  0x00000000009af384 in enqueue_incoming(unsigned short, IncomingMessage*) ()
[1] #10 0x0000000000a41b35 in MessageRawArgs<Realm::NodeAnnounceMessage::RequestArgs, 141, &(void dummy_short_handlerRealm::NodeAnnounceMessage::RequestArgs(Realm::NodeAnnounceMessage::RequestArgs)), &(Realm::NodeAnnounceMessage::handle_request(Realm::NodeAnnounceMessage::RequestArgs, void const*, unsigned long)), 8>::handler_medium(void*, void*, unsigned long, int, int, int, int, int, int, int, int) ()
[1] #11 0x0000000000d16898 in gasnetc_processPacket ()
[1] #12 0x0000000000d1bb51 in gasnetc_rcv_reap ()
[1] #13 0x0000000000d1c0c9 in gasnetc_AMPoll ()
[1] #14 0x00000000009b02fd in do_some_polling() ()
[1] #15 0x00000000009b03f5 in gasnet_poll_thread_loop(void*) ()
[1] #16 0x00002b1d72f2c9d1 in start_thread () from /lib64/libpthread.so.0
[1] #17 0x00002b1d755a98fd in clone () from /lib64/libc.so.6
[1] Thread 4 (Thread 0x2b1d7c155700 (LWP 5364)):
[1] #0  0x00002b1d72f305bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[1] #1  0x00000000009af246 in IncomingMessageManager::get_messages(int&, bool) ()
[1] #2  0x00000000009af3cf in Realm::HandlerThread::thread_main() ()
[1] #3  0x0000000000a07d6f in Realm::PreemptableThread::thread_entry(void*) ()
[1] #4  0x00002b1d72f2c9d1 in start_thread () from /lib64/libpthread.so.0
[1] #5  0x00002b1d755a98fd in clone () from /lib64/libc.so.6
[1] Thread 3 (Thread 0x2b1d7c356700 (LWP 5365)):
[1] #0  0x00002b1d72f305bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[1] #1  0x00000000009af246 in IncomingMessageManager::get_messages(int&, bool) ()
[1] #2  0x00000000009af3cf in Realm::HandlerThread::thread_main() ()
[1] #3  0x0000000000a07d6f in Realm::PreemptableThread::thread_entry(void*) ()
[1] #4  0x00002b1d72f2c9d1 in start_thread () from /lib64/libpthread.so.0
[1] #5  0x00002b1d755a98fd in clone () from /lib64/libc.so.6
[1] Thread 2 (Thread 0x2b1d7c557700 (LWP 5366)):
[1] #0  0x00002b1d72f305bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
[1] #1  0x00000000009af246 in IncomingMessageManager::get_messages(int&, bool) ()
[1] #2  0x00000000009af3cf in Realm::HandlerThread::thread_main() ()
[1] #3  0x0000000000a07d6f in Realm::PreemptableThread::thread_entry(void*) ()
[1] #4  0x00002b1d72f2c9d1 in start_thread () from /lib64/libpthread.so.0
[1] #5  0x00002b1d755a98fd in clone () from /lib64/libc.so.6
[1] Thread 1 (Thread 0x2b1d79f901e0 (LWP 5359)):
[1] #0  0x00002b1d72f2e565 in pthread_mutex_trylock () from /lib64/libpthread.so.0
[1] #1  0x0000000000d0c419 in gasnetc_hsl_trylock ()
[1] #2  0x00000000009b6438 in EndpointManager::push_messages(int, bool) ()
[1] #3  0x00000000009b02f8 in do_some_polling() ()
[1] #4  0x0000000000a36de9 in Realm::NodeAnnounceMessage::await_all_announcements() ()
[1] #5  0x0000000000a3e34b in Realm::RuntimeImpl::init(int*, char***) ()
[1] #6  0x0000000000a3b5ae in Realm::Runtime::init(int*, char***) ()
[1] #7  0x0000000000c72b8c in LegionRuntime::HighLevel::Runtime::start(int, char**, bool) ()
[1] #8  0x0000000000a5f507 in LegionRuntime::HighLevel::HighLevelRuntime::start(int, char**, bool) ()
[1] #9  0x000000000097ae6b in main ()
*** Caught a signal: SIGTERM(15) on node 0/4
mpirun noticed that process rank 2 with PID 24412 on node mpa155 exited on signal 6 (Aborted).

It seems you and Noah had reported this bug before (
https://github.com/StanfordLegion/legion/issues/36). And based on Sean's
opinion, the cause could be a race condition. Is it an always reproducible
bug?
Sincerely,
Zhihao
On Tue, Sep 1, 2015 at 1:04 PM, gshipman<EMAIL_ADDRESS>wrote:

Different failure:
Reported: 4 (out of 4) daemons - 4 (out of 4) procs
Pinned Memory Usage: GASNET=256, RMEM=1024, LMB=8, SDP=64, total=1352
tester_io: /users/gshipman/local/src/legion-2/runtime//activemsg.cc:717:
void enqueue_incoming(gasnet_node_t, IncomingMessage
): Assertion incoming_message_manager != 0' failed. *** Caught a fatal signal: SIGABRT(6) on node 2/4 tester_io: /users/gshipman/local/src/legion-2/runtime//activemsg.cc:717: void enqueue_incoming(gasnet_node_t, IncomingMessage *): Assertion incoming_message_manager != 0' failed. *** Caught a fatal
signal: SIGABRT(6) on node 3/4
tester_io: /users/gshipman/local/src/legion-2/runtime//activemsg.cc:717:
void enqueue_incoming(gasnet_node_t, IncomingMessage
): Assertion `incoming_message_manager != 0' failed. *** Caught a fatal
signal: SIGABRT(6) on node 1/4
[3] /usr/bin/gstack 30952
[3] Thread 7 (Thread 0x2b9416c00700 (LWP 30953)):
[3] #0 0x00002b9411de3373 in select () from /lib64/libc.so.6
[3] #1 https://github.com/StanfordLegion/legion/issues/1
0x00002b940fe7765f in service_thread_start () from
/usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[3] #2 https://github.com/StanfordLegion/legion/pull/2
0x00002b940f76d9d1 in start_thread () from /lib64/libpthread.so.0
[3] #3 https://github.com/StanfordLegion/legion/pull/3
0x00002b9411dea8fd in clone () from /lib64/libc.so.6
[3] Thread 6 (Thread 0x2b9417433700 (LWP 30954)):
[3] #0 0x00002b9411de10d3 in poll () from /lib64/libc.so.6
[3] #1 https://github.com/StanfordLegion/legion/issues/1
0x00002b940fe75ef0 in btl_openib_async_thread () from
/usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[3] #2 https://github.com/StanfordLegion/legion/pull/2
0x00002b940f76d9d1 in start_thread () from /lib64/libpthread.so.0
[3] #3 https://github.com/StanfordLegion/legion/pull/3
0x00002b9411dea8fd in clone () from /lib64/libc.so.6
[3] Thread 5 (Thread 0x2b9418792700 (LWP 30955)):
[3] #0 0x00002b9411dd1287 in sched_yield () from /lib64/libc.so.6
[3] #1 https://github.com/StanfordLegion/legion/issues/1
0x0000000000d31e7e in gasneti_bt_gstack ()
[3] #2 https://github.com/StanfordLegion/legion/pull/2
0x0000000000d32fbf in gasneti_print_backtrace ()
[3] #3 https://github.com/StanfordLegion/legion/pull/3
0x0000000000d96a0e in gasneti_defaultSignalHandler ()
[3] #4 https://github.com/StanfordLegion/legion/issues/4
[3] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002b9411d34625 in raise () from /lib64/libc.so.6
[3] #6 https://github.com/StanfordLegion/legion/issues/6
0x00002b9411d35e05 in abort () from /lib64/libc.so.6
[3] #7 https://github.com/StanfordLegion/legion/pull/7
0x00002b9411d2d74e in __assert_fail_base () from /lib64/libc.so.6
[3] #8 https://github.com/StanfordLegion/legion/pull/8
0x00002b9411d2d810 in __assert_fail () from /lib64/libc.so.6
[3] #9 https://github.com/StanfordLegion/legion/issues/9
0x00000000009af384 in enqueue_incoming(unsigned short, IncomingMessage
) () [3] #10 https://github.com/StanfordLegion/legion/pull/10
0x0000000000a41b35 in
MessageRawArgs(Realm::NodeAnnounceMessage::RequestArgs)),
&(Realm::NodeAnnounceMessage::handle_request(Realm::NodeAnnounceMessage::RequestArgs,
void const, unsigned long)), 8>::handler_medium(void, void, unsigned
long, int, int, int, int, int, int, int, int) ()
[3] #11 https://github.com/StanfordLegion/legion/pull/11
0x0000000000d16898 in gasnetc_processPacket ()
[3] #12 https://github.com/StanfordLegion/legion/pull/12
0x0000000000d1bb51 in gasnetc_rcv_reap ()
[3] #13 https://github.com/StanfordLegion/legion/pull/13
0x0000000000d1c0c9 in gasnetc_AMPoll ()
[3] #14 https://github.com/StanfordLegion/legion/pull/14
0x00000000009b02fd in do_some_polling() ()
[3] #15 https://github.com/StanfordLegion/legion/pull/15
0x00000000009b03f5 in gasnet_poll_thread_loop(void
) () [3] #16 https://github.com/StanfordLegion/legion/issues/16
0x00002b940f76d9d1 in start_thread () from /lib64/libpthread.so.0 [3] #17
https://github.com/StanfordLegion/legion/issues/17 0x00002b9411dea8fd in
clone () from /lib64/libc.so.6 [3] Thread 4 (Thread 0x2b9418996700 (LWP
30957)): [3] #0 0x00002b940f7715bc in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0 [3] #1
https://github.com/StanfordLegion/legion/issues/1 0x00000000009af246 in
IncomingMessageManager::get_messages(int&, bool) () [3] #2
https://github.com/StanfordLegion/legion/pull/2 0x00000000009af3cf in
Realm::HandlerThread::thread_main() () [3] #3
https://github.com/StanfordLegion/legion/pull/3 0x0000000000a07d6f in
Realm::PreemptableThread::thread_entry(void) ()
[3] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002b940f76d9d1 in start_thread () from /lib64/libpthread.so.0
[3] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002b9411dea8fd in clone () from /lib64/libc.so.6
[3] Thread 3 (Thread 0x2b9418b97700 (LWP 30958)):
[3] #0 0x00002b940f7715bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[3] #1 https://github.com/StanfordLegion/legion/issues/1
0x00000000009af246 in IncomingMessageManager::get_messages(int&, bool) ()
[3] #2 https://github.com/StanfordLegion/legion/pull/2
0x00000000009af3cf in Realm::HandlerThread::thread_main() ()
[3] #3 https://github.com/StanfordLegion/legion/pull/3
0x0000000000a07d6f in Realm::PreemptableThread::thread_entry(void
) () [3] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002b940f76d9d1 in start_thread () from /lib64/libpthread.so.0 [3] #5
https://github.com/StanfordLegion/legion/pull/5 0x00002b9411dea8fd in
clone () from /lib64/libc.so.6 [3] Thread 2 (Thread 0x2b9418d98700 (LWP
30959)): [3] #0 0x00002b940f7715bc in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0 [3] #1
https://github.com/StanfordLegion/legion/issues/1 0x00000000009af246 in
IncomingMessageManager::get_messages(int&, bool) () [3] #2
https://github.com/StanfordLegion/legion/pull/2 0x00000000009af3cf in
Realm::HandlerThread::thread_main() () [3] #3
https://github.com/StanfordLegion/legion/pull/3 0x0000000000a07d6f in
Realm::PreemptableThread::thread_entry(void) ()
[3] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002b940f76d9d1 in start_thread () from /lib64/libpthread.so.0
[3] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002b9411dea8fd in clone () from /lib64/libc.so.6
[3] Thread 1 (Thread 0x2b94167d11e0 (LWP 30952)):
[3] #0 0x0000000000d34840 in gasneti_AMPSHMPoll ()
[3] #1 https://github.com/StanfordLegion/legion/issues/1
0x0000000000d1c117 in gasnetc_AMPoll ()
[3] #2 https://github.com/StanfordLegion/legion/pull/2
0x00000000009b02fd in do_some_polling() ()
[3] #3 https://github.com/StanfordLegion/legion/pull/3
0x0000000000a36de9 in Realm::NodeAnnounceMessage::await_all_announcements()
()
[3] #4 https://github.com/StanfordLegion/legion/issues/4
0x0000000000a3e34b in Realm::RuntimeImpl::init(int*, char*
) () [3] #5 https://github.com/StanfordLegion/legion/pull/5
0x0000000000a3b5ae in Realm::Runtime::init(int, char*
) () [3] #6 https://github.com/StanfordLegion/legion/issues/6
0x0000000000c72b8c in LegionRuntime::HighLevel::Runtime::start(int, char
, bool) () [3] #7 https://github.com/StanfordLegion/legion/pull/7
0x0000000000a5f507 in
LegionRuntime::HighLevel::HighLevelRuntime::start(int, char
, bool) () [3] #8 https://github.com/StanfordLegion/legion/pull/8
0x000000000097ae6b in main () [2] /usr/bin/gstack 24412 [2] Thread 7
(Thread 0x2acd2f865700 (LWP 24413)): [2] #0 0x00002acd2aa48373 in select ()
from /lib64/libc.so.6 [2] #1
https://github.com/StanfordLegion/legion/issues/1 0x00002acd28adc65f in
service_thread_start () from
/usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[2] #2 https://github.com/StanfordLegion/legion/pull/2 0x00002acd283d29d1
in start_thread () from /lib64/libpthread.so.0 [2] #3
https://github.com/StanfordLegion/legion/pull/3 0x00002acd2aa4f8fd in
clone () from /lib64/libc.so.6 [2] Thread 6 (Thread 0x2acd30098700 (LWP
24414)): [2] #0 0x00002acd2aa460d3 in poll () from /lib64/libc.so.6 [2] #1
https://github.com/StanfordLegion/legion/issues/1 0x00002acd28adaef0 in
btl_openib_async_thread () from
/usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[2] #2 https://github.com/StanfordLegion/legion/pull/2 0x00002acd283d29d1
in start_thread () from /lib64/libpthread.so.0 [2] #3
https://github.com/StanfordLegion/legion/pull/3 0x00002acd2aa4f8fd in
clone () from /lib64/libc.so.6 [2] Thread 5 (Thread 0x2acd313f7700 (LWP
24415)): [2] #0 0x00002acd2aa36287 in sched_yield () from /lib64/libc.so.6
[2] #1 https://github.com/StanfordLegion/legion/issues/1
0x0000000000d31e7e in gasneti_bt_gstack () [2] #2
https://github.com/StanfordLegion/legion/pull/2 0x0000000000d32fbf in
gasneti_print_backtrace () [2] #3
https://github.com/StanfordLegion/legion/pull/3 0x0000000000d96a0e in
gasneti_defaultSignalHandler () [2] #4
https://github.com/StanfordLegion/legion/issues/4 [2] #5
https://github.com/StanfordLegion/legion/pull/5 0x00002acd2a999625 in
raise () from /lib64/libc.so.6 [2] #6
https://github.com/StanfordLegion/legion/issues/6 0x00002acd2a99ae05 in
abort () from /lib64/libc.so.6 [2] #7
https://github.com/StanfordLegion/legion/pull/7 0x00002acd2a99274e in
__assert_fail_base () from /lib64/libc.so.6 [2] #8
https://github.com/StanfordLegion/legion/pull/8 0x00002acd2a992810 in
__assert_fail () from /lib64/libc.so.6 [2] #9
https://github.com/StanfordLegion/legion/issues/9 0x00000000009af384 in
enqueue_incoming(unsigned short, IncomingMessage) ()
[2] #10 https://github.com/StanfordLegion/legion/pull/10
0x0000000000a41b35 in
MessageRawArgs(Realm::NodeAnnounceMessage::RequestArgs)),
&(Realm::NodeAnnounceMessage::handle_request(Realm::NodeAnnounceMessage::RequestArgs,
void const*, unsigned long)), 8>::handler_medium(void*, void
, unsigned long, int, int, int, int, int, int, int, int) () [2] #11
https://github.com/StanfordLegion/legion/pull/11 0x0000000000d16898 in
gasnetc_processPacket () [2] #12
https://github.com/StanfordLegion/legion/pull/12 0x0000000000d1bb51 in
gasnetc_rcv_reap () [2] #13
https://github.com/StanfordLegion/legion/pull/13 0x0000000000d1c0c9 in
gasnetc_AMPoll () [2] #14
https://github.com/StanfordLegion/legion/pull/14 0x00000000009b02fd in
do_some_polling() () [2] #15
https://github.com/StanfordLegion/legion/pull/15 0x00000000009b03f5 in
gasnet_poll_thread_loop(void) ()
[2] #16 https://github.com/StanfordLegion/legion/issues/16
0x00002acd283d29d1 in start_thread () from /lib64/libpthread.so.0
[2] #17 https://github.com/StanfordLegion/legion/issues/17
0x00002acd2aa4f8fd in clone () from /lib64/libc.so.6
[2] Thread 4 (Thread 0x2acd315fb700 (LWP 24417)):
[2] #0 0x00002acd283d65bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[2] #1 https://github.com/StanfordLegion/legion/issues/1
0x00000000009af246 in IncomingMessageManager::get_messages(int&, bool) ()
[2] #2 https://github.com/StanfordLegion/legion/pull/2
0x00000000009af3cf in Realm::HandlerThread::thread_main() ()
[2] #3 https://github.com/StanfordLegion/legion/pull/3
0x0000000000a07d6f in Realm::PreemptableThread::thread_entry(void
) () [2] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002acd283d29d1 in start_thread () from /lib64/libpthread.so.0 [2] #5
https://github.com/StanfordLegion/legion/pull/5 0x00002acd2aa4f8fd in
clone () from /lib64/libc.so.6 [2] Thread 3 (Thread 0x2acd317fc700 (LWP
24418)): [2] #0 0x00002acd283d65bc in pthread_cond_wait@@GLIBC_2.3.2 ()
from /lib64/libpthread.so.0 [2] #1
https://github.com/StanfordLegion/legion/issues/1 0x00000000009af246 in
IncomingMessageManager::get_messages(int&, bool) () [2] #2
https://github.com/StanfordLegion/legion/pull/2 0x00000000009af3cf in
Realm::HandlerThread::thread_main() () [2] #3
https://github.com/StanfordLegion/legion/pull/3 0x0000000000a07d6f in
Realm::PreemptableThread::thread_entry(void) ()
[2] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002acd283d29d1 in start_thread () from /lib64/libpthread.so.0
[2] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002acd2aa4f8fd in clone () from /lib64/libc.so.6
[2] Thread 2 (Thread 0x2acd319fd700 (LWP 24419)):
[2] #0 0x00002acd283d65bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[2] #1 https://github.com/StanfordLegion/legion/issues/1
0x00000000009af246 in IncomingMessageManager::get_messages(int&, bool) ()
[2] #2 https://github.com/StanfordLegion/legion/pull/2
0x00000000009af3cf in Realm::HandlerThread::thread_main() ()
[2] #3 https://github.com/StanfordLegion/legion/pull/3
0x0000000000a07d6f in Realm::PreemptableThread::thread_entry(void
) () [2] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002acd283d29d1 in start_thread () from /lib64/libpthread.so.0 [2] #5
https://github.com/StanfordLegion/legion/pull/5 0x00002acd2aa4f8fd in
clone () from /lib64/libc.so.6 [2] Thread 1 (Thread 0x2acd2f4361e0 (LWP
24412)): [2] #0 0x00000000009b02fd in do_some_polling() () [2] #1
https://github.com/StanfordLegion/legion/issues/1 0x0000000000a36de9 in
Realm::NodeAnnounceMessage::await_all_announcements() () [2] #2
https://github.com/StanfordLegion/legion/pull/2 0x0000000000a3e34b in
Realm::RuntimeImpl::init(int, char
) () [2] #3 https://github.com/StanfordLegion/legion/pull/3
0x0000000000a3b5ae in Realm::Runtime::init(int, char
) () [2] #4 https://github.com/StanfordLegion/legion/issues/4
0x0000000000c72b8c in LegionRuntime::HighLevel::Runtime::start(int, char
, bool) () [2] #5 https://github.com/StanfordLegion/legion/pull/5
0x0000000000a5f507 in
LegionRuntime::HighLevel::HighLevelRuntime::start(int, char
, bool) () [2] #6 https://github.com/StanfordLegion/legion/issues/6
0x000000000097ae6b in main () [1] /usr/bin/gstack 5359 [1] Thread 7 (Thread
0x2b1d7a3bf700 (LWP 5360)): [1] #0 0x00002b1d755a2373 in select () from
/lib64/libc.so.6 [1] #1 https://github.com/StanfordLegion/legion/issues/1
0x00002b1d7363665f in service_thread_start () from
/usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[1] #2 https://github.com/StanfordLegion/legion/pull/2 0x00002b1d72f2c9d1
in start_thread () from /lib64/libpthread.so.0 [1] #3
https://github.com/StanfordLegion/legion/pull/3 0x00002b1d755a98fd in
clone () from /lib64/libc.so.6 [1] Thread 6 (Thread 0x2b1d7abf2700 (LWP
5361)): [1] #0 0x00002b1d755a00d3 in poll () from /lib64/libc.so.6 [1] #1
https://github.com/StanfordLegion/legion/issues/1 0x00002b1d73634ef0 in
btl_openib_async_thread () from
/usr/projects/hpcsoft/toss2/mapache/openmpi/1.6.5-gcc-4.4/lib/libmpi.so.1
[1] #2 https://github.com/StanfordLegion/legion/pull/2 0x00002b1d72f2c9d1
in start_thread () from /lib64/libpthread.so.0 [1] #3
https://github.com/StanfordLegion/legion/pull/3 0x00002b1d755a98fd in
clone () from /lib64/libc.so.6 [1] Thread 5 (Thread 0x2b1d7bf51700 (LWP
5362)): [1] #0 0x00002b1d75590287 in sched_yield () from /lib64/libc.so.6
[1] #1 https://github.com/StanfordLegion/legion/issues/1
0x0000000000d31e7e in gasneti_bt_gstack () [1] #2
https://github.com/StanfordLegion/legion/pull/2 0x0000000000d32fbf in
gasneti_print_backtrace () [1] #3
https://github.com/StanfordLegion/legion/pull/3 0x0000000000d96a0e in
gasneti_defaultSignalHandler () [1] #4
https://github.com/StanfordLegion/legion/issues/4 [1] #5
https://github.com/StanfordLegion/legion/pull/5 0x00002b1d754f3625 in
raise () from /lib64/libc.so.6 [1] #6
https://github.com/StanfordLegion/legion/issues/6 0x00002b1d754f4e05 in
abort () from /lib64/libc.so.6 [1] #7
https://github.com/StanfordLegion/legion/pull/7 0x00002b1d754ec74e in
__assert_fail_base () from /lib64/libc.so.6 [1] #8
https://github.com/StanfordLegion/legion/pull/8 0x00002b1d754ec810 in
__assert_fail () from /lib64/libc.so.6 [1] #9
https://github.com/StanfordLegion/legion/issues/9 0x00000000009af384 in
enqueue_incoming(unsigned short, IncomingMessage) ()
[1] #10 https://github.com/StanfordLegion/legion/pull/10
0x0000000000a41b35 in
MessageRawArgs(Realm::NodeAnnounceMessage::RequestArgs)),
&(Realm::NodeAnnounceMessage::handle_request(Realm::NodeAnnounceMessage::RequestArgs,
void const*, unsigned long)), 8>::handler_medium(void*, void
, unsigned long, int, int, int, int, int, int, int, int) () [1] #11
https://github.com/StanfordLegion/legion/pull/11 0x0000000000d16898 in
gasnetc_processPacket () [1] #12
https://github.com/StanfordLegion/legion/pull/12 0x0000000000d1bb51 in
gasnetc_rcv_reap () [1] #13
https://github.com/StanfordLegion/legion/pull/13 0x0000000000d1c0c9 in
gasnetc_AMPoll () [1] #14
https://github.com/StanfordLegion/legion/pull/14 0x00000000009b02fd in
do_some_polling() () [1] #15
https://github.com/StanfordLegion/legion/pull/15 0x00000000009b03f5 in
gasnet_poll_thread_loop(void) ()
[1] #16 https://github.com/StanfordLegion/legion/issues/16
0x00002b1d72f2c9d1 in start_thread () from /lib64/libpthread.so.0
[1] #17 https://github.com/StanfordLegion/legion/issues/17
0x00002b1d755a98fd in clone () from /lib64/libc.so.6
[1] Thread 4 (Thread 0x2b1d7c155700 (LWP 5364)):
[1] #0 0x00002b1d72f305bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[1] #1 https://github.com/StanfordLegion/legion/issues/1
0x00000000009af246 in IncomingMessageManager::get_messages(int&, bool) ()
[1] #2 https://github.com/StanfordLegion/legion/pull/2
0x00000000009af3cf in Realm::HandlerThread::thread_main() ()
[1] #3 https://github.com/StanfordLegion/legion/pull/3
0x0000000000a07d6f in Realm::PreemptableThread::thread_entry(void
) () [1] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002b1d72f2c9d1 in start_thread () from /lib64/libpthread.so.0 [1] #5
https://github.com/StanfordLegion/legion/pull/5 0x00002b1d755a98fd in
clone () from /lib64/libc.so.6 [1] Thread 3 (Thread 0x2b1d7c356700 (LWP
5365)): [1] #0 0x00002b1d72f305bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0 [1] #1
https://github.com/StanfordLegion/legion/issues/1 0x00000000009af246 in
IncomingMessageManager::get_messages(int&, bool) () [1] #2
https://github.com/StanfordLegion/legion/pull/2 0x00000000009af3cf in
Realm::HandlerThread::thread_main() () [1] #3
https://github.com/StanfordLegion/legion/pull/3 0x0000000000a07d6f in
Realm::PreemptableThread::thread_entry(void) ()
[1] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002b1d72f2c9d1 in start_thread () from /lib64/libpthread.so.0
[1] #5 https://github.com/StanfordLegion/legion/pull/5
0x00002b1d755a98fd in clone () from /lib64/libc.so.6
[1] Thread 2 (Thread 0x2b1d7c557700 (LWP 5366)):
[1] #0 0x00002b1d72f305bc in pthread_cond_wait@@GLIBC_2.3.2 () from
/lib64/libpthread.so.0
[1] #1 https://github.com/StanfordLegion/legion/issues/1
0x00000000009af246 in IncomingMessageManager::get_messages(int&, bool) ()
[1] #2 https://github.com/StanfordLegion/legion/pull/2
0x00000000009af3cf in Realm::HandlerThread::thread_main() ()
[1] #3 https://github.com/StanfordLegion/legion/pull/3
0x0000000000a07d6f in Realm::PreemptableThread::thread_entry(void
) () [1] #4 https://github.com/StanfordLegion/legion/issues/4
0x00002b1d72f2c9d1 in start_thread () from /lib64/libpthread.so.0 [1] #5
https://github.com/StanfordLegion/legion/pull/5 0x00002b1d755a98fd in
clone () from /lib64/libc.so.6 [1] Thread 1 (Thread 0x2b1d79f901e0 (LWP
5359)): [1] #0 0x00002b1d72f2e565 in pthread_mutex_trylock () from
/lib64/libpthread.so.0 [1] #1
https://github.com/StanfordLegion/legion/issues/1 0x0000000000d0c419 in
gasnetc_hsl_trylock () [1] #2
https://github.com/StanfordLegion/legion/pull/2 0x00000000009b6438 in
EndpointManager::push_messages(int, bool) () [1] #3
https://github.com/StanfordLegion/legion/pull/3 0x00000000009b02f8 in
do_some_polling() () [1] #4
https://github.com/StanfordLegion/legion/issues/4 0x0000000000a36de9 in
Realm::NodeAnnounceMessage::await_all_announcements() () [1] #5
https://github.com/StanfordLegion/legion/pull/5 0x0000000000a3e34b in
Realm::RuntimeImpl::init(int, char
) () [1] #6 https://github.com/StanfordLegion/legion/issues/6
0x0000000000a3b5ae in Realm::Runtime::init(int, char
) () [1] #7 https://github.com/StanfordLegion/legion/pull/7
0x0000000000c72b8c in LegionRuntime::HighLevel::Runtime::start(int, char
, bool) () [1] #8 https://github.com/StanfordLegion/legion/pull/8
0x0000000000a5f507 in
LegionRuntime::HighLevel::HighLevelRuntime::start(int, char*, bool) ()
[1] #9 https://github.com/StanfordLegion/legion/issues/9
0x000000000097ae6b in main ()
*** Caught a signal: SIGTERM(15) on node 0/4 mpirun noticed that process
rank 2 with PID 24412 on node mpa155 exited on signal 6 (Aborted).
—
Reply to this email directly or view it on GitHub
https://github.com/StanfordLegion/legion/issues/51#issuecomment-136845009
.

I have fixed two different issues that might have been causing this.  Zhihao, can you retest this
once you get the merge of master->dma sorted?

No updates in a month.  This bug will be closed tomorrow unless somebody offers a reason not to.