Opened 8 years ago

Closed 8 years ago

Last modified 8 years ago

#1100 closed bug (fixed)

--enable-threads=multiple results in deadlocks

Reported by: buntinas Owned by: balaji
Priority: major Milestone: mpich2-1.3.1
Component: mpich Keywords:
Cc: robl@…

Description

Configuring with --enable-threads=multiple breaks something in the threading code. E.g., test/mpi/attr/attrt :

Error in system call pthread_mutex_lock: Resource deadlock avoided
    ../../../../mpich2/src/include/mpiimplthreadpost.h:35
Assertion failed in file ../../../../mpich2/src/include/mpiimplthreadpost.h at line 35: err_ == MPIU_THREAD_SUCCESS
mutex_lock failed, err_=35 (Resource deadlock avoided)
internal ABORT - process 0

If we feel that enable-threads=multiple is not useful (since we already have runtime thread checks be default), then we should remove this option.

Dave, I'm assigning this to you since you did the fine-grained stuff, and you might have more insight, but feel free to reassign it.

Change History (25)

comment:1 Changed 8 years ago by balaji

My vote is to get rid of multiple. If we want to retain it, we should at least move the current "multiple" to "force-multiple", and let "multiple" have the same meaning as "runtime". I've seen too many people incorrectly do performance measurements for single threaded programs when MPICH2 is configured with "multiple" (though they really meant "runtime").

comment:2 Changed 8 years ago by thakur

I don't mind making multiple = runtime to avoid that problem.

comment:3 Changed 8 years ago by thakur

  • Resolution set to fixed
  • Status changed from new to closed

Set multiple = runtime in [94a0c255c4be2a4fdf0897ba6125ec5e232e58b4]. We can revert it back later if multiple is really needed.

comment:4 Changed 8 years ago by balaji

  • Resolution fixed deleted
  • Status changed from closed to reopened

As pointed out by Bill off list, only providing --enable-threads=single and --enable-threads=runtime is not sufficient for testing. We have a lot of tests that initialize as MPI_Init(). The --enable-threads=multiple option allows us to test them as if they were initialized with MPI_Init_thread(MULTIPLE). It's a different issue whether we want to call this forced multiple option as "multiple" or "force-multiple".

Reopening this ticket.

comment:5 Changed 8 years ago by balaji

  • Owner changed from goodell to balaji
  • Status changed from reopened to accepted

comment:6 Changed 8 years ago by gropp

You can now duplicate the problems by building with the default (runtime) and setting the following environment variables:

MPICH_THREADLEVEL_DEFAULT=MULTIPLE
MTEST_THREADLEVEL_DEFAULT=MULTIPLE

This causes MPI_Init and MTest_Init to select a threadlevel of THREAD_MULTIPLE. Setting these is enough to cause test/mpi/attr/attrt to (appear to) hang. In my testing, I see apparent hangs in (a partial list; the tests are still running and timing out):

  • attrt
  • attric
  • attrorder
  • baseatt[2]
  • keyval_double_free
  • allredmany
  • op_commutative
  • bcast2
  • bcast3
  • ctxsplit
  • sendflood
  • cartzero
  • cartsuball
  • distgraph1
  • commcreatep
  • rdwrord
  • rdwrzero
  • getextent
  • setinfo
  • setviewcur
  • i_noncontig
  • async
  • async_any
  • userioerr
  • resized
  • iwriteatf
  • iwritef
  • iwriteshf
  • writef
  • writeatf
  • writeallf
  • writeshf
  • writeordf
  • writeatallf
  • writeatallbef
  • writeallbef
  • writeordbef
  • fileerrf
  • fileinfof
  • shpositionf
  • atomicityf
  • miscfilef
  • setviewcurf
  • c2f2ciof
  • attrtx
  • iwriteatx
  • iwritex
  • iwriteshx
  • writex
  • writeatx
  • writeallx
  • writeshx
  • writeordx
  • writeatallx
  • writeatallbex
  • writeallbex
  • writeordbex
  • iwriteatnosx
  • iwritenosx
  • iwriteshnosx
  • writenosx
  • writeatnosx
  • writeshnosx
  • writeordnosx
  • writeatallnosx
  • writeatallbenosx
  • writeallbenosx

Of course, there are probably just a few bugs that are responsible for all of these.

comment:7 Changed 8 years ago by balaji

[ef07da033782ab375385f5bc9017a1a2bd76c651], [270abc670a030763450e3edfb27c5a82bde6a00e] and [b8c9ab1c1f609cdc120a826115fc7b5de70194bd] fix some of these. I'm still running tests to see what else is hanging or failing.

comment:8 Changed 8 years ago by balaji

  • Milestone changed from mpich2-1.3 to mpich2-1.3.1

Everything other than the I/O tests seem to pass with --enable-threads=multiple. I'm pushing this to 1.3.1.

comment:9 Changed 8 years ago by gropp

I built MPICH2 (current SVN version) with the default (runtime) thread support and then ran the MPICH2 test suite twice - once using MPICH_THREADLEVEL_DEFAULT and MTEST_THREADLEVEL_DEFAULT to force the thread level to MULTIPLE, and once without. Many tests still failed (usually timing out) when MULTIPLE was selected. Leaving out the I/O tests that failed, the failing tests included:

  • allredmany
  • op_commutative
  • bcast2
  • bcast3
  • ctxsplit
  • sendflood
  • commcreatep
  • attrtx
  • errgetx

comment:10 Changed 8 years ago by thakur

I am able to reproduce these failures on my laptop.

comment:11 Changed 8 years ago by buntinas

Aside from op_commutative (that I fixed), I can't reproduce these (at least the non c++ ones). Can you guys rerun this with --enable-g=all? This should tell us if these failures are due to recursive locking.

comment:12 Changed 8 years ago by buntinas

Is it possible that the timeouts in make testing aren't long enough in the (slower) multiple case?

comment:13 Changed 8 years ago by thakur

When you run mpiexec -n 4 allredmany on your laptop, it returns instantly. When I run it on mine (with the same configure options) it runs forever.

comment:14 Changed 8 years ago by thakur

allredmany actually completed, but it took 4 minutes. No idea what is going on.

comment:15 Changed 8 years ago by gropp

I've also run MPICH2 against the Intel test suite and see a similar pattern - when THREAD_MULTIPLE is forced, many tests take much longer (typically timing out). Has anyone checked the performance of MPICH2 with THREAD_MULTIPLE selected?

comment:16 Changed 8 years ago by thakur

There is the thread performance test suite that could be run (at least the latency test).

Have you been running the tests with multiple forced regularly? Do you know since when they started timing out?

There is probably some extra lock acquisition and release happening that shouldn't in the global mode.

comment:17 Changed 8 years ago by gropp

I've found a simple test - if I run mpptest this way

mpiexec -n 4 ./mpptest  -logscale -threadmultiple

the rendezvous mode sized messages take forever:

#p0     p1      dist    len     ave time (us)   rate
...
0       3       3       16384   5.559921        2946.804e+6
0       3       3       32768   9.829998        3333.470e+6
0       3       3       65536   14051.148891    4.664e+6
0       3       3       131072  13697.981834    9.569e+6

while if run as

mpiexec -n 4 ./mpptest  -logscale

the performance is very good. This is on my two-core Macbook Pro, so it is, in part, an oversubscription issue.

#p0     p1      dist    len     ave time (us)   rate
...
0       3       3       16384   6.191730        2646.110e+6
0       3       3       32768   11.501312       2849.066e+6
0       3       3       65536   93.810558       698.599e+6
0       3       3       131072  109.698772      1194.836e+6

It looks like the threadmultiple case is waiting for a context switch, based on the large jump in time. I suspect any simple test that sends rendezvous-sized messages and has more processes than cores will perform poorly on OSX with thread multiple set.

comment:18 Changed 8 years ago by goodell

  • Resolution set to fixed
  • Status changed from accepted to closed

We needed to call sched_yield (or similar) in our "interthread" yield.

Fixed by [a40b1ffa6cbb11567866b55b5b98c850844d6894], [d142f825bed26fc786fe12a3e7874ce84dcbe287], [3938023edec971fa6db5faf9680d1837e6b2210c]. Resolving.

comment:19 Changed 8 years ago by thakur

  • Resolution fixed deleted
  • Status changed from closed to reopened

A large number of I/O tests are still failing with threads=multiple. Do we want to look at these before the release? Otherwise no one will be able to use thread_multiple with I/O.

# rdwrord
# rdwrzero
# getextent
# setinfo
# setviewcur
# i_noncontig
# async
# async_any
# userioerr
# resized
# iwriteatf
# iwritef
# iwriteshf
# writef
# writeatf
# writeallf
# writeshf
# writeordf
# writeatallf
# writeatallbef
# writeallbef
# writeordbef
# fileerrf
# fileinfof
# shpositionf
# atomicityf
# miscfilef
# setviewcurf
# c2f2ciof
# attrtx
# iwriteatx
# iwritex
# iwriteshx
# writex
# writeatx
# writeallx
# writeshx
# writeordx
# writeatallx
# writeatallbex
# writeallbex
# writeordbex
# iwriteatnosx
# iwritenosx
# iwriteshnosx
# writenosx
# writeatnosx
# writeshnosx
# writeordnosx
# writeatallnosx
# writeatallbenosx
# writeallbenosx 

comment:20 Changed 8 years ago by goodell

Are they also timing out? Or do they fail with different problems? Did these work in previous releases? (I suspect not)

There are 130 .c files in src/mpi/romio/mpi-io but only 48 of them have a CS_ENTER/EXIT, so there's a good chance that the thread safety in ROMIO is just broken right now.

comment:21 Changed 8 years ago by thakur

They are timing out.

comment:22 Changed 8 years ago by thakur

  • Cc robl@… added

Many tests in src/mpi/romio/test also hang with --enable-threads=multiple. Even the most simple one, simple.c, hangs. The tests can be run with "runtests" in that directory.

comment:23 Changed 8 years ago by thakur

BTW, the Northwestern guys use I/O with threads for their caching/delegation stuff.

Rob, how important is it that I/O work with threads for the 1.3 release?

comment:24 Changed 8 years ago by goodell

  • Resolution set to fixed
  • Status changed from reopened to closed
Note: See TracTickets for help on using tickets.