Deadlock between PJSUA LOCK, transaction group lock, and UA mutex

[ming]

Deadlock when the following three events occur simultaneously:

1. Hangup call using pjsua_call_hangup().
2. The same call is also receiving a CANCEL message.
3. An INVITE is being received for a new separate call.

Thread 1 holds transaction lock in pjsip_tsx_send_msg() then attempts to obtain PJSUA lock.

#4  PJSUA_LOCK () at ../include/pjsua-lib/pjsua_internal.h:575              /* blocked on PJSUA_LOCK() */
#5  pjsua_call_on_state_changed () at ../src/pjsua-lib/pjsua_call.c:3807
#6  inv_set_state () at ../src/pjsip-ua/sip_inv.c:317
#7  inv_on_state_incoming () at ../src/pjsip-ua/sip_inv.c:4310
#8  mod_inv_on_tsx_state () at ../src/pjsip-ua/sip_inv.c:717
#9  pjsip_dlg_on_tsx_state () at ../src/pjsip/sip_dialog.c:2064
#10 mod_ua_on_tsx_state () at ../src/pjsip/sip_ua_layer.c:178
#11 tsx_set_state () at ../src/pjsip/sip_transaction.c:1235
#12 tsx_on_state_proceeding_uas () at ../src/pjsip/sip_transaction.c:2819
#13 pjsip_tsx_send_msg () at ../src/pjsip/sip_transaction.c:1751            /* obtained pj_grp_lock_acquire(tsx->grp_lock) */
#14 pjsip_dlg_send_response () at ../src/pjsip/sip_dialog.c:1529
#15 pjsip_inv_send_msg () at ../src/pjsip-ua/sip_inv.c:3227
#16 pjsua_call_hangup (code=606) at ../src/pjsua-lib/pjsua_call.c:2407

Thread 2 holds mod_ua.mutex and attempts to obtain transaction lock

#6  pj_grp_lock_acquire () at ../src/pj/lock.c:478
#7  pjsip_tsx_layer_find_tsx () at ../src/pjsip/sip_transaction.c:672    /* blocked on pj_grp_lock_acquire(tsx->grp_lock) */
#8  find_dlg_set_for_msg () at ../src/pjsip/sip_ua_layer.c:554
#9  mod_ua_on_rx_request () at ../src/pjsip/sip_ua_layer.c:615           /* obtains pj_mutex_lock(mod_ua.mutex) */
#10 pjsip_endpt_process_rx_data () at ../src/pjsip/sip_endpoint.c:887

Thread 3 holds PJSUA lock and attempts to obtain mod_ua.mutex

#4  pjsip_ua_register_dlg () at ../src/pjsip/sip_ua_layer.c:298            /* blocked on pj_mutex_lock(mod_ua.mutex) */
#5  create_uas_dialog () at ../src/pjsip/sip_dialog.c:533
#6  pjsip_dlg_create_uas_and_inc_lock () at ../src/pjsip/sip_dialog.c:590
#7  pjsua_call_on_incoming () at ../src/pjsua-lib/pjsua_call.c:1386        /* obtained PJSUA_LOCK() */
#8  mod_pjsua_on_rx_request () at ../src/pjsua-lib/pjsua_core.c:577
#9  pjsip_endpt_process_rx_data () at ../src/pjsip/sip_endpoint.c:887

This may be related to ticket #1464.

Thank you to David Richards for the report and the stack trace.

[ming]

In 5573:

Fixed #2002: Deadlock between PJSUA LOCK, transaction group lock, and UA mutex