Deadlock between dialog lock and transaction group lock

[ming]

Sample scenario:

• PJSIP receives a re-INVITE and send 200 OK.
• The ACK is delayed and received at the same time the timeout timer fires.

Thread 1 receives the delayed ACK:

- pjsip_dlg_on_rx_request() -> pjsip_dlg_inc_lock(dlg)
- pjsip_tsx_terminate() -> pj_grp_lock_acquire(tsx->grp_lock)

Thread 2 handles the timeout:

- tsx_timer_callback() -> pj_grp_lock_acquire(tsx->grp_lock) then (*tsx->state_handler)(tsx, &event);
- tsx_on_state_completed_uas()
- tsx_set_state() -> (*tsx->tsx_user->on_tsx_state)(tsx, &e),
- pjsip_dlg_on_tsx_state() -> pjsip_dlg_inc_lock(dlg);

Thread 1's stack trace:

#5  0x00007f4748aac5a2 in pj_grp_lock_acquire (grp_lock=<optimized out>) at ../src/pj/lock.c:478
#6  0x00007f4749be20cd in pjsip_tsx_terminate (tsx=0x7f466ee29bb8, code=408) at ../src/pjsip/sip_transaction.c:1636
#7  0x00007f474a021dc6 in mod_inv_on_rx_request (rdata=0x7f46c4074c38) at ../src/pjsip-ua/sip_inv.c:586
#8  0x00007f4749be4b01 in pjsip_dlg_on_rx_request (dlg=dlg@entry=0x7f466eacb138, rdata=rdata@entry=0x7f46c4074c38) at ../src/pjsip/sip_dialog.c:1694
#9  0x00007f4749be6147 in mod_ua_on_rx_request (rdata=0x7f46c4074c38) at ../src/pjsip/sip_ua_layer.c:699
#10 0x00007f4749bccf77 in pjsip_endpt_process_rx_data (endpt=<optimized out>, rdata=rdata@entry=0x7f46c4074c38, p=p@entry=0x7f46d0004c40 <param>, p_handled=p_handled@entry=0x7f466155ecf4) at ../src/pjsip/sip_endpoint.c:887

Thread 2's stack trace:

#4  0x00007f4749be3b57 in pjsip_dlg_inc_lock (dlg=0x7f466eacb138) at ../src/pjsip/sip_dialog.c:885
#5  0x00007f4749be4dd6 in pjsip_dlg_on_tsx_state (dlg=0x7f466eacb138, tsx=0x7f466ee29bb8, e=0x7f46cf18dc10) at ../src/pjsip/sip_dialog.c:2047
#7  0x00007f4749be06b0 in tsx_on_state_completed_uas (tsx=0x7f466ee29bb8, event=0x7f46cf18dce0) at ../src/pjsip/sip_transaction.c:3198
#8  0x00007f4749bdfa86 in tsx_timer_callback (theap=<optimized out>, entry=0x7f466ee29d70) at ../src/pjsip/sip_transaction.c:1171
#9  0x00007f4748ab7207 in pj_timer_heap_poll (ht=0x7f47504b8d80, next_delay=next_delay@entry=0x7f46cf18de00) at ../src/pj/timer.c:643

Thanks to Alex Hermann for the report and the stack trace.

[ming]

In 5572:

Fixed #2001: Deadlock between dialog lock and transaction group lock