Deadlock in transaction code when hanging up calls (thanks Dennis Struble for the report)

[bennylp]

Deadlock may occur when application is hanging up a ringing call while final response is received at the same time. Detailed scenario:

Thread 1:

• the app is hanging up the call
• dialog mutex acquired in acquire_call() in pjsua_call.c
• attempting to acquire transaction mutex in call to pjsip_tsx_set_timeout() function from pjsip_inv_end_session()

Thread 2:

• receives failure final response to INVITE request
• transaction mutex acquired
• attempting to acquire dialog mutex

[bennylp]

(In [3311]) Fix #1121 (Deadlock in transaction code when hanging up calls (thanks Dennis Struble for the report))

[bennylp]

Reopened:

the fix may cause race condition, which leads to this assertion in pjsip_tsx_set_timeout():

    PJ_ASSERT_ON_FAIL(tsx->status_code < 200,
                  { unlock_timer(tsx); return PJ_EINVALIDOP; });

The race condition is: although the status code has been checked by the invite session before calling pjsip_tsx_set_timeout(), final response may have arrived after that and before the assert above, leading to the assertion. This is because the invite session doesn't hold the transaction lock for that brief period of time.

Thanks Dennis Struble for the report.

[bennylp]

(In [3328]) Closed #1121 (Deadlock in transaction code when hanging up calls): fixed the race condition when setting up transaction timeout