Assertion on incoming call after media initialization failure due to failed STUN server

[nanang]

Assertion line:

Assertion failed: dlg->dlg_set, file d:\projects\pjproject\pjsip\src\pjsip\sip_ua_layer.c, line 371

Call stack:

abort()  Line 59 + 0x7 bytes	C
_wassert(const wchar_t * expr=0x007983a8, const wchar_t * filename=0x00798118, unsigned int lineno=371)  Line 212	C
pjsip_ua_unregister_dlg(pjsip_module * ua=0x00804ca8, pjsip_dialog * dlg=0x0260c96c)  Line 371 + 0x2b bytes	C
unregister_and_destroy_dialog(pjsip_dialog * dlg=0x0260c96c, int unlock_mutex=1)  Line 791 + 0x10 bytes	C
pjsip_dlg_dec_lock(pjsip_dialog * dlg=0x0260c96c)  Line 941 + 0xb bytes	C
create_uas_dialog(pjsip_module * ua=0x00804ca8, pjsip_rx_data * rdata=0x0260961c, const pj_str_t * contact=0x0322f590, int inc_lock=1, pjsip_dialog * * p_dlg=0x0322f58c)  Line 563 + 0x9 bytes	C
pjsip_dlg_create_uas_and_inc_lock(pjsip_module * ua=0x00804ca8, pjsip_rx_data * rdata=0x0260961c, const pj_str_t * contact=0x0322f590, pjsip_dialog * * p_dlg=0x0322f58c)  Line 595 + 0x17 bytes	C
pjsua_call_on_incoming(pjsip_rx_data * rdata=0x0260961c)  Line 1376 + 0x17 bytes	C

Reproducing steps using pjsua app:

1. Set stun_ignore_failure to PJ_FALSE (by directly modifying the code as there is no param to disable it).
2. Configure pjsua to use a STUN server.
3. Once pjsua is up, shutdown (connection to) the STUN server.
4. Make call to pjsua using UDP transport, this call will be rejected due to media channel initialization failure.
5. Assertion is raised.

After investigation, we found that the problem is introduced by incoming INVITE retransmission that triggers a new dialog creation after the original INVITE is rejected (due to media initialization failure) and the original dialog is destroyed. However, the UAS transaction creation fails because the original transaction has not been destroyed, so dialog is created but not registered (to pjsip_ua). The new dialog deinitialization takes place and assertion in dialog unregistration is raised because the new dialog actually has not been registered yet.

Thanks Marcus Froeschl for the report.

[nanang]

In 5316:

Fix #1924: Put check whether dialog has been registered to pjsip UA before calling pjsip_ua_unregister_dlg() to avoid assertion.