Race condition in OpenSSL socket

[ming]

A race condition may occur in OpenSSL socket if more than one worker threads are used. It happens when a handshake failure in one thread causes it to destroy SSL objects while other threads are still accessing them.

Stack traces:

#0  0x00007ffff74415e3 in __memcpy_ssse3_back () from /lib64/libc.so.6
#1  0x00000000006f3c65 in pj_memcpy (dst=0x7fffe8072d20, src=0x7fffe7ffb170, size=18446744073709551614) at ../include/pj/string.h:654
#2  0x00000000006f60a2 in flush_write_bio (ssock=0x7fffe801d308, send_key=0x7fffe801d6c8, orig_len=0, flags=0) at ../src/pj/ssl_sock_ossl.c:1484
#3  0x00000000006f62cc in do_handshake (ssock=0x7fffe801d308) at ../src/pj/ssl_sock_ossl.c:1559
#4  0x00000000006f6406 in asock_on_data_read (asock=0x7fffe8061158, data=0x7fffe8062138, size=343, status=0, remainder=0x7fffe7ffb260) at ../src/pj/ssl_sock_ossl.c:1615
#5  0x00000000006eb300 in ioqueue_on_read_complete (key=0xb390c8, op_key=0x7fffe8064148, bytes_read=343) at ../src/pj/activesock.c:493
#6  0x00000000006e536c in ioqueue_dispatch_read_event (ioqueue=0xb33458, h=0xb390c8) at ../src/pj/ioqueue_common_abs.c:605

And:

...
#23 0x000000000056b3aa in ssl_cert_free ()
#24 0x0000000000567b7f in SSL_CTX_free ()
#25 0x00000000006f4bf4 in destroy_ssl (ssock=0x7fffe801d308) at ../src/pj/ssl_sock_ossl.c:802
#26 0x00000000006f4c30 in reset_ssl_sock_state (ssock=0x7fffe801d308) at ../src/pj/ssl_sock_ossl.c:818
#27 0x00000000006f762f in pj_ssl_sock_close (ssock=0x7fffe801d308) at ../src/pj/ssl_sock_ossl.c:2251
#28 0x00000000006f59b4 in on_handshake_complete (ssock=0x7fffe801d308, status=470005) at ../src/pj/ssl_sock_ossl.c:1218
#29 0x00000000006f6d2f in asock_on_accept_complete (asock=0xfc1578, newsock=32, src_addr=0xfc18a0, src_addr_len=16) at ../src/pj/ssl_sock_ossl.c:1943

If the race condition causes more than one threads to close the activesock (and unregister the same ioqueue key more than once), it could cause ioqueue->count to go below zero and produce error, such as: "pj_ioqueue_register_sock error: too many files" (PJ_ETOOMANY) when trying to register a new key via pj_ioqueue_register_sock()/pj_ioqueue_register_sock2().

Thanks to Peter Koletzki from the report.

[ming]

In 5338:

Close #1930: Race condition in OpenSSL socket

A workaround to solve the race condition based on ticket #985.

[ming]

In 5343:

Re #1930: Fixed race condition when more than one thread try to close the same activesock.

[ming]

In 5344:

Re #1930: Revert back the change to comment do_handshake() in on_accept_complete() committed in r5343.

[nanang]

In 5367:

Re #1930: Fixing stuck issue or server socket not closed after an incoming connection fails, e.g: due to no shared cipher, and timer heap is not set.