SSL connection suddenly gets closed after sending packets intensively

[nanang]

When a sender sends packets intensively to a receiver that has packet processing rate lower than the sending rate, the packet will be buffered by PJLIB SSL socket and after sometime the sender will get PJ_ENOMEM error and eventually drop the connection while there are still some pending outgoing packets in PJLIB SSL socket. Application cannot trace which packets have been sent and which packets are still pending (buffered in SSL socket).

After investigation, when sending buffer is full, any sending operation will still write to OpenSSL BIO while it should not, it should simply return error, e.g: PJ_ENOMEM. After sometime, it somehow cause OpenSSL to initiate SSL renegotiation and when renegotiation fails (e.g: due to timeout), SSL connection will be dropped.

Thanks Peter Koletzki for the report.

[nanang]

In 5751:

Fixed #2099: Fixed SSL socket (OpenSSL backend), when sending buffer is full, any further send operation should be rejected immediately without writing to OpenSSL SSL BIO.