TLS certificate verification (thanks Rohit Agrawal, Klaus Darilion, and Pierre-Luc Bacon for the suggestions)

[nanang]

Issues:

• Domain name verification, IETF RFC draft could be found ​here.
• Add callback in PJSUA-LIB to notify transport state (e.g. transport is connected or disconnected). There are several use cases for this:
  • application send re-REGISTER when the connection is down. This should reconnect the transport.
  • application can inspect TLS connection/verification error and reconfigure TLS transport respectively (for example, to disable domain verification if TLS connection fails due to certificate error (perhaps after asking for user confirmation))

Specifications:

• Verification in SSL socket:
  • If verify_peer is set, SSL socket will disconnect the connection whenever there is any error occurred in the verification.
  • If verify_peer is not set, SSL socket will ignore any error in verification and continue establishing SSL handshake. Application may inspect the verification result in the SSL socket info or do further verification (e.g: domain name verification) in on_connect_complete() callback and decide whether SSL connection will be closed.
  • On both cases, the application may inspect the verification result in the SSL socket info.
  • The verification result in SSL socket info should show all verification errors, for example when remote certificate is expired and untrusted, the verification result should show both errors.
  • As domain name verification rules may vary among applications (e.g: SIP requires exact match and allows SIP URI identity, while HTTP allows wildcard, '*' and doesn't not say anything about URI identity), domain name verification is in application level and the SSL socket will just provide remote identities parsed from remote certificate, i.e: subjectAltName extension and subject common name.

• Verification in SIP TLS transport:
  • Add destination host name into pjsip_tx_data. The TLS transport will use it to get the server name for TLS verification.
  • Introduce a new SIP transport callback to notify transport states, i.e: connected, disconnected, TLS verification error, or combination of them, see below for more detail.
  • Add a verification step, i.e: domain name verification, then include its result to verification status of SSL socket info.
  • If verify_server/client is set, TLS transport will disconnect the transport and notify the application via the new callback, with event type 'disconnected + TLS verification error', whenever there is any TLS verification error.
  • If verify_server/client is not set, TLS transport will just notify the application via the new callback, with event type 'connected + TLS verification error', whenever there is any TLS verification error, the return value of the callback will be used to decide whether transport should be shutdown. In case there is no verification error, the application will be notified via the new callback with event type 'connected'.