TLS certificate verification (thanks Rohit Agrawal, Klaus Darilion, and Pierre-Luc Bacon for the suggestions)

[nanang]

Issues:

• Domain name verification, IETF RFC draft could be found ​here.
• Add callback in PJSUA-LIB to notify transport state (e.g. transport is connected or disconnected). There are several use cases for this:
  • application send re-REGISTER when the connection is down. This should reconnect the transport.
  • application can inspect TLS connection/verification error and reconfigure TLS transport respectively (for example, to disable domain verification if TLS connection fails due to certificate error (perhaps after asking for user confirmation))

Specifications:

• Verification in SSL socket:
  • If verify_peer is set, SSL socket will stop the SSL session whenever there is any error occurred in the verification.
  • If verify_peer is not set, SSL socket will ignore any error in verification and continue establishing SSL handshake. Application may do the verification in on_connect_complete() callback by inspecting SSL socket info.
  • On both cases, the application may inspect the verification result in the SSL socket info.
  • The verification result in SSL socket info should show all verification errors, for example when remote certificate is expired and untrusted, the verification result should show both errors.
  • As domain name verification rules may vary among applications (e.g: SIP requires exact match and allows SIP URI identity, while HTTP allows wildcard, '*' and doesn't not say anything about URI identity), domain name verification is in application level and the SSL socket will just provide remote identities parsed from remote certificate, i.e: subjectAltName extension and subject common name.

• Verification in SIP TLS transport:
  • Add destination host name into pjsip_tx_data. The TLS transport will use it to get the server name for TLS verification.
  • Introduce a new SIP transport callback to notify transport states, i.e: connected, disconnected, and TLS verification errors.
  • Add verification step, domain name verification, and add this to verification status of SSL socket info.
  • If verify_server/client is set, TLS transport will disconnect the transport and notify the application via the new callback whenever there is any TLS verification error.
  • If verify_server/client is not set, TLS transport will just notify the application via the new callback, the return value of the callback will be used to decide whether transport should be shutdown (or application must always shutdown the transport when returning PJ_FALSE?).