TLS certificate verification and callback to notify TCP/TLS transport state (thanks Rohit Agrawal, Klaus Darilion, and Pierre-Luc Bacon for the suggestions)

[nanang]

Issues:

• Domain name verification, IETF RFC draft could be found ​here.
• Add callback in PJSUA-LIB to notify transport state (e.g. transport is connected or disconnected). There are several use cases for this:
  • application send re-REGISTER when the connection is down. This should reconnect the transport.
  • application can inspect TLS connection/verification error and reconfigure TLS transport respectively (for example, to disable domain verification if TLS connection fails due to certificate error (perhaps after asking for user confirmation))

Specifications:

• Verification in SSL socket:
  • If verify_peer is set, SSL socket will disconnect the connection whenever there is any error occurred in the verification.
  • If verify_peer is not set, SSL socket will ignore any error in verification and continue establishing SSL handshake. Application may inspect the verification result in the SSL socket info or do further verification (e.g: domain name verification) in on_connect_complete() callback and decide whether SSL connection will be closed.
  • On both cases, the application may inspect the verification result in the SSL socket info.
  • The verification result in SSL socket info should show all verification errors, for example when remote certificate is expired and untrusted, the verification result should show both errors.
  • As domain name verification rules may vary among applications (e.g: SIP requires exact match and allows SIP URI identity, while HTTP allows wildcard, '*' and doesn't not say anything about URI identity), domain name verification is in application level and the SSL socket will just provide remote identities parsed from remote certificate, i.e: subjectAltName extension and subject common name.

• Verification in SIP TLS transport:
  • Add destination host name into pjsip_tx_data. The TLS transport will use it to get the server name for TLS verification.
  • Introduce a new SIP transport callback to notify transport states, i.e: connected, disconnected.
  • Add a verification step, i.e: domain name verification, then include its result to verification status of SSL socket info.
  • If verify_server/client is set and there is TLS verification error, TLS transport will disconnect the transport and notify the application via the new callback with event type 'disconnected', application can examine disconnection status code and TLS verification result via transport state info.
  • If verify_server/client is not set and there is TLS verification error, TLS transport will notify the application via the new callback with event type 'connected', application can examine TLS verification result via transport state info.

[nanang]

In r3106:

• Initial version of server domain name verification:
  • Updated SSL certificate info, especially identities info
  • Updated verification mechanism as in the specifications in ticket desc.
  • Added server domain name info in pjsip_tx_data.
  • Added alternative API for acquiring transport and creating transport of transport factory to include pjsip_tx_data param.
  • Server identity match criteria:
    • full host name match
    • wild card not accepted
    • if identity is URI, it must be SIP/SIPS URI
• Initial version of transport state notifications:
  • Added new API to set transport state callback in PJSIP and PJSUA.
  • Defined states: connected/disconnected, accepted/rejected, verification errors.
• Minors:
  • Updated SSL socket test: dump verification result, test of requiring client cert, and few minors.
  • Updated test cert to include subjectAltName extensions.
  • Added SSL certificate dump function.
  • Updated max number of socket async operations in Symbian sample apps (RSocketServ::Connect()) to 32 (was default 8).

[nanang]

In r3110:

• Updated transport state notification callback to return void.
• Updated transport state enum to only contain connected and disconnected, no more bitmask value.
• Added direction field to SIP transport.
• Removed remote hostname hash from transport key.
• Updated cert info dump to return -1 when buffer is insufficient.
• Added new error code PJSIP_TLS_ECERTVERIF.
• Updated get_cert_name() in ssl_sock_symbian.c to use heap buffer instead of stack.
• Minors, e.g: added prefix PJ in cipher types, docs.

[bennylp]

In r3112:

• fixed minor gcc warnings

[nanang]

In r3113:

• Minor fixed transport TCP of missing returning PJ_FALSE when connect attempt failed.

[bennylp]

Few fixes in r3119:

• PJSUA-LIB transport callback, if installed, will call the previously registered callback, to allow multiple transport callbacks to be installed
• there seem to be a bug with the use of "pjsip_tp_state_callback" everywhere (the "pjsip_tp_state_callback" type is pointer, but most variables of this type are declared to pointer too)