New codec interface for incremental encoding to improve performance with multiple frames per packet

[bennylp]

When stream is configured to send multiple frames per packet, currently this will cause the stream to encode and pack N frames simultaneously in one call to put_frame(). This may put significant load on the CPU since we are encoding that many frames at once and some device may be sensitive to the delay. It has been reported that this may cause audio disruption on iPhone 4S.

This ticket implements new interface to the codec to enable encoding frame one by one on each put_frame() cycle and only pack them together once we have enough frame to send. The new codec interface looks like the following:

    /**
     * Encode a single audio frame without performing any packing to
     * the output. Normally the input frame should have exactly the same
     * ptime as the codec frame time, however when the codec frame ptime
     * is 10 ms, this may be called with input frame containing 20ms
     * worth of samples.
     *
     * @param codec	The codec instance.
     * @param input	Input audio frame.
     * @param out_size	The length of buffer in the output frame.
     * @param output	The output frame.
     *
     * @return		PJ_SUCCESS on success or the appropriate error code.
     */
    pj_status_t (*encode_single)(pjmedia_codec *codec,
				 const struct pjmedia_frame *input,
				 unsigned out_size,
				 struct pjmedia_frame *output);

    /**
     * Pack one or more encoded audio frames into packet, suitable to be
     * placed as payload of a transport such as RTP.
     *
     * @param codec	The codec instance.
     * @param pack_cfg	Codec packing settings to specify how audio frames
     * 			should be arranged in the output buffer. Use NULL
     * 			to use codec's default packing setting.
     * @param frame_cnt	Number of input frames.
     * @param inputs	Input frames, which is an array of encoded frames.
     * @param out_size	Size of the output buffer.
     * @param output	Frame buffer to receive the output.
     *
     * @return		PJ_SUCCESS on success.
     */
    pj_status_t (*pack)(pjmedia_codec *codec,
		        pjmedia_codec_pack_cfg *pack_cfg,
			unsigned frame_cnt,
			pjmedia_frame inputs[],
			unsigned out_size,
			struct pjmedia_frame *output);

The stream will be modified to support this feature.

There will also be a new field in codec param to enable/disable the new feature on per stream basis:

struct pjmedia_codec_param
{
    ...

    struct {
        ...
	unsigned    pack:1;	    /**< Enable encode_single()&pack()  */
        ...
    } setting;
};

Only if pjmedia_codec_param.setting.pack is set to 1, the stream will use the new interface, and only if the codec supports it.

The first implementation will add this feature to G.711 and OpenCore AMR codecs.

[bennylp]

Patch to implement it. Tested, but it doesn't seem to help the iPhone 4S case hence it's pending

[nanang]

TeluuCon? 2018: recent CPUs should have no issue with multiple frames audio encoding load.