Opened 9 years ago

Closed 8 years ago

#1182 closed enhancement (fixed)

vstream (video stream) implementation and sample

Reported by: bennylp Owned by: nanang
Priority: normal Milestone: release-2.0-dev
Component: pjmedia Version: 2.0-dev-branch
Keywords: Cc:
Backport to 1.x milestone: Backported:

Description (last modified by nanang)

  • Either extend existing stream.c to support video, or create a new vstream.c.
  • Use vstreamutil.c as demo.

There is already video stream implementation in the stream.c and sample in the vstreamutil.c, however it's just proof-of-concept version, so the to do/consider list will be:

  • proper picture bitstream generation from RTP packet, currently the video stream will just 'concatenate' incoming RTP to get a complete picture bitstream, and everytime it got end-of-picture flag, RTP header m==1, push the picture into jitter buffer, so in case of unordered RTP the picture bitstreams will be truncated
  • perhaps jitter buffer frame content/granularity should be RTP payload instead of a picture bitstream (picture bitstream will be generated right before decoding, instead of inline with incoming RTP packets), however current jitter buffer only has RTP sequence (no RTP timestamp), so need to research if the current jitter buffer fits this requirements along with optimization consideration (e.g: frame size, minimal memcpy)
  • audio-video sync
  • encoder & decoder size can be different, e.g: remote endpoint, which is an embedded/mobile, only supports QCIF while local endpoint, which is desktop, prefers CIF, so the issues will be:
    • which size should be put in the stream info (as currently it only has one 'format info' field)?
    • the decoder output format (e.g: format id & size) can only be ensured after a successful picture decoding (the format setting in opening the decoder is rather unusable and the local SDP may offer multiple sizes), need a way to notify application about the active format.

Change History (15)

comment:1 Changed 9 years ago by bennylp

  • Description modified (diff)

comment:2 Changed 9 years ago by nanang

  • Description modified (diff)

comment:3 Changed 9 years ago by nanang

(In [3418]) Re #1182: Reverted back video part in stream.h/c

comment:4 Changed 9 years ago by nanang

(In [3420]) Re #1182:

  • Added video stream interface in vid_stream.h, the video stream will be able to handle different video formats in encoding and decoding direction.
  • Renamed video device stream class identifiers from 'pjmedia_vid_stream*' to 'pjmedia_vid_dev_stream*' as 'pjmedia_vid_stream' is used by video stream interface.
  • Added ffmpeg video capability to be able to parse SDP format param for H263 and also decide video format for encoding direction based on remote preference and local format-capability setting.
  • Added some new APIs in jitter buffer for handling video stream: pjmedia_jbuf_put_frame3(), pjmedia_jbuf_get_frame3(), pjmedia_jbuf_peek_frame(), and pjmedia_jbuf_remove_frame().
  • Moved pjmedia_stream_info_from_sdp() from session to stream

comment:5 Changed 9 years ago by nanang

(In [3425]) Re #1182:

  • Renamed vstreamutil.c to vid_steamutil.c just for filename format consistency reason.
  • Updated sample app simpleua.c and vid_streamutil.c to sync with updated API, e.g: strip session usage, two media ports exported video streams for each dir.
  • Added vid_streamutil.c capability to be able to stream video file (involving transcoding when video codec used in the file different to the video stream codec), also updated AVI player and ffmpeg codecs to be able to read and decode XVID/MPEG4 codec.
  • Fixed bug wrong media type check in stream.c and vid_stream.c in creating stream info from SDP.
  • Minor update: docs, logs, app samples makefiles.

comment:6 Changed 9 years ago by ming

(In [3431]) Re #1182:

  • support for format modification after creating video port (currently for renderer with active role only).
  • support for format modification after opening SDL renderer.

comment:7 Changed 9 years ago by nanang

(In [3432]) Re #1182:

  • Fixed ffmpeg codec: decoding buffer size check (should not use assertion), reset output bit_info before decoding, removed unused code of auto resize when decoder output size changed.
  • Fixed bug in sdl_factory_default_param() of sdl_dev.c in device direction check.
  • Fixed compile errors sample app vid_streamutil.c of bad local variable definiton.
  • Updated sample app vid_streamutil.c default renderer size to 640 x 480.

comment:8 Changed 9 years ago by nanang

(In [3435]) Re #1182:

  • Added remote frame-rate detection in to video stream.
  • Fixed bitrate settings in ffmpeg codec.
  • Fixed SDL dev to update internal SDL info when format changed.
  • Minor fixes/updates, e.g:
    • added cleanup steps, fixed logs, etc, in sample app simpleua.c and vid_streamutil.c
    • fixed/added docs of the new APIs in the jitter buffer.

comment:9 Changed 9 years ago by bennylp

(In [3445]) Re #1182: added the video stream to pjmedia Makefile

comment:10 Changed 9 years ago by bennylp

(In [3447]) Re #1182: renamed pjmedia_vid_stream to pjmedia_vid_dev_stream due to conflict with "the real" vid_stream

comment:11 Changed 8 years ago by bennylp

(In [3513]) Re #1182:

  • Removed RTCP-XR from video stream.
  • Fixed few compile warnings on GCC for incompatible pointers.

comment:12 Changed 8 years ago by bennylp

(In [3514]) Re #1182: fixed misc error with RTCP XR

comment:13 Changed 8 years ago by nanang

(In [3534]) Re #1182: Added macro setting PJMEDIA_STREAM_RESV_PAYLOAD_LEN to let stream reserve some space for application extra data in the RTP payload, e.g: SRTP auth tag.

comment:14 Changed 8 years ago by nanang

(In [3536]) Re #1182: Applied PJMEDIA_STREAM_RESV_PAYLOAD_LEN (see r3534) in stream.c.

comment:15 Changed 8 years ago by bennylp

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.