http://www.linphone.org/eng/documentation/dev/mediastreamer2.html

Mediastreamer2 - the multimedia streaming engine

Mediastreamer2 is a powerful and lightweighted streaming engine specialized for voice/video telephony applications .

It is the library that is responsible for all the receiving and sending of multimedia streams in linphone, including voice/video capture, encoding and decoding, and rendering.

Features

Read/Write from to an alsa device, an oss device, a windows waveapi device
Send and receive RTP packets
Encode and decode the following formats: speex, G711, GSM, H263, theora, iLBC, MPEG4, and H264.
Read and write from/to a wav file
Read YUV pictures from a webcam (provided that it has video4linux v1 or v2 driver)
Display YUV pictures (using SDL library or native apis on windows)
Dual tones generation
Echo cancelation, using the extraordinary echo canceler algorithm from the speex library
Audio conferencing
Audio parametric equalizer using a FIR filter
Volume control, automatic gain control

Mediastreamer2 can be extended with dynamic plugins, currently a H264 and an ILBC codec plugins are available.

Portability

Linux/x86 Linux/x86_64
Embedded Linux: ARM and Blackfin
Windows XP, Vista and 7
Mac OS X
Google Android

Design and principles

Each processing entity is contained within a MSFilter object. MSFilter(s) have inputs and/or outputs that can be used to connect from and to other MSFilters.

A trivial example to understand:

MSRtpRecv is a MSFilter that receives RTP packets from the network, unpacketize them and post them on its only output.
MSSpeexDec is a MSFilter that takes everything on its input assuming these are speex encoded packets, and decodes them and put the result on its output.
MSFileRec is a MSFilter that takes everything on its input and write it to wav file (assuming the input is 16bit linear pcm).

MSFilters can be connected together to become filter chain. If we assemble the three above examples, we obtain a processing chain that receives RTP packet, decode them and write the uncompressed result into a wav file.

The execution of the media processing work is scheduled by a MSTicker object, a thread that wakes up every 10 ms to process data in all the MSFilter chains it manages. Several MSTicker can be used simultaneously, for example one for audio filters, one for video filters, or one on each processor of the machine where it runs.

Mediastreamer2 is easy to use

If your intent is simply to create audio and video streams, a simple API is defined in audiostream.h and videostream.h to create audio and video streams.

If your intent is to add new functionalities to mediastreamer2, you'll be glad to know that implementing a mediastreamer2 filter is very straightforward. The mediastreamer2 filter encapsulation is very light.

Thanks to this lightweighted framework, developers can concentrate on what matters: the implementation of the signal/image processing algorithm !

Documentation

Mediastreamer2 is documented using doxygen. You can browse the API documention here.

Mediastreamer2 is suitable for embedded systems

Mediastreamer2 is light. For example on linux/x86 the full-featured shared library takes around 800ko unstripped and compiled with -g (debug). Data messages that carries the media data within mediastreamer2 chains are optimized using the famous sys-V mblk_t structure. This is to avoid copies as long it is possible and allow low cost fragmentation/re-assemble operations that are very common especially when processing video streams.
Mediastreamer2 is written in C
Mediastreamer2 compiles on arm with gcc.
Mediastreamer2 has only oRTP and libc as minimal dependencies. Others (ffmpeg, speex, alsa...) can be added optionnaly if you need all features.
Thanks to its plugin architecture, mediastreamer2 can be extended to interface with hardware codecs, for example video codecs dsp.

http://download-mirror.savannah.gnu.org/releases/linphone/mediastreamer/doc/group__mediastreamer2.html

mediastreamer2 library - a modular sound and video processing and streaming

Detailed Description

mediastreamer2 Version 2.4.0

See also:

http://savannah.gnu.org/projects/linphone

What is mediastreamer2

mediastreamer2 is a powerful engine to make audio and video streams. mediastreamer2 is GPL (COPYING). Please understand the licencing details before using it!

For any use of this library beyond the rights granted to you by the GPL license, please contact antisip at <jack@atosc.org >.

Some definitions.

Filter: A filter is a mediastreamer2 component that process data. A filter have 0 or several INPUT pins and 0 or several OUTPUT pins. Here is a list of possible use of filters:

   capture audio or video data.   play audio or display video data.   send or receive RTP data.   encode or decode audio or video data.   transform (resize video, resample audio...) data.   duplicate any kind of data.   mix audio/video data.

Graph: A graph is a manager of filters connected together. It will transfer data from OUTPUT pins to INPUT pins and will be responsible for running filters.

How do I use mediastremer2?

Mediastreamer2 can be used for a lot of different purpose. The primary use is to manage RTP audio and video session. You will need to use the API to build filters, link them together in a graph. Then the ticker API will help you to start and stop the graph.

Basic graph sample:

  AUDIO CAPTURE   -->   ENCODE  -->     RTP      FILTER      -->   FILTER  -->    FILTER

The above graph is composed of three filters. The first one has no input: tt captures audio data directly from the drivers and provide it to the OUTPUT pin. This data is sent to the INPUT pin of the encoder which of course encode the data and send it to its OUTPUT pin. This pin is connected to the INPUT pin of a filter capable to build and send RTP packets.

The modular design helps you to encode in many different format just by replacing the "ENCODE FILTER" with another one. mediastreamer2 contains internal support for g711u, g711a, speex and gsm. You can add new encoding format by implementing new filters which can then be dynamically loaded.

List of existing filters.

mediastreamer2 already provides a large set of filters. Here is a complete list of built-in filters.

 All supported platforms:   RTP receiver   RTP sender   tee (duplicate data)

 Audio Filters:   audio capture   audio playback     mme API (windows)     alsa API (linux)     oss API (linux)     arts API (linux)     portaudio API (macosx and other)   macsnd API (native macosx API -please do more testing...-)   several audio encoder/decoder: PCMU, PCMA, speex, gsm   wav file reader.   wav file recorder.   resampler.   conference bridge.   volume analyser.   acoustic echo canceller.   dtmf generation filter.

 Video Filters:   video capture     v4w API (windows)     directshow API (windows)     video4linux API (linux)   video display     v4w API (windows)     SDL API (linux, macosx...)   several audio encoder/decoder: H263-1998, MP4V-ES, theora   image resizer.   format converter. (RBG24, I420...)

 Plugin Filters:  iLBC decoder/encoder.

http://mirror.yongbok.net/nongnu/linphone/mediastreamer/doc/

Project : mediastreamer2 - a modular sound and video processing and streamingEmail : simon.morlat_at_linphone.orgLicense : GPLHome Page : http://savannah.gnu.org/projects/linphoneMediastreamer2 is a GPL licensed library to make audio andvideo real-time streaming and processing. Written in pure C,it is based upon the ortp library.Design:------Using mediastreamer2 will allow you to chain filters in a graph. Eachfilter will be responsible for doing some kind of processing and willdeliver data to the next filter. As an example, you could get somedata from network and unpack it in an RTP filter. This RTP filter willdeliver the data to a decoder (speex, G711...) which will deliver itto a filter that is able to play the PCM data or record it into a .wavfile.There is a doxygen documentation for more information.Features:--------mediastreamer2 already provides a large set of filters.Here is a complete list of built-in filters. All supported platforms: * RTP receiver * RTP sender * tee (duplicate data) Audio Filters: * audio capture * audio playback * mme API (windows) * alsa API (linux) * oss API (linux) * arts API (linux) * portaudio API (macosx and other) * macsnd API (native macosx API -please do more testing...-) * aq (audio queue, macos API too) * several audio encoder/decoder: PCMU, PCMA, speex, gsm * wav file reader. * wav file recorder. * resampler. * conference bridge. * volume analyser, gain control, and automatic gain control. * acoustic echo canceller. * dtmf generation filter. * parametric equalizer, can be used to compensate the spectral response of a bad quality speaker or microphone Video Filters: * video capture * v4w API (windows, deprecated) * directshow API (windows) * video4linux and video4linux2 APIs (linux) * video display * v4w API (windows) * SDL API (linux, macosx...) * several audio encoder/decoder: H263-1998, MP4V-ES, theora * image resizer. * format converter. (RBG24, I420...) Plugin Filters: * iLBC decoder/encoder. * H264 codec, based on x264Note that, you can build your own components/filters to do yourown processing or support other codecs.Installation procedure:-----------------------The program is known to run on linux, but might workon any unix and windows systems. $> ./configure $> make $> su -c 'make install'Contact information:--------------------For more information on mediastreamer2, any contributions, or any remarks,you can contact me at <simon.morlat_at_linphone.org>.Use the *linphone* mailing list for question about mediastreamer2. <linphone-developers@nongnu.org>.Subscribe by writing to: <linphone-developers-request@nongnu.org> with a subject set to "subscribe".Commercial support and licensing is provided by Belledonne Communicationshttp://www.belledonne-communications.com

Mediastreamer2