Android Hardware Decoding with MediaCodec2013-1-11

Android
OpenMAX

Finally, I must say, finally, we get low-level media APIs in Android, the Android hardware decoding and encoding APIs are finally available. It was a thing since Android 4.1 in Google IO 2012, but until now, when a new Android version 4.2 has been released, those low-level APIs are still too hard to use. There’re so many Android vendors, we can’t even get the same results from all Google’s Nexus devices.

Despite the annoyings, I still tried these APIs and prayed them to get better in next Android release, and next Android release would swallow market quikly.

I just finished a simple video player with the combination of MediaExtractor and MediaCodec, you may find the full project from my Github.

private MediaExtractor extractor;private MediaCodec decoder;private Surface surface;public void run() {  extractor = new MediaExtractor();  extractor.setDataSource(SAMPLE);  for (int i = 0; i < extractor.getTrackCount(); i++) {    MediaFormat format = extractor.getTrackFormat(i);    String mime = format.getString(MediaFormat.KEY_MIME);    if (mime.startsWith("video/")) {      extractor.selectTrack(i);      decoder = MediaCodec.createDecoderByType(mime);      decoder.configure(format, surface, null, 0);      break;    }  }  if (decoder == null) {    Log.e("DecodeActivity", "Can't find video info!");    return;  }  decoder.start();  ByteBuffer[] inputBuffers = decoder.getInputBuffers();  ByteBuffer[] outputBuffers = decoder.getOutputBuffers();  BufferInfo info = new BufferInfo();  boolean isEOS = false;  long startMs = System.currentTimeMillis();  while (!Thread.interrupted()) {    if (!isEOS) {      int inIndex = decoder.dequeueInputBuffer(10000);      if (inIndex >= 0) {        ByteBuffer buffer = inputBuffers[inIndex];        int sampleSize = extractor.readSampleData(buffer, 0);        if (sampleSize < 0) {          Log.d("DecodeActivity", "InputBuffer BUFFER_FLAG_END_OF_STREAM");          decoder.queueInputBuffer(inIndex, 0, 0, 0, MediaCodec.BUFFER_FLAG_END_OF_STREAM);          isEOS = true;        } else {          decoder.queueInputBuffer(inIndex, 0, sampleSize, extractor.getSampleTime(), 0);          extractor.advance();        }      }    }    int outIndex = decoder.dequeueOutputBuffer(info, 10000);    switch (outIndex) {    case MediaCodec.INFO_OUTPUT_BUFFERS_CHANGED:      Log.d("DecodeActivity", "INFO_OUTPUT_BUFFERS_CHANGED");      outputBuffers = decoder.getOutputBuffers();      break;    case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED:      Log.d("DecodeActivity", "New format " + decoder.getOutputFormat());      break;    case MediaCodec.INFO_TRY_AGAIN_LATER:      Log.d("DecodeActivity", "dequeueOutputBuffer timed out!");      break;    default:      ByteBuffer buffer = outputBuffers[outIndex];      Log.v("DecodeActivity", "We can't use this buffer but render it due to the API limit, " + buffer);      // We use a very simple clock to keep the video FPS, or the video      // playback will be too fast      while (info.presentationTimeUs / 1000 > System.currentTimeMillis() - startMs) {        try {          sleep(10);        } catch (InterruptedException e) {          e.printStackTrace();          break;        }      }      decoder.releaseOutputBuffer(outIndex, true);      break;    }    if ((info.flags & MediaCodec.BUFFER_FLAG_END_OF_STREAM) != 0) {      Log.d("DecodeActivity", "OutputBuffer BUFFER_FLAG_END_OF_STREAM");      break;    }  }  decoder.stop();  decoder.release();  extractor.release();}

Compared to the OMXCodec methods introduced in an early post, this method is more recommended if you’re targeting Android 4.1 and later. But it’s still too limited, you can’t do anything with the decoded video frame but render them to surface, because there’re more than 40 color formats to deal with and many of them are vendor proprietary without documentation.

Use Android Hardware Decoder with OMXCodec in NDK2012-12-20

FFmpeg
Android
NDK
OpenMAX

Android API dosen’t expose interface to the device’s hardware decoding module, but the OS does have a beautiful private library called stagefright, which encapsulates elegant hardware accelerated decoding & encoding API, which can be used accompanied with FFmpeg or GStreamer.

Don’t fear the private API would change frequently suggested by many developers or Google’s Android engineers, many famous apps tell us these APIs won’t change hardly. The Adobe Flash player, VPlayer and Firefox for Android have been using the stagefright API to do hardware decoding for quite a long time.

To procceed in this post, you need to be familiar with the techinques menthioned in How to Render Image Buffer in Android NDK Native Code.

MediaSource

Android engineers use OpenMAX specification to communicate with different devices’ hardware decoders, in order to simplify the OpenMAX procedure, they created the stagefright library.

To understand this library, the MediaSource class is one of the most important keys. MediaSourceis declared in frameworks/av/include/media/stagefright/MediaSource.h. Simply put, a MediaSourceshould provide one main method called read, which should read some data from some data source and return the result to a MediaBuffer.

To fulfill a complete video decoding work, we need two different MediaSource, one to read encoded video data from file, it’s something like FFmpeg’s demuxer, and the other one acts just like FFmpeg’s decoder, which will read and decode the encoded data from the demuxer, then produce decoded video frames.

AVFormatSource

AVFormatSource, a very simple MediaSource implementation based on FFmpeg’s libavformat, is the first MediaSource, which works as the demuxer.

It’s the plenty of powerful demuxers why I use libavformat to create a fresh new MediaSourcerather than using the existing ones in AOSP.

#define FFMPEG_AVFORMAT_MOV "mov,mp4,m4a,3gp,3g2,mj2"using namespace android;class AVFormatSource : public MediaSource {  public:    AVFormatSource(const char *videoPath);        virtual status_t read(MediaBuffer **buffer, const MediaSource::ReadOptions *options);    virtual sp getFormat() { return mFormat; }    virtual status_t start(MetaData *params) { return OK; }    virtual status_t stop() { return OK; }  protected:    virtual ~AVFormatSource();  private:    AVFormatContext *mDataSource;    AVCodecContext *mVideoTrack;    AVBitStreamFilterContext *mConverter;    MediaBufferGroup mGroup;    sp mFormat;    int mVideoIndex;};AVFormatSource::AVFormatSource(const char *videoPath) {  av_register_all();  mDataSource = avformat_alloc_context();  avformat_open_input(&mDataSource, videoPath, NULL, NULL);  for (int i = 0; i < mDataSource->nb_streams; i++) {    if (mDataSource->streams[i]->codec->codec_type == AVMEDIA_TYPE_VIDEO) {      mVideoIndex = i;      break;    }  }  mVideoTrack = mDataSource->streams[mVideoIndex]->codec;  size_t bufferSize = (mVideoTrack->width * mVideoTrack->height * 3) / 2;  mGroup.add_buffer(new MediaBuffer(bufferSize));  mFormat = new MetaData;  switch (mVideoTrack->codec_id) {    case CODEC_ID_H264:      mConverter = av_bitstream_filter_init("h264_mp4toannexb");      mFormat->setCString(kKeyMIMEType, MEDIA_MIMETYPE_VIDEO_AVC);      if (mVideoTrack->extradata[0] == 1) {        mFormat->setData(kKeyAVCC, kTypeAVCC, mVideoTrack->extradata, mVideoTrack->extradata_size);      }      break;    case CODEC_ID_MPEG4:      mFormat->setCString(kKeyMIMEType, MEDIA_MIMETYPE_VIDEO_MPEG4);      if (mDataSource->iformat && mDataSource->iformat->name && !strcasecmp(mDataSource->iformat->name, FFMPEG_AVFORMAT_MOV)) {        MOVContext *mov = (MOVContext *)(mDataSource->priv_data);        if (mov->esds_data != NULL && mov->esds_size > 0 && mov->esds_size < 256) {          mFormat->setData(kKeyESDS, kTypeESDS, mov->esds_data, mov->esds_size);        }      }      break;    case CODEC_ID_H263:      mFormat->setCString(kKeyMIMEType, MEDIA_MIMETYPE_VIDEO_H263);      break;    default:      break;  }  mFormat->setInt32(kKeyWidth, mVideoTrack->width);  mFormat->setInt32(kKeyHeight, mVideoTrack->height);}status_t AVFormatSource::read(MediaBuffer **buffer, const MediaSource::ReadOptions *options) {  AVPacket packet;  status_t ret;  bool found = false;  while (!found) {    ret = av_read_frame(is->formatCtx, &packet);    if (ret < 0) {      return ERROR_END_OF_STREAM;    }    if (packet.stream_index == mVideoIndex) {      if (mConverter) {        av_bitstream_filter_filter(mConverter, mVideoTrack, NULL, &packet.data, &packet.size, packet.data, packet.size, packet.flags & AV_PKT_FLAG_KEY);      }      ret = mGroup.acquire_buffer(buffer);      if (ret == OK) {        memcpy((*buffer)->data(), packet.data, packet.size);        (*buffer)->set_range(0, packet.size);        (*buffer)->meta_data()->clear();        (*buffer)->meta_data()->setInt32(kKeyIsSyncFrame, packet.flags & AV_PKT_FLAG_KEY);        (*buffer)->meta_data()->setInt64(kKeyTime, packet.pts);      }      found = true;    }    av_free_packet(&packet);  }  return ret;}AVFormatSource::~AVFormatSource() {  if (mConverter) {    av_bitstream_filter_close(mConverter);  }  av_close_input_file(mDataSource);}

I’ve omitted all the header includes and error checks to make code more clean and easy to read.

From the code above, I only try to decode H.263, H.264 and MPEG4, all other formats are not supported from Android’s documentation officially, though some vendors support more formats such as WMV, FLV or RM.

Pay attention to the FFmpeg code I used, it’s the patched version from VPlayer for Android, I’ve also written a simple post on building FFmpeg for Android.

OMXCodec

The other MediaSource is OMXCodec, which do the actual decoding work.

You can locate the OMXCodec.cpp at AOSP tree’s frameworks/av/media/libstagefright/OMXCodec.cpp, and OMXCodec.h at frameworks/av/include/media/stagefright/OMXCodec.h. Then you will find it’s so easy to use the OMXCodec class in our own NDK code, especially if you found some sample usage fromlibstagefright/AwesomePlayer.cpp.

// At first, get an ANativeWindow from somewheresp mNativeWindow = getNativeWindowFromSurface();// Initialize the AVFormatSource from a video filesp mVideoSource = new AVFormatSource(filePath);// Once we get an MediaSource, we can encapsulate it with the OMXCodec nowOMXClient mClient;mClient.connect();sp mVideoDecoder = OMXCodec::Create(mClient.interface(), mVideoSource->getFormat(), false, mVideoSource, NULL, 0, mNativeWindow);mVideoDecoder->start();// Just loop to read decoded frames from mVideoDecoderfor (;;) {    MediaBuffer *mVideoBuffer;    MediaSource::ReadOptions options;    status_t err = mVideoDecoder->read(&mVideoBuffer, &options);    if (err == OK) {        if (mVideoBuffer->range_length() > 0) {            // If video frame availabe, render it to mNativeWindow            sp metaData = mVideoBuffer->meta_data();            int64_t timeUs = 0;            metaData->findInt64(kKeyTime, &timeUs)            native_window_set_buffers_timestamp(mNativeWindow.get(), timeUs * 1000);            err = mNativeWindow->queueBuffer(mNativeWindow.get(), mVideoBuffer->graphicBuffer().get());            if (err == 0) {                metaData->setInt32(kKeyRendered, 1);            }        }        mVideoBuffer->release();    }}// Finally release the resourcesmVideoSource.clear();mVideoDecoder->stop();mVideoDecoder.clear();mClient.disconnect();

I think you will figure that OMXCodec is also a child class of MediaSource, and it’s constructed from another MediaSource, which is AVFormatSource in this example.

Indeed, if you take a look at OMXCodec.cpp, you should notice its read method is implemented in three main steps:

Read encoded video data with mVideoSource.read to a MediaBuffer. The video data is just anAVPacket in this FFmpeg sample.
Pass the MediaBuffer to device’s decoding module, which will do the actual decoding work.
Return the decoded video frame as a MediaBuffer, which we can render to screen.

Troubleshooting

It’s a pain to deal with the product of so many Android devices, Android versions and video formats. As my experience, the chips from TI, Nvidia, QCom and other vendors all have slight differences to each other in using OMXCodec. It’s harder than deal with different cameras and different screen sizes, but it deserves if you want to play high quality videos with video format not supported by Android!

I’m not able to list all the problems appeared to me, but count some main fields where the problems most occur.

Different color space used in the decoded frames, it’s best to find those information from CyanogenMod’s source code.
The difference in managing MediaBuffer memory, mainly when releasing MediaBuffer.
Some decoders require you to crop the decoded frames’ dimensions.
It’s important to reorder the decoded frames’ timestamps for some decoders, but not for the others.

Resources

This approach has been used in several open source projects listed below.

The libstagefright.cpp in FFmpeg’s source code is the most simple start point to understand the concepts. But look out, don’t use it in production code, it’s full of bugs.
Firefox for Android added H.264 hardware decoding recently, the related code is available in$MOZILLA/content/media.
VLC for Android is also capable of hardware decoding, but I haven’t looked its code yet.

It’s not an easy work to use hardware decoding in Android, but all these projects have proved that it’s possible and nearly perfect.

Screenshot_from_2012-12-20_00_45_06.png (20.5 KB)

Use Android(安卓)Hardware Decoder with OMXCodec in NDK