Welcome to Pete Brown's 10rem.net

First time here? If you are a developer or are interested in Microsoft tools and technology, please consider subscribing to the latest posts.

You may also be interested in my blog archives, the articles section, or some of my lab projects such as the C64 emulator written in Silverlight.

(hide this)

Silverlight 3 – Creating Video from Raw Bits using a MediaStreamSource

Pete Brown - 18 March 2009

There are a number of ways to get pixels on the screen in Silverlight 3. One you may not have considered, especially with the new Bitmap API being an otherwise obvious choice, is the newly enhanced MediaStreamSource API

The MediaStreamSource API was available in Silverlight 2, but it required you to transcode into a supported format. The new version of the API lets you create raw video images much more easily. There are a number of use-cases for this including complete bitmap-based games, creating chroma-keyed/alpha video, and in my case, using it to show the display of my Commodore 64 emulator at 50 frames per second (PAL standard)

(You can also use the MediaStreamSource to create audio from raw bits. Look for that in another upcoming post.)

image

So how hard is this to do? Well, once you get the basic setup down, it's actually really easy. The thing to remember is your code will be called by Silverlight, not the other way around. Therefore you just need to respond appropriately when, for example, Silverlight requests the next frame.

First, you want to create a class that inherits from MediaStreamSource

public class VideoMediaStreamSource : MediaStreamSource

Pixel Buffers

The emulated MOSS 6569 VIC-II chip (the equivalent of a video card today) in my code pushes pixels out to my media stream source until it reaches the end of the current frame. Once that happens, it calls Flip(). I keep two pixel buffers around to eliminate any tearing or half-updates on the screen where Silverlight requests a frame that is not yet complete. One buffer is what the VIC is writing to, the other is what Silverlight is reading from. When the VIC calls Flip(), I swap the two.

private byte[][] _frames = new byte[2][];
public VideoMediaStreamSource(int frameWidth, int frameHeight)
{
    _frameWidth = frameWidth;
    _frameHeight = frameHeight;

    _framePixelSize = frameWidth * frameHeight;
    _frameBufferSize = _framePixelSize * BytesPerPixel;

    // PAL is 50 frames per second
    _frameTime = (int)TimeSpan.FromSeconds((double)1 / 50).Ticks;

    _frames[0] = new byte[_frameBufferSize];
    _frames[1] = new byte[_frameBufferSize];

    _currentBufferFrame = 0;
    _currentReadyFrame = 1;
}
(yes, I could have done something clever with xor here)
public void Flip()
{
    int f = _currentBufferFrame;
    _currentBufferFrame = _currentReadyFrame;
    _currentReadyFrame = f;
}

Writing a pixel is pretty easy, as long as you get the format correct. Notice the order of the component R, G, B and A parts. BytesPerPixel is defined as 4 as this is a 32 bit color value.

public void WritePixel(int position, Color color)
{
    int offset = position * BytesPerPixel;

    _frames[_currentBufferFrame][offset++] = color.B;
    _frames[_currentBufferFrame][offset++] = color.G;
    _frames[_currentBufferFrame][offset++] = color.R;
    _frames[_currentBufferFrame][offset++] = color.A;

}

Opening your Video 

Surprisingly, this is much easier for video than it is for audio. For audio, you need to fill out a whole WaveFormatEx structure. For video, you simply need to set the appropriate FourCC code (like RGBA or YV12), the frame width and the frame height.

protected override void OpenMediaAsync()
{
    // Init
    Dictionary<MediaSourceAttributesKeys, string> sourceAttributes = 
        new Dictionary<MediaSourceAttributesKeys, string>();
    List<MediaStreamDescription> availableStreams = 
        new List<MediaStreamDescription>();

    PrepareVideo();

    availableStreams.Add(_videoDesc);

    // a zero timespan is an infinite video
    sourceAttributes[MediaSourceAttributesKeys.Duration] = 
        TimeSpan.FromSeconds(0).Ticks.ToString(CultureInfo.InvariantCulture);

    sourceAttributes[MediaSourceAttributesKeys.CanSeek] = false.ToString();

    // tell Silverlight that we've prepared and opened our video
    ReportOpenMediaCompleted(sourceAttributes, availableStreams);
}

private void PrepareVideo()
{
    _frameStream = new MemoryStream();

    // Stream Description 
    Dictionary<MediaStreamAttributeKeys, string> streamAttributes = 
        new Dictionary<MediaStreamAttributeKeys, string>();

    streamAttributes[MediaStreamAttributeKeys.VideoFourCC] = "RGBA";
    streamAttributes[MediaStreamAttributeKeys.Height] = _frameHeight.ToString();
    streamAttributes[MediaStreamAttributeKeys.Width] = _frameWidth.ToString();
    
    MediaStreamDescription msd = 
        new MediaStreamDescription(MediaStreamType.Video, streamAttributes);

    _videoDesc = msd;
}

Requesting Frames

Ignore for a moment that the code checks for both audio and video here. While I have snipped some of that from the other examples above, the code is actually set up to eventually serve up the audio stream from the Silverlight SID chip as well as the video from the VIC chip.

protected override void GetSampleAsync(MediaStreamType mediaStreamType)
{
    if (mediaStreamType == MediaStreamType.Audio)
    {
        GetAudioSample();
    }
    else if (mediaStreamType == MediaStreamType.Video)
    {
        GetVideoSample();
    }
}

Now in the GetVideoSample() method, I'm creating a new stream with each request. I don't like the approach, but I was getting out of memory errors with early Silverlight 3 builds if I tried to reuse the stream and simply provide offsets into it. I'll debug this some more before posting the final code.

private void GetVideoSample()
{
    _frameStream = new MemoryStream();
    _frameStream.Write(_frames[_currentReadyFrame], 0, _frameBufferSize);

    // Send out the next sample
    MediaStreamSample msSamp = new MediaStreamSample(
        _videoDesc,
        _frameStream,
        0,
        _frameBufferSize,
        _currentVideoTimeStamp,
        _emptySampleDict);

    _currentVideoTimeStamp += _frameTime;

    ReportGetSampleCompleted(msSamp);
}

One key part of the code above is the _currentVideoTimeStamp += _frameTime. Frame Time is a constant equal to 1/50 of a second in 100 nanosecond intervals. Setting the frametime there tells Silverlight that we're serving up 50 frames per second. I haven't tried varying that number, but you could probably get some interesting performance characteristics by playing around with that in real-time.

Wiring it Up

Hooking it up to Silverlight couldn't be easier:

<MediaElement x:Name="VideoDisplay"
              Grid.Row="0"
              Grid.Column="0"
              Stretch="Uniform"
              IsHitTestVisible="False"
              Margin="4" />

VideoDisplay.SetSource(_c64.Display.Video);

_c64.Display.Video is an instance of my MediaStreamSource class.

Conclusion

So there you have it, runtime-generated video inside a Silverlight 3 application. Pretty cool if you ask me!

I'll make all of this code available as part of various projects on codeplex later. Stay tuned. In the mean time, the complete source code for this one class is available here.

     
posted by Pete Brown on Wednesday, March 18, 2009
filed under:      

29 comments for “Silverlight 3 – Creating Video from Raw Bits using a MediaStreamSource”

  1. Alexandresays:
    Hello,
    thanks for this sample!
    I really interested by using dynamic audio with the audio raw pipeline.
    I tried to modify your sample but the GetSampleAsync method never gets called.

    Do you know any reason for that? Have you been able to test it?
    Thanks!
  2. Pete Brownsays:
    @Alexandra you'll need to do some additional setup to tell Silverlight you have a video stream available. I did that for the C64 emulator and will blog the details soon. The code will also come out when the hanselminutes podcast on that subject goes live.

    @Anot Definitely! If you can read the ogg format, you'll be able to do it. THere are lots of other cool uses for this too, in addition to transcoding. BTW, didn't know ogg was video, always thought it was just audio?
  3. Alexandresays:
    I managed to have a fully sample working (see http://silverlight.net/forums/t/82270.aspx), but now, the issue is to achieve a very low audio latency... not sure it is possible in SL3...
  4. Raghusays:
    Hi,

    Nice examples. I was going through the sample and created a sample application but it gives error for WaveFormatEx. can u please tell me what i need to add my project to solve this and also i m not getting any out put in mediasource. Could u please share the complete source code?
  5. Pete Brownsays:
    @Raghu

    That's pretty vague. What error are you getting?

    Also, you posted this on the video example. Are you doing audio as well? If not, you don't need WaveFormatEx

    The video example source code is linked in the post above. See last paragraph.

    Pete
  6. Raghusays:
    Hi Pete,
    Thanx for ur reply. I am able to solve the WaveFormatEx problem.

    I downloaded the VideoMediaStreamSource and written a sample application. But when i try to play a file i m not getting any thing in the media player.I am not getting any error also.
    My code is as follows:
    VideoMediaStreamSource ms = new VideoMediaStreamSource(ofd.File.OpenRead(), 320, 240);
    me.SetSource(ms);
    am i doing something wrong here. and what type of file i need to provide it to work?
  7. Pete Brownsays:
    @Raghu

    You're not giving me much to work from here. It's like me sending you an email that says "My car is making a noise, what's wrong with it?" :)

    What exactly are you trying to do?

    If you're loading a file, you will need to create a custom codec that will pull bytes from the file and send frames via MediaStreamSource.

    If you're trying to use a supported Silverlight media format (WMV, WMA, MP3, MP4/h.264), you don't need to create a media stream source; you just set the stream as the source for the media element.

    The forums are a great place for stuff like this; that way you can post more code and get more than one opinion: http://silverlight.net

    Pete
  8. Alexsays:
    Good Job.
    But I can't test your code.
    I made own test code with MediaStreamSource.
    But in my test sample, I can't see any image from screen.
    But MediaElememt called GetSampleAsync()continuously.
    What is the problem?
    I want to render RBGA raw data on silverlight MediaElement.
    Do you have any idea?
  9. Pete Brownsays:
    @Alex

    Double-check that you're not doing too much work and making it so Silverlight never gets a chance to paint the screen.

    Try returning a screen full of a single color (randomized for each frame). That way you can see if the pixels are getting pushed.

    In the C64 app, I have to do all the screen generation on a separate thread, and then just push final pixel colors to the MSS which is running on a different thread.

    Pete
  10. kiransays:
    Dear Pete Brown,

    I have tried the same way which you have suggested above for rendering the raw video data (YV12).
    But i couldn't get call to GetSampleAsync() API

    Please let me know any more things that need to be done for registering in OpenMediaAsync() API

    Kiran.L
  11. Muhammad Azharsays:
    hello

    i am have one marker at 0 sec , but sometime it is'nt fire, i am using custom media stream by your source code.i have added mulitple marks in markes collection of media element. but at 0 sec does not fire.
    can you tell why?
  12. RAMsays:
    I have written sample.to render YUV data .I Gave data to GetSampleAsync() but only green frame is displayed.Please let me know any more things that need to be done to dispaly the orginal frame.
  13. Johnsays:
    Hello,

    I get an error on WaveFormatEx type. The code that defines WaveFormatEx type is not in the download code here. Can you please guide me to that code?

    BTW: Do you have any code that builds MediaStreamSource from an MPEG-2 Transport Stream MP4 H264 video file?

    John
  14. Petesays:
    @munch

    Fixed. My old domain isn't properly redirecting, and I haven't yet had a chance to figure out why. If you ever run across a 404 link on my site and that link starts with www.irritatedvowel.com, you can swap in 10rem.net and it will almost certainly work.

    Pete

Comment on this Post

Remember me

11 trackbacks for “Silverlight 3 – Creating Video from Raw Bits using a MediaStreamSource”

  1. Community Blogssays:
    For those of you watching the agenda for MIX09, the announcement of Silverlight 3 probably didn’t
  2. POKE 53280,0: Pete Brown's Blogsays:
    Creating sound from raw bits is, believe it or not, slightly more involved than creating video from raw
  3. ASPInsiderssays:
    I had the pleasure of interviewing Pete Brown this last week and talking about the Silverlight 3 Commodore
  4. Microsoft Weblogssays:
    I had the pleasure of interviewing Pete Brown this last week and talking about the Silverlight 3 Commodore
  5. Scott Hanselmansays:
    I had the pleasure of interviewing Pete Brown this last week and talking about the Silverlight 3 Commodore
  6. VBandi's blogsays:
    In early May, I gave a talk about the new features in Silverlight 3. As I’ve started to gather material
  7. I Hate Linuxsays:
    Forward: I would advise not looking for potential meanings of the below post or infer potential new features