Welcome to Pete Brown's 10rem.net

First time here? If you are a developer or are interested in Microsoft tools and technology, please consider subscribing to the latest posts.

You may also be interested in my blog archives, the articles section, or some of my lab projects such as the C64 emulator written in Silverlight.

(hide this)

Lighting up on Windows 7 with Native Extensions for Microsoft Silverlight

Pete Brown - 14 March 2011

The Native Extensions for Microsoft Silverlight (NESL) is a library that helps bridge the gap between what Silverlight can do out of the box, and what WPF and the full .NET framework can do to integrate with Windows.

Here's the description from the code gallery page:

While Silverlight 4 supports accessing COM automation components from elevated trust OOB applications, many Windows platform features are currently not available through COM automation. This makes them inaccessible to such Silverlight OOB apps. Native Extensions for Microsoft Silverlight(NESL) is an effort to incrementally build a library of components that expose select Windows 7 features through COM automation, making them easily available to Silverlight 4 OOB applications running with elevated trust. The current version of NESL provides access to Windows 7 features like Sensors, Portable Devices, Speech, Taskbar and more. NESL is made up of a set of COM automation based runtime libraries, Silverlight wrapper libraries usable from Silverlight 4 OOB applications, sample applications with source, API documentation, and a developer's guide.

Hats off to Jit Ghosh for writing this awesome library! He also worked with some vendors to create some cool samples.

The samples included with the source code install include a cute gem of a game that includes a number of the features. You have built-in Silverlight features such as the TV screen showing an image from the webcam, as well as NESL features like the text-to-speech reading each word displayed at the bottom of the screen.

image

If you have a light sensor hooked up, the game will also respond to changes in that. (You may get errors if you don't have a light sensor wired up at runtime)

image

You can navigate around the rooms by either clicking on a room name, or by saying that name. It's using the inproc version of speech recognition, so there are no ugly on-screen speech tools.

image

Hey, there's a poster of me coughing on that wall :). In this room, you have to tell jack to go Right, then up, then right, then down, then right, then up. The recognition works really well.

There are other demo applications included with the install including a travel journal application, and several discrete function examples. Definitely check them out.

Installation

The runtime download is a standard MSI that installs all the native code COM servers that Silverlight relies on. This is primarily a large amount of compiled C++ code that interfaces with the Windows API.

As a developer, you'll run the installer yourself. You'll also need to reference the Silverlight wrapper DLLs (or include the source code) in your own project.

Behind the Scenes: How it Works

There are two main components to the NESL: the COM Automation Servers, and the Silverlight libraries that wrap all the IDispatch call ugliness and present a .NET-friendly interface. The COM automation servers are made up of a very large amount of C++ code (source currently not available, but Jit is looking into that) that is installed either ahead of time, or as part of the application first-run. If the latter is chosen, the user will need admin rights to run the installer as we'll see below.

Any serious API blog post needs one of these PPTX-created layer diagrams. Enjoy.

Most of the Windows APIs are available via p/invoke or native COM. However, with a few exceptions, they are not available to automation clients like Silverlight 4 and script. To bridge this gap, Jit wrote a metric ton of C++ code to surface those APIs as automation (IDispatch) servers.

At runtime, your application calls the friendly Silverlight wrappers, which in-turn call into the COM Automation servers which then call the native API. This is an approach you can use to surface just about anything you want from Windows or from applications on Windows, as long as you're willing to write the automation server code. The installation process below can even be adapted to your own application requirements.

End-user Installation

The NESL Silverlight source code includes an Installer class. This class is responsible for checking to see if the native runtime is installed on the machine. If it isn't, it will extract the MSI from your XAP and run it to install and register the COM extensions. Of course, the end-user will need admin rights to run the MSI. On the surface, that seems a little ugly, but if you want to have access to the OS services without using something like WPF, this is a small price to pay. The approach the extensions take is pretty slick.

Here's the installer code, since I found it interesting.

public static void InstallNESL(Uri NESLPackageUri,
bool Remote = true, bool ShowProgress = true)
{
dynamic Installer = AutomationFactory.CreateObject("WindowsInstaller.Installer");
Installer.UILevel = ShowProgress ? INSTALLUILEVEL_BASIC | INSTALLUILEVEL_HIDECANCEL : INSTALLUILEVEL_NONE;
string PackagePath = (Remote == true) ? NESLPackageUri.ToString() : ExtractNESLPackageFromXAP(NESLPackageUri);
try
{
Installer.InstallProduct(PackagePath, "ACTION=INSTALL");
}
catch
{
throw;
}
finally
{
System.IO.File.Delete(PackagePath);
}

return;
}
#endregion

#region NON PUBLIC API
private static readonly int INSTALLSTATE_DEFAULT = 5;
private static readonly int INSTALLUILEVEL_NONE = 2;
private static readonly int INSTALLUILEVEL_BASIC = 3;
private static readonly int INSTALLUILEVEL_HIDECANCEL = 32;
private static string ProductCode = "{D2C886C3-765D-40CA-8111-7D4D79AE2400}";

private static string ExtractNESLPackageFromXAP(Uri NESLPackageUri)
{
string PackageTempPath = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments),
Guid.NewGuid().ToString() + ".msi");

StreamResourceInfo sri = Application.GetResourceStream(NESLPackageUri);
if (sri != null)
{
FileStream fs = new FileStream(PackageTempPath, FileMode.Create, FileAccess.ReadWrite, FileShare.None);
BinaryReader br = new BinaryReader(sri.Stream);
fs.Write(br.ReadBytes((int)sri.Stream.Length), 0, (int)sri.Stream.Length);
fs.Close();
sri.Stream.Close();
}
else
throw new Exception(string.Format(Resources.MSG_ERROR_LIBRARYNOTFOUND, NESLPackageUri));

return PackageTempPath;
}
#endregion

Despite the all-caps for constants (Jit, your C++ is showing!) the code is pretty interesting. Silverlight devs, even if they don't want to use the NESL, can learn a few nice installation tricks here for other things they may need to install along with their apps.

When an end-user runs the app, it performs the install check (you control this) and if not installed, displays the minimal installer UI.

image

If the user has installation rights, it will proceed and install the COM automation servers, making them available for use from Silverlight. Again, if you want to use this in a more controlled environment, you can push the MSI down ahead of time and eliminate the at-runtime installation and admin requirements.

Integrating with the Task Bar

The task bar API is very simple. For many tasks, you use the TaskbarButton class. For example, to show and hide an overlay icon, the code looks like this:

private void ShowOverlayIcon_Click(object sender, RoutedEventArgs e)
{
using (BinaryReader br = new BinaryReader(
Application.GetResourceStream(
new Uri("Cloud20x20.png", UriKind.RelativeOrAbsolute)).Stream))
{
TaskbarButton.Current.SetOverlayIcon(
br.ReadBytes((int)br.BaseStream.Length), "Cloudy!",
ButtonImageDataType.PNG);
}
}

private void HideOverlayIcon_Click(object sender, RoutedEventArgs e)
{
// yes, passing null to clear it is hokey at best, but that's
// how the windows API works as well
TaskbarButton.Current.SetOverlayIcon(
null, string.Empty, ButtonImageDataType.PNG);
}

Note that that is a little different from the approach taken in .NET 4. I asked Jit about this, and he pointed out that he was mainly familiar with the native API, and so based the code on that. However, once the C++ Automation Servers are opened up and documented, it should be fairly easy to create a .NET 4-compatible version if you would like.

Another example is the taskbar progress bar. This is one of my favorite taskbar features in Windows 7. The code here uses the progress bar on the screen and on the taskbar button.

private void ShowTaskbarProgress_Click(object sender, RoutedEventArgs e)
{
TaskbarButton.Current.SetProgressState(TaskbarItemProgressState.Normal);

BackgroundWorker worker = new BackgroundWorker();
worker.DoWork += (s, ev) =>
{
for (int i = 0; i < 100; i += 10)
{
Thread.Sleep(1000);
((BackgroundWorker)s).ReportProgress(i);
}

};

worker.ProgressChanged += (s, ev) =>
{
ProgressBar.Value = ev.ProgressPercentage;
TaskbarButton.Current.SetProgressValue((ulong)ev.ProgressPercentage, 100);
};

worker.RunWorkerCompleted += (s, ev) =>
{
ProgressBar.Value = 100;
TaskbarButton.Current.SetProgressValue(100, 100);
};

worker.WorkerReportsProgress = true;
worker.RunWorkerAsync();

}

Very cool stuff. Another cool integration point is the Windows 7 Sensor API.

Using the Sensor API

I've written some WPF tutorials for the Windows 7 Sensor API in the past. The sensor API can be used for all sorts of inputs including accelerometers, light sensors, temperature sensors and much more. I've modified my 2 axis joystick example to work with the NESL version of the accelerometer sensor API.

public class Joystick
{
public Joystick()
{
}

private double _rawXValue;
public double RawXValue
{
get { return _rawXValue; }
private set { _rawXValue = value; }
}

private double _rawYValue;
public double RawYValue
{
get { return _rawYValue; }
private set { _rawYValue = value; }
}


public bool IsJoystickPointedNorth
{
get { return RoundValue(_rawYValue) > 0; }
}

public bool IsJoystickPointedSouth
{
get { return RoundValue(_rawYValue) < 0; }
}

public bool IsJoystickPointedWest
{
get { return RoundValue(_rawXValue) < 0; }
}

public bool IsJoystickPointedEast
{
get { return RoundValue(_rawXValue) > 0; }
}


private int RoundValue(double value)
{
return (int)Math.Round((double)value, 0);

// example making the joystick more sensitive
// return Math.Abs(value) > 0.25 ? 1 * Math.Sign(value) : 0;
}

private SensorManager _sensorManager;
private PropertyKey _propertyKeyAccelX = null;
private PropertyKey _propertyKeyAccelY = null;

public void Initialize()
{
try
{
_sensorManager = new SensorManager();

var sensorList =
_sensorManager.GetSensorsByType(
MotionSensorTypes.SENSOR_TYPE_ACCELEROMETER_3D);

if (sensorList != null)
{
Sensor accel = sensorList.FirstOrDefault();

_propertyKeyAccelX = new PropertyKey(MotionDataFields.Base,
MotionDataFields.SENSOR_DATA_TYPE_ACCELERATION_X_G);
_propertyKeyAccelY = new PropertyKey(MotionDataFields.Base,
MotionDataFields.SENSOR_DATA_TYPE_ACCELERATION_Y_G);

accel.SetProperties(new PropertyValue[] {
new PropertyValue(new PropertyKey(SensorProperties.Base,
SensorProperties.SENSOR_PROPERTY_CURRENT_REPORT_INTERVAL),
(uint)1000)});

accel.SensorDataUpdated += (s, e) =>
{
List<PropertyValue> gValues =
e.NewData.GetSensorValues(
new PropertyKey[] { _propertyKeyAccelX, _propertyKeyAccelY })
.ToList();

RawXValue = Convert.ToDouble(gValues[0].Value);
RawYValue = Convert.ToDouble(gValues[1].Value);
};

}
}
catch (Exception ex)
{
throw;
}
}
}

This simply rounds off some of the accelerometer values to treat it like a joystick. As far as the client is concerned, this is polling-based, so you call this from code in a timer to get updated values.

private void StartJoystick_Click(object sender, RoutedEventArgs e)
{
var joystick = new Joystick();

joystick.Initialize();

DispatcherTimer timer = new DispatcherTimer();
timer.Interval = TimeSpan.FromMilliseconds(200);
timer.Tick += (s, ea) =>
{
// show x and y positions
xValue.Text = joystick.RawXValue.ToString();
yValue.Text = joystick.RawYValue.ToString();

Brush ActiveBrush = (Brush)this.Resources["ActiveDirectionBrush"];
Brush InactiveBrush = (Brush)this.Resources["InctiveDirectionBrush"];

if (joystick.IsJoystickPointedSouth)
South.Fill = ActiveBrush;
else
South.Fill = InactiveBrush;

if (joystick.IsJoystickPointedWest)
West.Fill = ActiveBrush;
else
West.Fill = InactiveBrush;

if (joystick.IsJoystickPointedNorth)
North.Fill = ActiveBrush;
else
North.Fill = InactiveBrush;

if (joystick.IsJoystickPointedEast)
East.Fill = ActiveBrush;
else
East.Fill = InactiveBrush;
};

timer.Start();
}

Of course, it could be changed to be event-based as well. The UI is a 9-square grid that lights up based on the direction of the joystick. Note that for diagonal moves, both the side and the end will be lit up.

<navigation:Page.Resources>
<SolidColorBrush x:Key="ActiveDirectionBrush"
Color="Red" />
<SolidColorBrush x:Key="InactiveDirectionBrush"
Color="Gray" />
</navigation:Page.Resources>

<Grid x:Name="LayoutRoot">


<Grid.ColumnDefinitions>
<ColumnDefinition Width="*" />
<ColumnDefinition Width="*" />
<ColumnDefinition Width="*" />
</Grid.ColumnDefinitions>
<Grid.RowDefinitions>
<RowDefinition Height="*" />
<RowDefinition Height="*" />
<RowDefinition Height="*" />
</Grid.RowDefinitions>

<Rectangle x:Name="North" Grid.Row="0" Grid.Column="1"
Margin="20" />

<Rectangle x:Name="South" Grid.Row="2" Grid.Column="1"
Margin="20" />

<Rectangle x:Name="East" Grid.Row="1" Grid.Column="2"
Margin="20" />

<Rectangle x:Name="West" Grid.Row="1" Grid.Column="0"
Margin="20" />

<Grid Grid.Row="1"
Grid.Column="1">
<Grid.ColumnDefinitions>
<ColumnDefinition Width="*" />
<ColumnDefinition Width="2*" />
</Grid.ColumnDefinitions>
<Grid.RowDefinitions>
<RowDefinition Height="*" />
<RowDefinition Height="*" />
</Grid.RowDefinitions>

<TextBlock Text="X" Grid.Row="0" Grid.Column="0" />

<TextBlock Text="Y" Grid.Row="1" Grid.Column="0" />

<TextBlock x:Name="xValue" Text="0"
Grid.Row="0" Grid.Column="1" />

<TextBlock x:Name="yValue" Text="0"
Grid.Row="1" Grid.Column="1" />

</Grid>

<Button Content="Start Joystick"
Height="37"
HorizontalAlignment="Left"
Margin="40,31,0,0"
Name="StartJoystick"
VerticalAlignment="Top"
Width="119"
Click="StartJoystick_Click" />
</Grid>

That's it! This is a very cool library that will help you bridge what Silverlight can do out of the box, and what you'd usually need something like WPF to do. Now, if your application needs a ton of this type of stuff as its normal mode of operation, I still strongly recommend considering WPF.

Attached is the source code I'll build on at Silverlight Connections at the end of the month. It includes both the Silverlight NESL examples, as well as the WPF examples.

The code in this post was written using Native Extensions 2.0. Note that you'll need to download that and the Windows API Code pack (links are in the zip) in order to compile the whole solution. The archive was too large otherwise.

       

Source Code and Related Media

Download /media/74360/lightuponwindows7withsilverlightandwpf.zip
posted by Pete Brown on Monday, March 14, 2011
filed under:        

11 comments for “Lighting up on Windows 7 with Native Extensions for Microsoft Silverlight”

  1. Josh Einsteinsays:
    I'm sorry but this makes absolutely no sense. Why on earth would someone write a Silverlight application that not only requires out-of-browser elevated permissions and Windows... But also requires the end user to install a COM DLL as an administrator.

    "Small price to pay"...? I have a better idea. File - New Project - WPF Application, then set your ClickOnce settings and publish.

    Sorry to rant (and I'm not trying to dump on Jit's work as it is impressive) but this is exactly what I was afraid of when Silverlight added COM. Although I know you guys are probably going to nix WPF eventually, Silverlight isn't quite there yet to be considered an option for building Windows apps.
  2. Petesays:
    @Josh

    This will certainly eat into WPF a little. However, WPF offers a ton more than just the new Win7 APIs when it comes to integration.

    With this approach, and writing a bunch of C++, you can extend the sandbox to do pretty much anything. However, any smart development team is going to look at that and weigh it against their needs and against the productivity you can get with WPF. The effort curves head in opposite directions.

    Pete
  3. Paulussays:
    Dear Pete,

    A most interesting topic and impressive work by Jit Ghosh indeed.

    Beyond this, can we read your article "between the lines" as a hint to preferably use such tools - where appropriate and if sufficient for the application - to gradually switch our focus from WPF to Silverlight as a platform for desktop LOB applications?

    Thanks for your kind advice,

    Paulus
  4. Petesays:
    @Paulus

    Not really. If your app's primary needs are all met by Silverlight, and you only need to jump out to this for some non-critical items, or items that are a small percentage of your overall effort, then go for it.

    Writing a bunch of core logic in C++ is not something most biz app devs are going to want to do. Not to mention, the admin install is no better than the click-once route you'd go with WPF, and marginally more confusing to all involved.

    Each app is different, though. Use the technology that's going to get you to the finish line quickest with the best application.

    We're working on WPF v.next. Silverlight is catching up quickly, but at this point, can't easily do everything WPF can do. There are also some things that are easier in Silverlight (webcam and mic come to mind).

    Silverlight has more momentum for sure, but WPF isn't going away any time soon :)

    Pete
  5. Jit Ghoshsays:
    @Josh:
    (Pete – sorry for the long reply on your blog)
    Let me see if I can explain the rationale behind NESL a little bit. I am a field resource at Microsoft, and work largely with our media vertical i.e. media companies, broadcasters, movie studios, publishers, social networks, telcos etc. For a very large number of these customers, consumer facing applications that require a lightweight runtime, and are easily deployable over the web without needing any specialized infrastructure, is a large part of their strategy. Without going into the debate of why Silverlight and why not WPF (which I would largely like to stay away from – I personally love both technologies), the reality is that most of these customers have gravitated towards Silverlight wherever they have decided to use Microsoft technologies for these kinds of apps. Aside from the technical merits debate, a lot of other factors play into corporate technology decisions as I am sure you know – skillset (a large number of them are also building Windows Phone apps), perception of ease (Flash/ActionScript devs moving to Silverlight),the choice of development agencies(many of them do not retain in house dev staff – the agency they choose end up guiding them with what technology to use).
    Having said that NESL was built to respond to a need that these customers have voiced consistently, since the day OOB and elevated trust capabilities were introduced in Silverlight. The ability to build applications that can operate cross platform, but look and feel like a native Windows application with some additional capabilities when running on Windows is indeed a pretty desirable feature to many of them. With that in mind, let me add one assertion to the question of “To WPF or To Silverlight” - no matter which way one decides to go my advice to all is NOT to use NESL as a deciding factor in that decision. There are many reasons to pick one over the other – the way NESL should be positioned, is that if you do decide to use Silverlight, and you do desire to take advantage of some Windows features, then NESL affords you that convenience. But in and of itself it is not a reason to pick one over the other.
    As to LOB apps that run inside a controlled network with a controlled deployment environment, I do agree with you that WPF may well be the first environment you look at. But again (based on my experience at my customers), we do see a large number of customers evaluating Silverlight for LOB apps as well.
    In conclusion NESL is not a proactive effort to guide the community in one particular direction, it is a reactive measure to respond to a few capabilities that our customers have been consistently asking us for. And neither is it an attempt to pollute the Silverlight ethos to remain cross platform – the SL product team continues to carry on their very noble efforts to remain as cross platform possible.
    HTH,

    - Jit
  6. Stefanosays:
    Silverlight should be cross-platform and cross-browser solution..so why COM support ?...becouse Silverlight run-time (no WP7 run-time) will not be cross-platform but windows only.... and Silverlight 5 beta features like P/Invoke is the same... Silverlight OOB(on windows) is useless... instead use WPF...
    Silverlight roadmap still no clear...
  7. Sebastiansays:
    Totally agree to Stefano.

    This features more sounds like being political stuff (hey, see what you can do with Silverlight and Windows) than a clear roadmap:

    COM, P/Invoke and this Extension makes absolutly no sense and the worst thing of all is that some people think, this is a good idea and force you to implement some very ugly and buggy hacks that are hard to maintain.

    Finally u want to jump to WPF and ClickOnce. Silverlight has some well defined borders and you should formulate them instead of trying to open them.
  8. Jit Ghoshsays:
    @Sebastian, @Stefano:

    I am not sure I understand your reasoning. As far the extensions go - you are totally right in saying WPF/ClickOnce is an option - as I said before, you should absolutely consider WPF if that works for you, and NESL should not be a reason for you to pick Silverlight over WPF.

    What I do not understand is the resistance towards the existance of COM and the soon to come P/Invoke support in SL.

    The existence of these abilities in the SL runtime, does not make it any less cross platform IMHO. You can write an application that completely ignores these features if you do not need to use them and stay cross plat. You can also write an app that uses these features incrementally /conditionally when running on Windows, but still have a more than acceptable feature set when running on the Mac.

    Historically, a lot of cross plat runtimes have had the hooks to plug into the native platform that the app is running on. Adobe AIR has it using its NativeWindow support. Even something like Java has it using JNI. I do not think that having those capabilities made either of those runtimes any less cross plat - it is a concious choice available to the app developer to opt into those and consequently enrich the apps with OS integration, but no one is forced into it.

    Also you mention ugly and buggy hacks. Would be curious to know what kind of hacks you might be thinking about.

    - Jit

Comment on this Post

Remember me