Skip to main content

Wpf Over Websockets (WoW) – How to Pump WPF windows & animations to browsers over HTML5 Websockets And Canvas

imageOver the weekend, I just put together a hack for pumping WPF windows over Websockets and rendering it using HTML5 canvas – also with basic mouse input support. For each user connecting to the server, a new window instance is created, and screen will be pumped over web sockets to the browser as base64 encoded images. These images are painted on an HTML5 canvas. The entire source code is here at codeplex – http://wow.codeplex.com

What does this mean?

Once HTML5 specification is final - it is almost certain that it’ll be the most widely adopted platform for delivering applications to end users. With Websockets in HTML5, we are going beyond the unidirectional, stateless http to a full duplex, high speed connection infrastructure between the server and the client. This will definitely bring up lot of interesting possibilities, including exciting and feature rich user experiences. WPF is already a widely adopted, feature rich desktop application development platform, and this hack is a quick POC to bring the WPF experience over the web to the end user using HTML5 web sockets and canvas.

You can see that I’m accessing my WPF application over multiple browsers in the below video. Just to clarify, it is rendered in the browser purely using HTML5 canvas – No Silverlight or Flash Smile

 

Rendering the screen in the browser

HTML5 draft specification includes Websockets - http://dev.w3.org/html5/websockets/ 

WebSocket is a technology providing for bi-directional, full-duplex communications channels, over a single Transmission Control Protocol (TCP) socket. It is designed to be implemented in web browsers and web servers but it can be used by any client or server application. The WebSocket API is being standardized by the W3C and the WebSocket protocol is being standardized by the IETF.

Here is a pretty minimized implementation of the page that opens a web socket, retrieve the image data, and draws the same on a canvas

var socket = new WebSocket("ws://localhost:8181/main");
socket.onopen = function () {

}

socket.onmessage = function (msg) {
	var canvas = document.getElementById("canvas");
	var ctx = canvas.getContext("2d");
	var img = new Image;
	img.onload = function () {
		ctx.clearRect(0, 0, canvas.width, canvas.height)
		ctx.drawImage(img, 0, 0); // Or at whatever offset you like
	};
	img.src = "data:image/gif;base64," + msg.data; ;
}

The server

I was planning to implement a quick socket server based on draft specifications – but I found that there are already a couple of .NET based implementations out there. One among them is Nugget - http://nugget.codeplex.com/ – I’m having a thin layer on top of Nugget for creating a window instance when ever a client socket connects to the server.
	var srv = new WebSocketServer(port, origin, "ws://" + server + ":" + port);
	srv.RegisterHandler<ElementSocket<T>>("/" + name);
	srv.Start();
	return srv;

Most of the logic required for creating window instances, and pumping them back to the connected client socket goes in my ElementSocket implementation.

Capturing User Inputs

Presently, the support for input devices (mouse, keyboard) etc is minimal. Only Mousebutton down and Mouse button up events are supported. To do this, we are capturing the mouse inputs using JQuery, and then pumping it back to the server.
            $("#canvas").mousedown(function (e) {
                var point = getXY(e);
                socket.send("mousedown;" + point.X + ";" + point.Y);
            });

            $("#canvas").mouseup(function (e) {
                var point = getXY(e);
                socket.send("mouseup;" + point.X + ";" + point.Y);
            });
On the server side, the incoming messages are used to simulate the events on the WPF window instance that corresponds to the socket, like
                var pnt=this.TranslatePoint(new Point(x, y), this);
                var res=VisualTreeHelper.HitTest(this, pnt);
            
                if (res != null && res.VisualHit != null && res.VisualHit is UIElement)
                {
                    SimulateEvent(res, action);
                }

Challenges

There are a number of challenges – Ideally we should push only the region copies once it is modified. As of now, I’m pushing the entire screen image (bad) as that’s pretty easy (This is a weekend hack, rememberWinking smile). Now, another challenge is proper user input simulation on the instance windows – I still need to find a good way to trigger user inputs to the specific windows, with out affecting the actual input devices. For example, when one user moves the mouse, the move message should be triggered to the corresponding server side window instance, with out affecting the actual mouse cursor. May need to have a look at the WM_ messages, not sure to what extend it can be applied on WPF top level windows.

Related Reads & Hacks

As I mentioned, the entire source code of this hack is there at http://wow.codeplex.com – Enjoy.

Popular posts from this blog

Top 7 Coding Standards & Guideline Documents For C#/.NET Developers

Some time back, I collated a list of 7 Must Read, Free EBooks for .NET Developers, and a lot of people found it useful. So, I thought about putting together a list of Coding Standard guidelines/checklists for .NET /C# developers as well.As you may already know, it is easy to come up with a document - the key is in implementing these standards in your organization, through methods like internal trainings, Peer Reviews, Check in policies, Automated code review tools etc. You can have a look at FxCop and/or StyleCop for automating the review process to some extent, and can customize the rules based on your requirements.Anyway, here is a list of some good Coding Standard Documents. They are useful not just from a review perspective - going through these documents can definitely help you and me to iron out few hidden glitches we might have in the programming portion of our brain. So, here we go, the listing is not in any specific order.1 – IDesign C# Coding StandardsIDesign C# coding stand…

Hack Raspberry Pi – How To Build Apps In C#, WinForms and ASP.NET Using Mono In Pi

Recently I was doing a bit of R&D related to finding a viable, low cost platform for client nodes. Obviously, I came across Raspberry Pi, and found the same extremely interesting. Now, the missing piece of the puzzle was how to get going using C# and .NET in the Pi. C# is a great language, and there are a lot of C# developers out there in the wild who are interested in the Pi.In this article, I’ll just document my findings so far, and will explain how develop using C# leveraging Mono in a Raspberry Pi. Also, we’ll see how to write few minimal Windows Forms & ASP.NET applications in the Pie as well.Step 1: What is Raspberry Pi?Raspberry Pi is an ARM/Linux box for just ~ $30. It was introduced with a vision to teach basic computer science in schools. How ever, it got a lot of attention from hackers all around the world, as it is an awesome low cost platform to hack and experiment cool ideas as Pi is almost a full fledged computer.  More About R-Pi From Wikipedia.The Raspberry Pi

5 Awesome Learning Resources For Programmers (To help you and your kids to grow the geek neurons)

Happy New Year, this is my first post in 2012. I’ll be sharing few awesome learning resources I’ve bookmarked, and will be pointing out some specific computer/programming related courses I've found interesting from these resources.Also, thought about saving this blog post for my kids as well - instead of investing in these Child education schemes (though they are too small as of today, 2 years and 60 days respectively ). Anyway, personally my new year resolution is to see as much videos from this course collections (assuming I can find some free time in between my regular job && changing my babies diapers).1 – Khan AcademyAs I mentioned some time back, you and your kids are missing some thing huge if you havn’t heard about Khan Academy.  It is an awesome learning resource, especially if you want to re-visit your basics in Math, Science etc.With a library of over 2,600 videos covering everything from arithmetic to physics, finance, and history and 268 practice exercises, th…