Wpf Over Websockets (WoW) – How to Pump WPF windows & animations to browsers over HTML5 Websockets And Canvas


ANOOP MADHUSUDANAN

Vote on HN

imageOver the weekend, I just put together a hack for pumping WPF windows over Websockets and rendering it using HTML5 canvas – also with basic mouse input support. For each user connecting to the server, a new window instance is created, and screen will be pumped over web sockets to the browser as base64 encoded images. These images are painted on an HTML5 canvas. The entire source code is here at codeplex – http://wow.codeplex.com

What does this mean?

Once HTML5 specification is final - it is almost certain that it’ll be the most widely adopted platform for delivering applications to end users. With Websockets in HTML5, we are going beyond the unidirectional, stateless http to a full duplex, high speed connection infrastructure between the server and the client. This will definitely bring up lot of interesting possibilities, including exciting and feature rich user experiences. WPF is already a widely adopted, feature rich desktop application development platform, and this hack is a quick POC to bring the WPF experience over the web to the end user using HTML5 web sockets and canvas.

You can see that I’m accessing my WPF application over multiple browsers in the below video. Just to clarify, it is rendered in the browser purely using HTML5 canvas – No Silverlight or Flash Smile

 

Rendering the screen in the browser

HTML5 draft specification includes Websockets - http://dev.w3.org/html5/websockets/ 

WebSocket is a technology providing for bi-directional, full-duplex communications channels, over a single Transmission Control Protocol (TCP) socket. It is designed to be implemented in web browsers and web servers but it can be used by any client or server application. The WebSocket API is being standardized by the W3C and the WebSocket protocol is being standardized by the IETF.

Here is a pretty minimized implementation of the page that opens a web socket, retrieve the image data, and draws the same on a canvas

var socket = new WebSocket("ws://localhost:8181/main");
socket.onopen = function () {

}

socket.onmessage = function (msg) {
	var canvas = document.getElementById("canvas");
	var ctx = canvas.getContext("2d");
	var img = new Image;
	img.onload = function () {
		ctx.clearRect(0, 0, canvas.width, canvas.height)
		ctx.drawImage(img, 0, 0); // Or at whatever offset you like
	};
	img.src = "data:image/gif;base64," + msg.data; ;
}

The server

I was planning to implement a quick socket server based on draft specifications – but I found that there are already a couple of .NET based implementations out there. One among them is Nugget - http://nugget.codeplex.com/ – I’m having a thin layer on top of Nugget for creating a window instance when ever a client socket connects to the server.
	var srv = new WebSocketServer(port, origin, "ws://" + server + ":" + port);
	srv.RegisterHandler<ElementSocket<T>>("/" + name);
	srv.Start();
	return srv;

Most of the logic required for creating window instances, and pumping them back to the connected client socket goes in my ElementSocket implementation.

Capturing User Inputs

Presently, the support for input devices (mouse, keyboard) etc is minimal. Only Mousebutton down and Mouse button up events are supported. To do this, we are capturing the mouse inputs using JQuery, and then pumping it back to the server.
            $("#canvas").mousedown(function (e) {
                var point = getXY(e);
                socket.send("mousedown;" + point.X + ";" + point.Y);
            });

            $("#canvas").mouseup(function (e) {
                var point = getXY(e);
                socket.send("mouseup;" + point.X + ";" + point.Y);
            });
On the server side, the incoming messages are used to simulate the events on the WPF window instance that corresponds to the socket, like
                var pnt=this.TranslatePoint(new Point(x, y), this);
                var res=VisualTreeHelper.HitTest(this, pnt);
            
                if (res != null && res.VisualHit != null && res.VisualHit is UIElement)
                {
                    SimulateEvent(res, action);
                }

Challenges

There are a number of challenges – Ideally we should push only the region copies once it is modified. As of now, I’m pushing the entire screen image (bad) as that’s pretty easy (This is a weekend hack, rememberWinking smile). Now, another challenge is proper user input simulation on the instance windows – I still need to find a good way to trigger user inputs to the specific windows, with out affecting the actual input devices. For example, when one user moves the mouse, the move message should be triggered to the corresponding server side window instance, with out affecting the actual mouse cursor. May need to have a look at the WM_ messages, not sure to what extend it can be applied on WPF top level windows.

Related Reads & Hacks

As I mentioned, the entire source code of this hack is there at http://wow.codeplex.com – Enjoy.

© 2012. All Rights Reserved. Amazedsaint.com