Multiplayer-enabling my Windows Phone 7 game: Day 1

I’m building a game to enter in Red Gate’s Windows Phone 7 App competition. The first challenge I set myself was to build the core of the game in 3 days from a standing start. Then Red Gate moved the goal posts. So I’ve set myself a new challenge: make the game multi-player in 3 days, using Windows Azure for the server side. Here’s Day 1:

9:00 Started researching networking capabilities in WP7. MSDN has a useful summary. In short, HttpWebRequest and WebClient are supported, as are parts of WCF. One notable absence are Duplex Services – so it looks like I’ll be doing my own client side polling. ChannelFactory.CreateChannel is not supported, so service proxies cannot be generated dynamically - they have to be created using Add Service Reference, or slsvcutil on the command line. Since I’m not a great fan of all the boilerplate that the Add Service wizard spits out, I think I’ll give WCF on the client side a miss. This gives me the idea of creating a RESTful service that I can call using plain old HttpWebRequests.

9:16 Given the recent hoo-ha  over Windows Phone 7 data leaks, I think I should do my bit to save bytes and minimise data usage. I'Since I’m quite keen on Google’s Protocol Buffers format, I’m pleased to discovered that ProtoBuf.Net, Marc Gravell's implementation of the Google Protocol Buffers format supports WP7.

9:22 Starting work on a Windows Azure project for testing communication between client and server. It’s neat how the Cloud project in VS 2010 has a Windows Azure emulator.

10:38 Downloaded and built Microsoft's WCF Web APIs Preview 3. This extends the REST support that WCF has, as of .Net 4.0. Using this, and helpful blog post from Jesus Rodriguez, got a test service sending back responses using Protocol buffers. One thing that caught me out was failing to decorate my data contract classes with the [ProtoContract] and [ProtoMember] attributes. The serializer just returned an empty stream without that. Now to get the phone talking to the server.

11:05 An interruption: the boss arrives back in the office with an HTC HD7 for me to test the game on!

Signed up for a developer account to allow the phone to be unlocked for testing my apps. To my dismay, I discovered that I can't unlock the phone for deploying test apps until the publisher verification process has been completed. What?! I can understand why Microsoft would want to check that our company is who it claims it is before it lets us submit Apps to the market place; but to my own phone? I think I know who I am!

11:39 Encountered my first real problem: Windows Azure compute emulator only listens on the loopback address, 127.0.0.1; this means that the Windows Phone 7 emulator can't talk to it. Seems I’m not the first to have encountered this. I tried a number of solutions:

• Using NCat, a Port fowarding tool. No joy with this at all.
• TcpTrace, another kind of port forwarding tool, worked, but not when Fiddler was listening in.

13:00 Thanks to some suggestions from Phil Haack I discovered that Fiddler will do the job on its own (is there anything it can’t do?). Following these instructions, I set up Fiddler as a reverse proxy. This means my WP7 app can connect to MyMachine:8888, and Fiddler will shuttle the traffic to and from the Azure emulator.

13:10 Discovered RestSharp, a library that builds on top of HttpWebRequest to simplify making calls to RESTful services. In particular, it can deserialize the Response bytes into objects for you.

13:17 Trying to write a ProtoBuf deserializer for RestSharp, but Protobuf is throwing  MissingMethodException. It looks like I might have to use pre-built Serialization assemblies

14:40: Following Mark Gravell's post on using Protobuf-net with WP7, I mange to get deserialization of messages working on WP7 by pre-building a serialization assembly. I have to resort to subterfuge to get the resulting Serializer assembly added to my WP7 project: The Add Reference dialog refuses to do it, claiming that the assembly is not a bona-fide WP7 assembly, so have to add it to the .csproj file by hand.

15:00 In order to allow users to setup multiplayer games, I’m going to need to show some kind of UI. WP7 doesn’t allow you to mix and match Silverlight and XNA within one App. I don’t really fancy rolling my own UI library, so I start casting around for one. A helpful post on the XNA forums provides a list libraries for XNA in general, though only a couple of them support WP7.

15:35 Start investigating Nuclex UserInterface library, a UI library for XNA that claims support for WP7. Initially it looks quite promising: I can get it to render a couple of buttons. But I can’t make the buttons respond when I tap them.

17:15. I give up on Nuclex, and look for something else. The most promising looking framework is XPF, from Red Badger. It’s ambitious: it aims to replicate some of the core features of Silverlight for the XNA framework, including layout, data binding and animation. They have nightly builds available now, but they’re only valid for 30 days from the build date, and there’s no word yet on a release date, or indeed what the cost of a license will be. Do I take a chance on it?

What are your thoughts? Have I missed a framework that will save me hours? Get in touch: @samuel_d_jack on Twitter, or by email.

Getting started with Protocol Buffers – and introducing Protocol Buffers Workbench

So I’ve fallen out of love with Xml, and decided that Google Protocol Buffers is a better serialization format for a key (and voluminous) data structure in our application (as has Twitter, incidentally). For various reasons, I decided to write my own Reader and Writer rather than go with the existing implementations. The first thing I needed to do then, was to understand how the Protocol Buffers encoding works.

Google have published some very good documentation, both for the message definition language, and the encoding, but nothing beats seeing it in action. For that there’s a Protocol Buffers compiler, protoc.exe which, amongst other things, will handle encoding and decoding messages (get it from the Protocol Buffers website - choose “Protocol Buffers compiler – Windows Binary”). The only thing is, protoc.exe is a command-line tool, and not very friendly if you’re just wanting to fiddle around. So I’ve summoned all my WPF powers, and created a graphical wrapper around it. Allow me to introduce Protocol Buffers Workbench:

In brief

Using the Workbench is easy:

1. Install it using the ClickOnce installer.
2. Put your message definition in the first column, and, from the drop-down pick the message that will be at the root of your document
3. Enter your message in the second column, then hit the encode button (“>”). The encoded message is shown in the third column.

Or, if you have an encoded message, you can paste it into the third column and hit decode (“<”) to have the message translated into text format. The Paste button above the third column is particularly useful if you want to decode protocol buffers messages stored in binary columns in Sql Server. If you view such columns in Sql Management Studio they are displayed like “0x0A1B2C4D…”: the Paste button will recognise text in the clipboard in that format and paste it as binary data.

If you’re wondering about the “C#” toggle beneath the binary editor: that will encode the binary data as a C# byte array – particularly useful if you’re wanting to seed your unit tests with authentic protocol buffer bytes!

And that’s it. Unless you want to know more about defining message types, and writing messages in text message format – in which case, read on.

Message Types

The first thing to understand are the message types. Message types are a bit like an Xml Schema. But whereas you can understand Xml just fine without a Schema, having the message types to hand is essential when parsing a Protocol Buffers message. That's because, in a bid to squeeze the output as small as possible, all the wordy field identifiers and even field type information is stripped out. All that's left for each piece of data is a field number, and a few bits telling you how big it is.

Message types look like this:

enum Binding { 
   PAPERBACK = 1;
   HARDBACK = 2;
} 
 
message Author { 
   required string name = 1; 
   optional string location = 2; 
} 
 
message Book { 
   required string title = 1; 
   required Binding binding = 2; 
   repeated Author author = 3; 
   optional int32 number_of_pages = 4; 
   optional double weight = 5; 
}

So for each field in a message, you define if it must be present ("required"), it may be present ("optional"), or if it can appear any number of times, including never ("repeated"). You then give its type - and you have a range to choose from, including enums and your own custom message types. Finally, you assign it a field number: this is the critical part, as this is all that will distinguish between the different pieces of data in the resulting messages.

Messages in Text Message Format

It took me a bit of puzzling to work out how to write messages in text format. For some reason, though Google have defined a text format for Protocol Buffer Messages, they don't appear to have documented it any where.

Here's an example, using the message types defined above:

title: "The Holy Bible"
binding: HARDBACK
author: {
  name: "Moses"
  location: "Sinai Peninsula"
}
author: {
   name: "Paul"
   location: "Tarsus"
}
author: {
   name: "Luke"
}
number_of_pages: 1723
weight: 0.225

As you can see, it has some similarities to JSON, but not many. Field names aren’t enclosed in strings, for example, and they aren’t terminated by commas.

Move over Xml – Make Way for Protocol Buffers

It was in the summer of 2001 that I fell in love with Xml. I was writing an Excel Add-in, and needed to store some configuration data to disk. VBA grudgingly offered me Open, Print, Input, and the like to do the job: “Nested fields, huh? Do it yerself!”. Then along came Xml with her DOMDocument class, elements and attributes. “So you’d like nested fields? Sure – just put one element inside another”.

I was smitten.I would run my program, then call colleagues over to look at the output in notepad. “Look, it’s human-readable!”, I’d say, and so obvious was my infatuation that my kind-hearted co-workers never asked, “Yes – but what kind of human would bother?”

Fast forward 9 years, and I’m over that crush. Xml and I are now just good friends. I still see her every couple of weeks, but now I have eyes for other message formats. Then I saw only pros. Now I can concede the cons.

Way back in 2004 Dare Obasanjo, on the Xml team at Microsoft, wrote the Xml Litmus test to help developers check that they’re using Xml appropriately. His criteria:

1. There is a need to interoperate across multiple software platforms.
2. One or more of the off-the-shelf tools for dealing with XML can be leveraged when producing or consuming the data.
3. Parsing performance is not critical.
4. The content is not primarily binary content, such as a music or image file.
5. The content does not contain control characters that aren't carriage return, line feed, or tab because they are illegal in XML.

Dip this litmus strip in your scenario, and see how may of those five points turn green. If you score less than 4 out of 5, then Dare suggests you look elsewhere for your serialization format.

In the application I’m currently working on, I fell into the trap of using Xml just because it’s there. I have a data structure that I need to store in a single column in the database, and I picked Xml as the format. Problem is, most of what our application does revolves around this data structure, and it ends up writing, storing and parsing an awful lot of it.

As our databases ballooned in size, it has become clear to me that Dare needs to add another point to his list:

1. Document size is not critical

There’s no getting away from it: Xml is verbose. And since my application is the only one that’s going to be using this data, and I don’t need to query it with XPath or validate it with Xml Schema or transform it with XSLT, and I do need to get at it fast, I’ve decided verbosity isn’t a price I’m willing to pay any longer. Xml, with regret, you’re fired! (at least for this part of the application).

Xml Alternatives

So what does one use when one does not want to use Xml? Being with Xml for so long has made me wary of proprietary binary formats, so I don’t want to go down the route of inventing my own.  I’m loathe to bring into the world further chunks of bytes that can only be interpreted with source code to hand.

So I’ve been investigating the state of the art.

An obvious first candidate was Binary Xml. But it has a major black mark against it in that that nobody can agree what Binary Xml should look like. Despite years of committee meetings, there is still no officially sanctioned or widely adopted Binary Xml specification. Lots of organisations have had a shot at it, including Microsoft who in invented their own with the specific needs of WCF in mind. Nicholas Allen, a WCF architect, has the details for that on his blog. 

JSON is the current darling of the Web 2.0 community, and BSON is its binary equivalent. James Newton-King has created a .Net implementation. I didn’t choose BSON for one simple reason: like JSON and Xml, it is self-describing with field names embedded in the message. This is a good thing if you want interoperability, but costs on compactness.

The last two widely adopted formats I found are Thrift and Protocol Buffers. These are both quite similar in their aims and accomplishments (not surprisingly, I understand, because Thrift was  developed by engineers at Facebook who liked Protocol Buffers when they’d worked with it while at Google and so reinvented it). Having established that much, and determined that Protocol Buffers is twice as popular as Thrift (currently 153 questions for Protocol Buffers vs 75 for Thrift on StackOverflow, so don’t argue!) I decided to dig into Protocol Buffers.

I like it.

There’s a little language you use for describing the structure of your messages – and of course, nested messages are supported. Then there’s a compiler that turns your message definitions into code in your language of choice – code for serializing and deserializing binary messages into objects matching your definitions.

There’s very clear documentation of the language and the encoding on the Protocol Buffers website. And if you want to try it out in .Net, there’s not one, but two rival implementations. In the red corner we have Protobuf.net by Marc Gravell, and in the blue corner DotNet-ProtoBufs, written by Jon Skeet (in a couple of idle minutes he had when Stack Overflow was down, no doubt).

I’ve spent the last week or so implementing enough of the Protocol Buffers spec for my needs, and I’ve got some very pleasing results. Storing my data in a Protocol Buffers format uses 45% of the space that SQL Server’s Xml data type requires, and it loads and deserializes in less than half the time.

Stick around. I’ve got a few Protocol Buffers tips to share.