Dan Newcome, blog

I'm bringing cyber back

Archive for February 2011

Consuming XML Web services in Backbone.js using Jath

with 13 comments

Update: This article is getting a lot of traffic lately so I want to draw attention to a discussion in the comments. Backbone no longer requires the patch that I have described below to work with XML. However the use of Jath to parse the results remains the same. Check out this article via the backlinks.

Over at Ubernote We recently released an initial version of our API for public consumption. Internally, all of our code uses SOAP/XML over the wire, so it made the most sense for us to release our public API using the same protocol. For some developers, SOAP is easy to deal with using proxy-generation tools — you’ll never have to deal with the data format at all, just native method calls in Java/C#/etc. We’ll probably be adding support for JSON natively at some point if there is sufficient developer interest in the API, but for now it is XML-only.

However, for HTML5 clients and some mobile application frameworks, JSON is the only way to do things. I’ve been playing with a the Underscore.js/Backbone.js/jQuery Mobile stack recently and I wanted to hook all that HTML5 goodness up to our newly-minted API.

Followers of my blog may recall a project that I created almost a year ago called Jath, which is an XML-to-JSON conversion library that works in the browser (and now in Node.js also). My plan to get all this working was to hook into the persistence layer of Backbone.js and convert the XML coming back from the Ubernote API to JSON before it hit the client-side data models.

Backbone.js was designed to be very flexible in dealing with data coming back from the server, so there is a hook for doing data manipulation before it hits the data models. However, it does not support fetching XML data out of the box. Since it uses jQuery under the hood the capability is there, it is just set for the common case which is JSON. To enable support for XML we only need to change two lines in the Backbone.js source. See the following code from the Backbone sources where we set dataType and contentType to their XML counterparts:

 Backbone.sync = function(method, model, options) {
    var type = methodMap[method];

    // Default JSON-request options.
    var params = _.extend({
      type:         type,
      contentType:  'application/xml',
      dataType:     'xml',
      processData:  false
    }, options);

Once this change is made we can call fetch() on a Backbone model and call an XML Web service.

Returning XML isn’t going to be useful unless we can convert it to JSON before it gets consumed by the models. We provide a function called ‘parse’ when creating a Backbone collection to perform the conversion. When data is fetched it will pass through this function on its way into the data models. The following code snippet creates a Backbone model for an Ubernote note and a collection to store a list of notes. It also performs the data conversion using Jath. This can all be done in something like 20 LOC since Jath templates are very concise:

			var Note = Backbone.Model.extend({});
			var NoteStore = Backbone.Collection.extend({
				model: Note,
				url: 'http://www.ubernote.com/API/V0_1/service.asmx/NotesGetList?sessionID=ABDF9A07-CB44-4231-8921-AC6D178C1F5F&sort=1&startRow=0&maxRows=10',
				parse: function( response ) { 
					var parsed = Jath.parse( 
						[ '//notes', { 	
							id: 'Note_ID', 
							title: 'Title', 
							body: 'Body',
							summary: 'Summary', 
							sourceType: 'Source_Type',
	 						createDT: 'Create_Dt',
							updateDT: 'Update_Dt',
							shareLevel: 'Share_Level', 
							noteType: 'Note_Type',
							star: 'Star',
							accessedDT: 'Accessed_Dt'
					} ], response );
					return parsed;
				}
			});

Note that I have the URL parameters including the session ID pre-configured, as I haven’t written a login page or any support functions for this code since it is intended to be an exploration/proof-of-concept only.

Once the model is set up all we have to do is create an instance of it and call fetch() to populate it:

var notes = new NoteStore();
notes.fetch();

Here is what the result looks like when the model is rendered using jQuery Mobile as the view UI:

I won’t go into the UI aspects of this but I’ll probably cover some other related topics in a future series of posts. But for now this should show you that it is possible and not too difficult to use the current crop of HTML5/Javascript frameworks with XML Web services.

Written by newcome

February 20, 2011 at 5:13 am

Posted in Uncategorized

Why aren’t there generic properties in C#?

with 3 comments

I ran across something interesting today about the C# generics language feature and its underlying CLR support.

While maintaining a library of helper functions I thought it would be nice to be able to index into a collection of mixed-type properties and cast the result using a type specified by a generic type parameter. The calling code would have looked something like this:

// not valid code
string val = collection<string>[ "key" ];

Seems simple enough, we can have generic fields in our classes, and generic methods, so it seems like generic properties and indexers would follow.

However, the difference is that a property is a wrapper around a backing store. With a generic type field, we cannot provide a type argument at any time other than the class instantiation obviously, since setting a field is a simple assignment.

The crucial difference is when the type is known. In the case of a generic field we know the type when the class is instantiated with the type argument.

For example:

public class Gen<T> {
    public T Field;
}

Gen<string> gen = new Gen<string>();
gen.Field = "foo";

So why can’t we have:

public class Gen {
    public T Field<T> { get; set; };
}

Gen gen = new Gen();
gen.Field<string> = "foo";

As I alluded to before, the reason has to do with the underlying implementation of the heap in the CLR. Since we don’t know the type when the class is instantiated, the CLR can’t allocate it properly on the heap since it doesn’t know how much space it needs.

The only explanation that I found is an old post by a member of the C# team here.

Thinking about this some more I think it should be possible technically if the type were constrained to reference types.

Using generic type parameter constraints we would write:

public class Gen {
    public T Field<T> where T : class  { get; set; };
}

As long as we are storing a reference type it shouldn’t matter what it is since the pointer is the same size.

Also, if we abused the getter/setter a bit and used a collection as the backing store, it seems like everything should work, but the compiler enforces the constraints based on the common use case of properties, which is to wrap private fields to fire events and do logging perhaps, not to substitute for methods that manage back end storage.

This is pure speculation, but it is an interesting thought exercise.

Written by newcome

February 18, 2011 at 7:56 pm

Posted in Uncategorized

Introducing Batbelt bootstrapper for Windows

with one comment

Superheroes always have their bag of tricks, able to pluck just the right tool for the job out of their bag. Unfortunately, this requires some suspension of disbelief and in real life we can’t have everything at hand all of the time. However, in the digital realm, can’t we have our cake and eat it too?

I have been tweaking a set of utility scripts over the last several months that would allow me to have the Batman-like ability to pull an endless assortment of utilities out of a small-footprint tool (much like the Batbelt – how does he fit all of those Batarangs in there?). So in that spirit, I have dubbed it the Batbelt. Unfortunately although it isn’t (yet) nearly as awesome as the real Batbelt, it is an ok start and has been useful to me so I thought it was time to clean it up and share it.

You can check it out on github here and the README follows.

About

Batbelt is a small toolkit for bootstrapping utilities on Windows machines. Batbelt is not a full package manager but has some things in common with tools like apt on Linux. It is implemented using simple Windows batch scripts and uses 7zip, pscp and curl for archiving and copying files.

Motivation

Batbelt was created out of a need for the me to have access to the many small tools that I download repeatedly from the Web every time I work on a new Windows machine in the course of my consulting work. Often times I need to leave the client’s machine clean as I found it afterward, so Batbelt was designed with zero footprint in mind.

For example, I would download Putty over and over again, and I would hesitate to put it in the path for fear that I would forget to delete it or remove it from the path when finished. Batbelt allows you to set up the tool once and download it transparently and on demand as it is needed. There is also only one path to worry about for all installed tools. This greatly simplifies use and cleanup.

Batbelt is a ‘bootstrapper’ meaning that once installed, other tools are available by simply typing the shortcut that would oridnarily be used to launch the tool. For example, typing ‘putty’ at the shell would lauch putty if it was available or download it from a repository and then launch it if it was not already available. This happens automatically, as not to interrupt your workflow.

Cleanup can be done by simply deleting the entire Batbelt folder when finished.

Concepts

At its core, Batbelt emulates the traditional Unix-style bin\ directory containing links or wrapper scripts to executables. Batbelt calls scripts in this folder ‘aliases’. Typically, installed packages will have a short alias that is easy to type. For example The fantastic Sysinternals Process Explorer might be shortened to just ‘pe’.

However, in addition to serving as an installation location for programs and short aliases, it is able to download programs on-demand if they aren’t available locally. This is accomplished using curl to download package archives from a web site configured as a remote package repository.

Repository

Batbelt is designed to use a simple http accessible repository for fetching new tools. If the tool is not installed locally, Batbelt will look at the configured remote repository to fetch the requested package.

Packages are simple zip files that can be created with any archiver or with the Batbelt utility scripts for deploying packages.

Setting up a repository

Create a web server directory that is readable using http and writable with sftp/scp.

Running deploy-batbelt.bat will create a self extracting archive of the main batbelt tool which includes the essential tools curl and 7zip. Batbelt requires these tools to bootstrap other programs so they are included in the base installation. Any further tools are set up using packages and aliases.

Deploying aliases is done using deploy-alias.bat. This will archive all alias files in alias\ and upload the archive to the repository using scp.

Adding a new program

Any program that can be installed via xcopy may be installed via Batbelt. Some Windows applications that provide an installer can be installed first and then the files copied for use with Batbelt. Not all applications can be installed this way however.

Two steps are necessary to add a new program to the repository – creating the package and creating an alias. To add a new package, arrange the files in a folder and give the folder the name of the desired package. Make sure that the executable is in the first level under the folder and not in a subfolder. Run deploy-tool.bat to install the tool to the remote repository.

To create a new alias, run the tool create-alias.bat giving the desired short name of the alias (this is what you will type to launch the program), the package name (this is what we named the package folder in the previous step) and the exectuable name of the program. This will create a new batch file in alias.

Deploying the alias is accomplished by redeploying all aliases. For simplicity and speed all aliases are kept in the same zip archive and are downloaded at once, thus, they are deployed at once as well. Simply run ‘deploy-alias.bat’ to deploy to the repository.

Installation

Getting Batbelt set up on a client requires downloading only a single file over the web. Use a web browser to navigate to the Batbelt repository on the web and download the self-extracting archive ‘Batbelt.exe’. Run the archive to unzip the contents of Batbelt to the current directory. Batbelt can be used directly from the location at which it was unpacked for a zero-footprint scenario or installed to a desired location on the system using ‘install.bat’, which will also add it to the user path. The second scenario is useful if you prefer convenience over zero-footprint, as it will modify the path.

Since the main Batbelt installer does not contain the aliases for configured programs we have to take one additional step to update the list of available programs. Run ‘get-aliases.bat’ to download aliases.zip and unpack it to the Batbelt client directory.

Notes

Batbelt uses 32 bit versions of curl and 7-zip for maximum compatibility. Many 64bit Windows systems are able to run 32bit programs transparently. However, later Windows Server versions don’t have WoW64 enabled by default, in which case 64bit versions of curl and 7-zip must be subsituted.

Further work

Although Batbelt is not intended to be a package manager, it could be made easier to use via some more tools for discovering packages and perhaps using multiple repositories. Currently it is designed to be as simple as possible using a single repository and supporting only simple ‘unzip’ installation semantics.

Written by newcome

February 6, 2011 at 12:27 pm

Posted in Uncategorized