Dan Newcome, blog

I'm bringing cyber back

Archive for May 2010

Distributed issue tracking

leave a comment »

I was checking out one of Zed Shaw’s latest projects, MulletDB when I realized that the project used Fossil, a source control system that I had never heard of before.

As I checked around the Fossil site, I realized that this is just what I needed for a lot of my projects. I want to keep the development notes and track issues right with the project itself. This means that you have to have some kind of bug tracker or wiki set up so that it can be run from the working folder, and the database must be checked into version control. This possibly means that in order to commit changes to the bug tracker you might have to stop the server, which would be just one more step in the check-in dance.

I’m sure that Fossil is a nice SCM, and having been developed by the creator of Sqlite, it probably is a very thoughtfully designed piece of software. However, I’ve become invested in git, and with Github being the SCM hosting provider of choice I’d really like to see something like this that uses the git repository format on the back end so that everything can be distributed and versioned using git. Even better would be if Github would update its own issue tracker when you pushed issues using git.

- Update: I just found TicGit, which stores all of its data in a special branch of your existing git repository. Bugs Everywhere is another interesting project, although I don’t think it uses git.

Written by newcome

May 29, 2010 at 2:44 pm

Posted in Uncategorized

Developers as toolsmiths

leave a comment »

I was thinking the other day about software development when it occurred to me that as developers, we are primarily creating tools. Using tools to create tools, more specifically. There is a huge continuum of tools that we might be involved with – tools for ourselves, tools for our users, tools for generating other tools – you get the idea. Even if we are creating a website, presumably that is a very high-level tool for the intended audience to do their jobs better or to be entertained, to name a few uses.

If we develop the idea further, perhaps we are doing tool-driven development (unfortunately TDD is already taken)!

Written by newcome

May 29, 2010 at 1:01 pm

Posted in Uncategorized

Embedding IronJS

with 5 comments

Update: This post covers an obsolete version of IronJS. For details on embedding the new stuff I have a new article here.

I’m looking at alternatives to JScript for my latest project, Node.net. I cut the initial code using JScript because the level of integration with the .NET framework made it possible to keep the design very simple and let me write the entire thing in Javascript. Some commenters pointed out that v8 provides some ECMAScript5 things whose usage is encouraged in Node.js programs. So in order to write a conforming implementation, I’d really need to embed the v8 Javascript engine or trust that an alternative Javascript engine keep up with the v8 implementation of the emerging standards.

So, I made a first cut at implementing the tcp server by writing a .net assembly in managed c++ that embedded the v8 engine. I got this working, but it feels kind of unnatural since we require a lot of wrapping managed handles up into opaque containers for storage by v8 to be used as callbacks. Writing managed c++ isn’t a big deal, and even calling out to unmanaged c and c++ code isn’t that bad, but here we have the case where the unmanaged v8 engine has to call back into native .net code, which requires some trickery that is new to me, so I’m not sure yet how robust it will be. Also, I really want to avoid just re-inventing what the Node.js guys have done. Eventually the existing Node.js server will probably run on Windows. I’m more interested in having a simple and hackable implementation of the fantastic Node.js API available for managed runtimes. A similar approach could be used to create a Node server for the Java runtime.

However, I’m also looking into using another fully-managed .NET Javascript implementation. The two contenders seem to be Jint and IronJS. Jint has been around for a while and IronJS is new on the scene. IronJS looks like it will have a significant performance advantage over Jint, and being based on the Dynamic Language Runtime, its performance should improve with improvements to the underlying runtime.

With apologies to the creators of Jint, I’m going to bet on IronJS for the moment, as I don’t have time to try both of them out. Community buzz is good for IronJS, and it aims to follow in the tradition of other successful DLR languages like IronPython. I was kind of surprised at how early-stage the IronJS code was given the level of buzz about it. Apart from the Github repository the only website seems to be the author’s blog, on which I couldn’t find any general info about the project. However, despite the lack of documentation, the code is very clean and I was able to figure most things out on my own. So, I’m going to post up a few lines of code here since there doesn’t seem to be an embedder’s guide for IronJS like there is for v8.

I grabbed the IronJS code from Github:

> git clone http://github.com/fholm/IronJS.git

The bleeding-edge stuff didn’t build for me, so I reverted to the 0.1 branch:

> git checkout 0.1

Once we are on the correct branch the code should build with msbuild or Visual Studio.

I looked around for a repl, but the closest thing that is included is a console application called IronJS.Console that runs a few simple tests from a file and writes the output to standard output (This actually means we should be able to write a repl in Javascript and use this project to run it). Taking this as a sample, we can see what is required to load a Javascript file, evaluate it, and run it.

var context = Context.Create();
var astBuilder = new Compiler.AstGenerator();
var etGenerator = new Compiler.EtGenerator();
var astNodes = astBuilder.Build( "Testing.js", Encoding.UTF8 );
Action<Scope> compiled = etGenerator.Build( astNodes, context );

There are a few more details exposed here than we’d probably like such as having to create the abstract syntax tree as a separate step, but it is interesting to have access to the syntax tree prior to code generation. Presumably we’d be able to manipulate the code programmatically before generating the final code, which could allow some interesting metaprogramming possibilities.

You may recognize Action<T> as the built-in .NET delegate that can represent any method that takes a single argument of the given parameterized type and returns null. I changed the original variable declaration from using `var’ in order to show more explicitly what we end up with after compilation. Just like any other delegate, we can invoke it just like a .NET function. However we need to create a scope for the code to run in first. The scope provides the global variables that we’d like to make available to the executing Javascript code.

Creating the global scope looks like this:

var globals = Scope.CreateGlobal(context);

We have enough code now to run arbitrary Javascript code, but we don’t have any way to communicate between the Javascript code and the host environment. For the sake of this example, lets say that we’d like to be able to execute the following code, which will be provided in the file `Testing.js’:

print("foo");
print( y );

In order to provide a print() function so that we can print to the console we can do something like this:

Func<object, object> print = ( obj ) => { Console.WriteLine( JsTypeConverter.ToString( obj ) ); return null; };
globals.Global( "print", print );

So all we needed to do to provide a print() function to the Javascript scope was to bind a delegate to a function name using Scope.Global(). Variables can be added in a similar fashion, but value types don’t seem to be supported directly, so in order to supply an integer variable, we just box it to an object before registering it using Scope.Global().

object y = 0;
globals.Global( "y", y );

Now that we’ve built the scope with the global variables and functions we need, the delegate can be invoked with the scope as its parameter.

compiled( globals );

One of the critical things that I need in Node.net is the ability to reach into the Javascript context and get references to objects and functions that are defined in the script file. In IronJS this is accomplished using Scope.Pull(). So in our example, if `Testing.js’ defined

var x = 2

We could access this as:

globals.Pull( "x" );

That’s it for now. Happy hacking.

Written by newcome

May 25, 2010 at 10:55 am

Posted in Uncategorized

Node.net – Node.js implemented in Javascript on the .NET runtime

with 27 comments

- Update:

This has gotten more attention after I posted this and I haven’t gotten around to doing another post. I started working on a v8 port of this after the initial JScript version just to see if it was feasible after I got some comments suggesting using v8 instead of JScript. There is currently a really rough version of the Tcp server done using v8 under v8\ along with my other hackings trying to figure out v8 embedding. I pushed the code up as soon as I had something working the other day, and c++ isn’t my first language so it is pretty rough. However I solved many of the tricky bits such as storing managed handles so that they can be accessed later from js.

I’ve been using Node.js recently and I’m loving it. I’ve dabbled in server-side Javascript on and off over the last two years or so with Rhino and later with writing my own hack servers using JScript on the .NET framework, and Node is hands-down the best way to run Javascript on the server to come along so far.

I got the idea last weekend that it should be possible to use the Microsoft JScript language to implement an event-driven server in .NET similar to Node.js. Before anyone gets too uppity about Node.net not being a full implementation of Node.js, let me say that this was a weekend hack that ended up working out, so I extended it just enough so that it should make sense to someone who is familiar with Node.js.

Goals

  • Same theory of operation as Node.js (single threaded eventing front-end, non-blocking IO back-end)
  • API compatibility with Node.js
  • Written entirely in Javascript (JScript.NET)
  • Runs on the .NET runtime

What is implemented

  • Enough of the `http’ module to implement a basic server that handles post data and sends back response
  • Enough of the `net’ module to implement a TCP server that echoes requests to stdout
  • Enough of the `stream’ api to support the above
  • require()
  • sys.puts()

Limitations

  • HTTP requests are cached by .NET even though the stream is read async. This was a limitation of using HttpListener in .NET. The TCP server is fully streaming however.
  • Writes are implemented as blocking calls currently. I didn’t have time to implement write queues to enforce write ordering.

Download

A binary distribution is available here. Grab the source from here.

Usage

On Windows:

If you have the .NET 2.0 framework installed (likely if you are on a Windows box), just run the build script. If you’d rather build against a different framework version, you can alter the path to jsc.exe.

C:\> node.exe server.js

On Linux, under Mono:

C:\> mono node.exe server.js

In order to run under Mono, you’ll need Microsoft’s version of Microsoft.JScript.dll, which is included in the .NET framework. I can’t provide it here since it is not redistributable individually.

Examples

Running an HTTP server that prints post data to the console and replies with an `All finished’ message when the request completes:

var sys = require( 'sys' ), http = require( 'http' );
http.createServer( function( request, response ) {
	request.addListener( 'data', function( data ) {
		sys.puts( data );
	});
	request.addListener( 'end', function() {
		response.write( '<html><body><p>All finished!<p></body></html>' );
		response.end();
	});
}).listen( 9981, 'localhost' );

To test it out, try something like this:

C:\>curl http://localhost:9981 -d "hello"
<html><body><p>All finished!<p></body></html>

Running a TCP server that listens on port 9982 and writes data sent to the console:

var sys = require( 'sys' ), net = require( 'net' );
net.createServer( function( stream ) {
	stream.addListener( 'data', function( data ) {
		sys.puts( data );
	});
}).listen( 9982, 'localhost' );

To test out the TCP server try sending it some data using telnet:

C:\>telnet localhost 9982

Although I’m aiming to expose everything as an API that is compatible with Node.js, you can access the .NET framework from within the Javascript that runs under Node.net.
For example, the TCP server shown above could have used Console.WriteLine() in order to write its output to the console:

var net = require( 'net' );
net.createServer( function( stream ) {
	stream.addListener( 'data', function( data ) {
		System.Console.WriteLine( data );
	});
}).listen( 9982, 'localhost' );

The preceding samples are exactly the same as the code you would have written to accomplish the tasks in Node.js (with the exception of Console.WriteLine() of course ).

Building from source

Get the source from github here.

On Windows:

C:\> build.bat

On Linux

Mono’s mjs compiler doesn’t compile Node.net. Let me know if you get it to work.

Future work

Non-blocking writes is the biggest unimplemented piece of the architecture. After having gotten the locking semantics figured out for the main dispatch loop to avoid races (I think it is correct — if you are a threading guru I invite you to double-check my work), I didn’t feel like tackling writes. The current HTTP implementation isn’t streaming, and should be rewritten to use the `net’ module instead. This would entail writing an HTTP parser, which I wasn’t prepared to spend time doing. Also, most of the Node.js API remains unimplemented. I haven’t looked at the source code for Node.js much, but I’m wondering if some of its Javascript code could  be ported to work with Node.net.

Written by newcome

May 8, 2010 at 8:34 am

Posted in Uncategorized