Dan Newcome, blog

I'm bringing cyber back

Archive for July 2010

Io on .NET

leave a comment »

No, I didn’t mis-title this post. I’m not talking about I/O, I’m talking about Io. Io is a dynamic programming language whose message-oriented syntax shares some similarity with Smalltalk and whose prototypal object system is similar to that of Javascript. I’m thinking a lot about metaprogramming these days, trying to encode functionality as data as much as possible for some very configurable CRM products.

In the past I’ve done metaprogramming for configurable applications using code generation techniques. The technologies I used were a mix of XSLT and T4 templates. However, I learned that generating code this way is very cumbersome and requires that you add an extra compilation step in addition to compiling the base libraries for your application framework. At the end of the project I had the feeling that using a dynamic language would have been a much better bet than trying to generate C# and compile on the fly.

So what does this all have to do with .NET or Io anyway? Well, having done some work with JScript.NET, I’m now watching out for dynamic languages on the .NET runtime that would be suitable for doing metaprogramming. I checked out Clojure-clr recently and (from the same author) DotLisp a long time ago. I just ran across a .NET implementation of Io during a google search while reading this article by _why on manipulating code as data.

I’ll write up another post when I get a chance to play around with things some more, but in the meantime, I’ll show you that we have full access to the .NET runtime from the Io REPL:

Io> MessageBox Show("Hello")

Written by newcome

July 22, 2010 at 8:45 pm

Posted in Uncategorized

Limitations of C# dynamic

with 5 comments

I’ve been using the new dynamic keyword along with anonymous types recently, but I think that these features kind of miss the mark. Anonymous types aren’t dynamic at all — they just define the type implicitly — you can’t add new items at runtime. However you can use the DLR’s ExpandoObject if you want to be able to add items at runtime. In either case the most serious limitation is that using dynamic doesn’t let you access members that are specified at runtime.

For example, this is not valid:

dynamic obj = GetMyObj();
string field = GetMyField();
object value = obj[ field ];

furthermore, trying to access a member that doesn’t exist throws an exception:

dynamic obj = new { field1 = "foo" };
// throws RuntimeBinderException
string val = obj.field2;

Since we can’t specify fields dynamically at runtime, it is not possible to create some function that will safely access an arbitrary member and return null if it doesn’t exist. In Javascript and other dynamic languages, it is idiomatic to say things like the following:

var foo = obj.field1 || "Default";

We know that accessing field1 will return undefined so we can assign a default value with a simple assertion. At the very least we’d like to be able to do:

object foo = DynamicFieldExists( obj, "field1" ) ? GetValue( obj, "field1" ) : "Default";

However since we can’t access dynamic fields using indexer notation, I don’t know how we would write the DynamicFieldExists and GetValue functions needed to wrap the access up into a try/catch to avoid throwing an exception if the field doesn’t exist. Here is what I’d like to write:

public static bool DynamicFieldExists( dynamic obj, string field ) {
    bool retval = false;        
    try {
        // can't write the following:
        var temp = obj[ field ];
        retval = true;
    }
    catch( RuntimeBinderException ) {}        
    return retval;
}

If anyone has any thoughts on this I’d love to know.

Written by newcome

July 22, 2010 at 10:56 am

Posted in Uncategorized

Refactoring to domain-specific languages

with one comment

Today I was adding new functionality to a library of Microsoft CRM functions that I’ve been building up to make performing queries more manageable, when I realized that I was duplicating a lot of code between different query types. For example, we might want to construct a query that has a simple filter on it that compares an entity field to a value. For example, the code might look something like this:

public static QueryBase FilterByField( string in_entityName, string in_fieldName, object in_fieldValue, ColumnSetBase in_columns, IEnumerable<String> in_orderfields ) {
	QueryExpression queryExpression = new QueryExpression();
	queryExpression.EntityName = in_entityName;
	queryExpression.ColumnSet = in_columns;

	FilterExpression filterExpression = new FilterExpression();
	filterExpression.FilterOperator = LogicalOperator.And;

        .....

	foreach( String orderfield in in_orderfields ) {
		if( ( orderfield != null ) && ( orderfield != "" ) ) {
			queryExpression.AddOrder( orderfield, OrderType.Ascending );
		}
	}
	return queryExpression;
}
	

In another query type where we want to do a join, we might have something like this:

public static QueryBase GetLinked( 
	string in_fromEntity, string in_fromField, string in_toEntity, string in_toField, 
		string in_filterField, object in_filterValue
	) {
	LinkEntity linkEntity = new LinkEntity();
	...
        //  create a query expression
	QueryExpression query = new QueryExpression( in_fromEntity );
        ...

        // here we add a filter
        linkEntity.LinkCriteria.FilterOperator = LogicalOperator.And;
        ....

    return query;
}

Looking closely, this code is not very DRY. Each of these query construction methods has to create an instance of QueryExpression, set up any filters, and the second function doesn’t even support ordering like FilterByField does. If we wanted to add ordering we’d have to add it by duplicating the code in FilterById and probably create a new overloaded function to support callers that expect the old function signature.

An obvious solution to the duplication of code in this example is to split things up by query clause. We’d end up with functions like AddLink() and AddFilter(). Most likely, the function signatures would look like this:

AddLink( QueryBase, ... );
AddFilter( QueryBase, ... );

In each case we pass the query that we are building into each method along with any other parameters needed to build the particular clause. As I began this refactoring, I realized that this was a perfect time to use method chaining to build up the query by calling successive methods on the same query object. By modifying each method to return the same instance of the query that was passed in, we can write something like this:

AddFilter( AddLink( new QueryExpression(), ... ), ... );

Not quite what we want, but let’s wrap the QueryExpression up in a new class. If we return the wrapper instance from each method, and the wrapper class has the functions defined on it, we can chain the calls like this:

new QueryWrapper().AddLink( ... ).AddFilter( ... );

At this point I was thinking that by naming the methods by their SQL counterparts, I could create a small internal domain-specific language out of this. An example would look like this:

new QueryWrapper().Join( ... ).Where( ... );

About 20 minutes later, after rearranging and renaming the code, I was able to write a query like this using the refactored code:

QueryBase query = QueryDsl
	.Select( new AllColumns() )
	.From( "childentity" )
	.Join( "childentity", "childentityid", 
		"parententity", "parententityid" )
	.Where( "parententity", "name", 
		ConditionOperator.Equal, new string[] { "myparent" } ).Query;

This is a huge improvement in flexibility while still remaining readable and understandable. Previously I would have had a separate query builder for each complex query like the one above.

To give you an idea of how this was implemented, I’m including the first basic revision of the code below.

using System;
using System.Collections;
using Microsoft.Crm.Sdk.Query;

namespace MSCrm.Query
{
	/**
	* QueryDsl is an experimental domain-specific language for building
	* Microsoft CRM QueryExpressions.
	*/
	public class QueryDsl
	{
		// We use Select() function instead of constructor
		private QueryDsl() { }

		/**
		 * QueryDsl wraps a CRM QueryExpression. Idiomatic usage chains calls 
		 * together, only accessing the Query as the last call in the chain.
		 */
		public QueryExpression Query {
			get { return m_query; }
		} 
		private QueryExpression m_query;
		
		/**
		 * Select serves as the constructor and the start of the 
		 * chain. By Sql convention, accepts the projection list
		 */
		public static QueryDsl Select( ColumnSetBase in_columns ) {
			QueryExpression query = new QueryExpression();
			query.ColumnSet = in_columns;
			QueryDsl dsl = new QueryDsl();
			dsl.m_query = query;
			return dsl;
		}

		/**
		 * From sets the entity type that the query will return
		 */
		public QueryDsl From( string in_entityName ) {
			m_query.EntityName = in_entityName;
			return this;
		}
		
		/**
		 * Join uses LinkEntity to establish a relation between two entities.
		 * Use Where to add criteria using the 'to' entity.
		 */
		public QueryDsl Join( string in_fromEntity, string in_fromField, string in_toEntity, string in_toField ) {
			LinkEntity linkEntity = new LinkEntity();
			linkEntity.LinkFromEntityName = in_fromEntity;
			linkEntity.LinkFromAttributeName = in_fromField;
			linkEntity.LinkToEntityName = in_toEntity;
			linkEntity.LinkToAttributeName = in_toField;
			linkEntity.JoinOperator = JoinOperator.Inner;
			
			// TODO: we only support joins against the entity defined in the
			// root query - should write support for nested LinkEntities
			m_query.LinkEntities.Add( linkEntity );
			return this;
		}

		public QueryDsl Where( string in_entity, string in_field, ConditionOperator in_operator, object[] in_values ) {
			FilterExpression filterExpression = new FilterExpression();
			filterExpression.FilterOperator = LogicalOperator.And;

			ConditionExpression ce = new ConditionExpression();
			ce.AttributeName = in_field;
			ce.Operator = in_operator;
			ce.Values = in_values;

			filterExpression.Conditions.Add( ce );

			// TODO: we overwrite any existing filter - we might 
			// want to do some logic to append instead of replace
			if( m_query.EntityName == in_entity ) {
				m_query.Criteria = filterExpression;
			}
			else {
				LinkEntity link = FindEntityLink( m_query.LinkEntities, in_entity );
				if( link != null ) {
					link.LinkCriteria = filterExpression;
				}
			}
			return this;
		}

		/**
		 * Used by Where to figure out which LinkEntity corresponds to the desired
		 * entity we wish to attach the criteria to
		 */
		private LinkEntity FindEntityLink( ArrayList in_linkEntities, string in_entityName ) {
			foreach( LinkEntity link in in_linkEntities ) {
				FindEntityLink( link.LinkEntities, in_entityName );
				if( link.LinkToEntityName == in_entityName ) {
					return link;
				}
			}
			return null;
		}

		/**
		 * OrderBy adds ordering fields to the query at the toplevel.
		 * 
		 * TODO: for full sql compliance we need to pass array of booleans
		 * since we can specify ascending/descending for each field
		 */
		public QueryDsl OrderBy( string[] in_orderfields, OrderType in_ordertype ) {
			foreach( String orderfield in in_orderfields ) {
				if( ( orderfield != null ) && ( orderfield != "" ) ) {
					m_query.AddOrder( orderfield, in_ordertype );
				}
			}
			return this;
		}

	} // class 
} // namespace

Written by newcome

July 13, 2010 at 1:41 am

Posted in Uncategorized

Stupid .NET serialization tricks

with 4 comments

I’ve been meaning to write up a few things about .NET serialization that I’ve come across recently. I have a basic set of maneuvers for getting the XML output to look like you need it to in the case that you are trying to use some .NET data structures as the basis for writing out some XML to be consumed by another service. However, I’m not going to cover that right now — rather, I’m going to illustrate a solution that I found to a potential show-stopper that I came across in some work for a client.

I needed a way to quickly feed some canned data to an application targeting Microsoft CRM. I was going to be tweaking the data a lot during the development of the solution, so I didn’t want to spend all day clicking around in the CRM customizations UI — I wanted to just edit a file. This would let me go between different versions of a customized entity also.

So I hacked up a dummy service that implemented ICrmService so that I could hand it off to my application without it knowing it was getting canned data. Microsoft dropped the ball by providing ICrmService, but not actually using it in their service implementation. This required another workaround, which I won’t go into here. Apart from this issue, the code took very little time to write, as all I did was take the business entities and throw them into a Dictionary<string, List> and pull them out when the application requested them.

The issue came when I wanted to persist the data structure to disk. The BusinessEntity and DynamicEntity instances were serializable, so I didn’t envision having difficulties just spinning the whole data structure to disk. For example, the following code worked just fine to serialize an instance of the CRM ‘contact’ entity:

XmlSerializer serializer = new XmlSerializer( typeof( contact ) );
serializer.Serialize( Console.OpenStandardOutput(), myContact );

So I started by just serializing my whole nested data structure type, Dictionary<string, List>

XmlSerializer serializer = new XmlSerializer( typeof( Dictionary<string, List<BusinessEntity>> ) );
serializer.Serialize( Console.OpenStandardOutput(), dataStructure );

Fist issue is that Dictionary is not serializable. I have run across this before, and somehow I seem to forget it each time. It doesn’t make much sense, since List is perfectly serializable. I have solved this in the past by subclassing the generic dictionary class. This time around I used an implementation that I found here.

The second issue was that types derived from the base types specified explicitly in the data structure type aren’t recognized by the default .NET serialization methods. I’m sure we could implement our own serializer, but that sounds like more trouble that it’s worth. In order to let the serializer know about other types that it may run into, we need to use the XmlInclude attribute on the base class. Everything derives from BusinessEntity, so we could fix things by adding the following attributes to the definition of BusinessEntity:

[XmlInclude( typeof( contact ) )]
[XmlInclude( typeof( subject ) )]
....

Unfortunately we can’t change BusinessEntity, since it is part of the CRM SDK. Even if we had the source, it would be ill-advised. The first thing I thought of was subclassing BusinessEntity like this:

	[XmlInclude( typeof( contact ) )]
	[XmlInclude( typeof( subject ) )]
	public class MockBusinessEntity : BusinessEntity
	{
	}

The problem with this is that in C# we can’t easily downcast and C++ style reinterpret_cast is completely out of the question. You’d probably have to disassemble everything to MSIL and hand tweak it. Even then, Anders Hejlsberg would probably come out from Redmond and punch you in the face. Or at the very least give you a noogie and a stern talking-to. Even if we managed to get the cast to work, I’d imagine that .NET serialization looks at the reflected runtime type of the object anyway, so it probably still wouldn’t work.

After some experimentation, I found that the XmlInclude attribute doesn’t have to appear in the inheritance hierarchy, just in the data structure itself. So Instead of using List I created a collection class called BusinessEntityList. Here is the code:

	[XmlInclude( typeof( contact ) )]
	[XmlInclude( typeof( subject ) )]
	public class BusinessEntityList : List<BusinessEntity>
	{
	}

Now our data structure type looks like this:

SerializableDictionary

And serialization works as expected

XmlSerializer serializer = new XmlSerializer( typeof( SerializableDictionary<string, BusinessEntityList>  ) );
serializer.Serialize( Console.OpenStandardOutput(), dataStructure );

One unfortunate limitation of this solution is that each type that we need to handle must be represented in an XmlInclude attribute. In CRM there are probably 100 different built-in types, so in order to handle them all, we’ll need to maintain a lot of attributes.

Update: I have since avoided this issue entirely by simply using BusinessEntityCollection in lieu of a List. This should have been blindingly obvious, but I didn’t think of it at the time I was dealing with this initially.

Written by newcome

July 8, 2010 at 2:07 pm

Posted in Uncategorized