Preprocessed Templates and DSLs

In this post I will explore a bit further how to use preprocessed text templates to generate code based on Domain Specific Language models.

The first thing you need (after installing VS2010 Beta, of course), is to download and install Microsoft Visual Studio 2010 DSL SDK Beta1, available here.

Step 1: Create a default Domain-Specific Language Designer

Select File | New | Project. Select Other Project Types | Extensibility and then Domain-Specific Language Designer. Use the default project name (eg. Language1). Click finish to create the project.

Step 2: Run the project and create a Preprocessed template

After the project is created, run it. A new instance of Visual Studio runs, opening the Debugging solution. Add a new Preprocessed Text Template named DslTextTemplate.tt. Copy the template code from the Language1Report.tt file. This just outputs the elements defined in the model.

Add new references to the project:

• Company.Language1.Dsl
• Microsoft.VisualStudio.Modeling.Sdk.10.0
• Microsoft.VisualStudio.TextTemplating.10.0
• Microsoft.VisualStudio.TextTemplating.VSHost.10.0
• WindowsBase (there was a strange error about INotifyCollectionChanged)

Build the project.

Step 3: Add the test template

Add a new Text Template named DslTextTemplateTest.tt. Add the template content:

<#@ template debug="false" hostspecific="false" language="C#" #>
<#@ output extension=".txt" #>
<#@ assembly name="C:\Users\MyUser\Documents\Visual Studio 10\Projects\Language1\Language1\Debugging\bin\Debug\Debugging.dll" #>
<#@ import namespace="Debugging" #>
<#
    DslTextTemplate t = new DslTextTemplate();
    t.Initialize();
    this.Write(t.TransformText());
#>

The important bit here is the call to Initialize(). The first time I ran the template without this line, there was a null reference error in the model. I went looking in the template generated class for any references to the model, and noticed that it is being loaded in the Initialize method:

this.examplemodelValue = Company.Language1.Language1SerializationHelper.Instance.LoadModel(serializationResult, this.Store, "C:\\Users\\MyUser\\Documents\\Visual Studio 10\\Projects\\Language1\\Language1\\Debugging" +
            "\\Sample.mydsl1", null, null, null);

The test template generates the expected output.

First impressions:

Preprocessed text templates seem an ideal tool to quickly create and deploy new code generators based on DSL models. However, there are a few things that don’t seem right. As you can see above, the path to the DSL model is hard coded in the Initialize method. As I usually say, Modeling and Code Generation are two separate concerns, so there should be a way to specify for which model I am generating code.

But there’s always a way, and the use of a partial class helps in working around this issue. Simply by creating another partial class with a new Initialize method with an argument specifying the model path:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;

namespace Debugging
{
    public partial class DslTextTemplate
    {
        public void Initialize(string modelPath)
        {
            this.AddDomainModel(typeof(Company.Language1.Language1DomainModel));

            base.Initialize();
            if ((this.Errors.HasErrors == false))
            {
                Microsoft.VisualStudio.Modeling.Transaction examplemodelTransaction = null;
                try
                {
                    Microsoft.VisualStudio.Modeling.SerializationResult serializationResult = new Microsoft.VisualStudio.Modeling.SerializationResult();
                    examplemodelTransaction = this.Store.TransactionManager.BeginTransaction("Load", true);
                    this.examplemodelValue = Company.Language1.Language1SerializationHelper.Instance.LoadModel(serializationResult, this.Store,
                        modelPath, null, null, null);
                    if (serializationResult.Failed)
                    {
                        throw new Microsoft.VisualStudio.Modeling.SerializationException(serializationResult);
                    }
                    examplemodelTransaction.Commit();
                }
                finally
                {
                    if ((examplemodelTransaction != null))
                    {
                        examplemodelTransaction.Dispose();
                    }
                }
            }
        }
    }
}

The call to Initialize now looks like this:

t.Initialize("C:\\Users\\MyUser\\Documents\\Visual Studio 10\\Projects\\Language1\\Language1\\Debugging\\Sample.mydsl1");

Conclusion

The use of partial classes helps in redefining the necessary behavior to make things work with Preprocessed templates. However, a more elegant solution should be available, such as exposing a model path property or an overload to Initialize that takes this parameter. The next step is suggesting this to the VS team .

Using Preprocessed Text Templates

In order to explore Preprocessed Text Templates a bit further, this post shows a small example using this new feature in a simple code generator. The basic ideia is to:

1. Create a “generic” code generation project that transforms a list of customers (the model) into a comma-separated text file, using preprocessed text templates.
2. Create a test project that enables viewing the transformation results.

Step 1: Create the Code Generator project

Run VS2010, select File | New | Project. Select Visual C# | Class Library and name the project CustomerCodeGenerator.

Step 2: Add the model class

Rename Class1.cs to Customer.cs. Add 3 public properties of type string: Name, Address and Phone.

namespace CustomerCodeGenerator
{
    public class Customer
    {
        public string Name
        {
            get;
            set;
        }

        public string Address
        {
            get;
            set;
        }

        public string Phone
        {
            get;
            set;
        }
    }
}

Step 3: Add the template

Add a new preprocessed text template. Name the file CustomerCSVGenerator.tt. Add a new class, name the file CustomerCSVGeneratorCustom.cs. Edit the file and set the class to partial and change it’s name to CustomerCSVGenerator. This is where the model will be set for the code generator – a list of customers.

using System.Collections.Generic;

namespace CustomerCodeGenerator
{
    public partial class CustomerCSVGenerator
    {
        public List<Customer> Customers
        {
            get;
            set;
        }
    }
}

Step 4: Add the template code

The template code is quite simple, generate a single comma-separated line for each customer in the list.

<#@ template language="C#" #>
<#
foreach (Customer customer in this.Customers)
{
#>
<#= customer.Name #>;<#= customer.Address #>;<#= customer.Phone #>
<#
}
#>

Step 5: Add the test project

Select File | Add | New Project. Select Visual C# | Class Library and name the project CustomerCodeGeneratorTest. Add a reference to the CustomerCodeGenerator project.

What’s the best way of quickly viewing the transformation results? A Text Template, of course!

Add a new Text Template named CustomerCodeGeneratorTest.tt. We will use this template to call the transformation class and view the generated output.

<#@ template debug="false" hostspecific="false" language="C#" #>
<#@ assembly name="CustomerCodeGenerator.dll" #>
<#@ import namespace="System.Collections.Generic" #>
<#@ import namespace="CustomerCodeGenerator" #>
<#@ output extension=".txt" #>
<#
List<Customer> customers = new List<Customer>();

customers.Add(new Customer { Name = "Customer1", Address = "Some Street", Phone="123" });
customers.Add(new Customer { Name = "Customer2", Address = "Some Other Street", Phone = "456" });
customers.Add(new Customer { Name = "Customer3", Address = "Yet Another Street", Phone = "789" });

CustomerCSVGenerator generator = new CustomerCSVGenerator();
generator.Customers = customers;
#>
<#= generator.TransformText() #>

Generated output:

Customer1;Some Street;123
Customer2;Some Other Street;456
Customer3;Yet Another Street;789

Conclusions

This is something that could be achieved with standard templates, by using includes or some other mechanism. However, preprocessed text templates enable a simple and clear way to implement code generators.

1. The model can be anything. In this example, a Customer class defines a model that the template “knows”. We can easily read customers from a database, a XML file or some other source and reuse the same code generation class.
2. Having the template as a class means that it is not necessary to use TextTransform.exe to automate running the transformations. This way, creating your own transformation tool is quite simple.

The next step is to see how these templates integrate with DSL Models.

Cool in VS2010 Beta1: Preprocessed Text Templates

This is a very cool feature in Visual Studio 2010. You are now able to transform a Text Template into a partial class that can be easily integrated into another component and extended.
Basically, you get the transformation class that was compiled into the temporary assembly, with all the necessary dependencies. Being a partial class, it can be extended by writing a custom class with additional properties. The transformation process, as usual, is triggered by calling the TransformText() method.

There are a few questions I would like to see answered, and may very well be the subject of future posts:

1. As described in the previous post, a model can be anything. How does this help in creating templates that read some kind of model?
2. Does this mean we don’t need TextTransform.exe any more? Can I build my own transformation tool?
3. How is all this related to DSL Tools?

Resources (@ Pablo Galiano’s blog):

VS10 Beta 1 / T4 Preprocessing part 1
VS10 Beta 1 / T4 Preprocessing part 2

Managing menu entries in a DSL

One of the great features of the DSL Tools is the possibility of integrating custom entries in the designer context menu. However, it is not yet a quick and easy process to set up. I found how to do it in Sebastian Talamoni’s blog (found via this blog). When you need to add a new menu entry, you have to go through the same tedious process of adding the new command and identifiers to the Commands.vsct file, defining the constants in the CommandSet class and registering the command.

So I thought, there must be a way to do more with less effort. The proposed solution relies on these parts:

1. A model which specifies the available menu entries.
2. A template which generates the Commands.vsct file content based on the model.
3. A template which generates a partial CommandSet class, containing the command definition and registration.

Here’s how to get everything running:

Step 1: Create the DSL project.

Open VS2008, choose File | New Project, select new Domain-Specific Language Designer and click OK.

Select Next on each step, making sure a default DSL project is created.

Step 2: Add the menu entry model.

After the solution is created, go to the DslPackage project. Add a new folder named CustomCode, with a subfolder named CommandSet. Add a new file to this folder named CommandSetConfig.tt. Go to the file properties and remove the “Custom Tool” field value. We do not want to generate code here, this file will only contain our menu entry model. I used the example you can find in Sebastian’s post:

<#
string languageNamespace = "Company.Language1";
string languageName = "Language1";

// List of commands
Collection<Dictionary<string, string>> commands = new Collection<Dictionary<string, string>>();

// Command ImportDbSchema

Dictionary<string, string> command = new Dictionary<string, string>();
command.Add("CanonicalName", "cmdImportDbSchema");
command.Add("ButtonText", "Import from Database Schema");
command.Add("ToolTipText", "Use this option to create the domain from an existing Database Schema");
command.Add("Id", "{EC120F9A-9E7F-469d-8D61-F4E2A97E5725}");

commands.Add(command);

// Command ImportClasses

command = new Dictionary<string, string>();
command.Add("CanonicalName", "cmdImportClasses");
command.Add("ButtonText", "Import from Existing Classes");
command.Add("ToolTipText", "");
command.Add("Id", "{EC120F9A-9E7F-469d-8D61-F4E2A97E5726}");

commands.Add(command);

#>

Step 3: Generate the Commands.vsct file contents.

Add a new file to the CommandSet folder named Commands.tt. Edit the file and add this content:

<#@ template inherits=”Microsoft.VisualStudio.TextTemplating.VSHost.ModelingTextTransformation" debug="true"#>
<#@ output extension=".vsct" #>
<#@ import namespace = "System.Collections.Generic" #>
<#@ import namespace = "System.Collections.ObjectModel" #>
<#@ include file="CommandSetConfig.tt" #>
<?xml version="1.0? encoding="utf-8">
<CommandTable xmlns="http://schemas.microsoft.com/VisualStudio/2005-10-18/CommandTable" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<!– –>
<!– This file contains custom command definitions. –>
<!– –>
<!– NOTE: Each time commands are added or changed, the "version" parameter to the –>
<!– ProvideMenuResource attribute in Shell\Package.tt should be incremented. –>
<!– This causes Visual Studio to re-merge the menu definitions for the package. –>
<!– Otherwise, changes won’t take effect until the next time devenv /setup is run. –>
<!– –>
<Extern href="stdidcmd.h"/>
<Extern href="vsshlids.h"/>
<Extern href="msobtnid.h"/>
<Extern href="virtkeys.h"/>
<Extern href="DSLToolsCmdID.h"/>
<Include href="GeneratedCode\GeneratedVsct.vsct"/>
<Commands package="guidPkg">

<Buttons>
<#
foreach (Dictionary<string, string> element in commands)
{
#>
<Button guid="<#= element["CanonicalName"] #>GUID" id="<#= element["CanonicalName"] #>ID" priority="0×0902" type="Button">
<Parent guid="guidCmdSet" id="grpidContextMain" />
<Strings>
<CanonicalName><#= element["CanonicalName"] #></CanonicalName>
<ButtonText><#= element["ButtonText"] #></ButtonText>
<ToolTipText><#= element["ToolTipText"] #></ToolTipText>
</Strings>
</Button>

<#
}
#>
</Buttons>
</Commands>

<Symbols>

<#
int index = 810;
foreach (Dictionary<string, string> element in commands)
{
#>
<GuidSymbol name="<#= element["CanonicalName"] #>GUID" value="<#= element["Id"] #>">
<IDSymbol name="<#= element["CanonicalName"] #>ID" value="0x<#= index #>" />
</GuidSymbol>

<#
index++;
}
#>
</Symbols>
</CommandTable>

Step 4: Generate the CommandSet partial class.

Now, we will generate a partial class which defines the menu entry constants and registers the commands. Add a new file to the same folder named CommandSet.tt. Edit the file and add this content:

<#@ template inherits=”Microsoft.VisualStudio.TextTemplating.VSHost.ModelingTextTransformation" debug="true"#>
<#@ output extension=".cs" #>
<#@ import namespace = "System.Collections.Generic" #>
<#@ import namespace = "System.Collections.ObjectModel" #>
<#@ include file="CommandSetConfig.tt" #>
using System;
using System.Collections;
using System.Collections.Generic;
using System.Collections.ObjectModel;
using System.ComponentModel.Design;
using Microsoft.VisualStudio.Modeling;
using Microsoft.VisualStudio.Modeling.Shell;

namespace <#= languageNamespace #>
{
    /// <summary>
    /// Custom code for <#= languageName #>CommandSet.
    /// </summary>
    internal partial class <#= languageName #>CommandSet : <#= languageName #>CommandSetBase
    {
        #region Constants

<#
int index = 810;

foreach (Dictionary<string, string> element in commands)
{
    string canonicalName = element["CanonicalName"];

    string constName = canonicalName.Substring(0, 1).ToUpper() + canonicalName.Substring(1);
#>
        /// <summary>
        /// <#= constName #> commmand identifier.
        /// </summary>
        private const int <#= constName #>ID = 0x<#= index #>;

<#
    index++;
}
#>
        #endregion

        #region Members

<#
foreach (Dictionary<string, string> element in commands)
{
    string canonicalName = element["CanonicalName"];
#>
        /// <summary>
        /// Enable related task commmand unique identifier.
        /// </summary>
        private Guid <#= canonicalName #>GUID = new Guid("<#= element["Id"] #>");

<#
}
#>
        #endregion

        #region Protected Methods

        /// <summary>
        /// Provide the menu commands that this command set handles.
        /// </summary>
        /// <returns>A list of commands.</returns>
        protected override IList<MenuCommand> GetMenuCommands()
        {
            // Execute base

            IList<MenuCommand> commands = base.GetMenuCommands();

<#
foreach (Dictionary<string, string> element in commands)
{
    string canonicalName = element["CanonicalName"];

    string constName = canonicalName.Substring(0, 1).ToUpper() + canonicalName.Substring(1);
#>
            // Add the new menu command (Enable related task)

            DynamicStatusMenuCommand <#= canonicalName #> =
            new DynamicStatusMenuCommand(
            new EventHandler(this.OnPopUpMenuDisplayAction),
            new EventHandler(this.OnPopUpMenuClick),
            new CommandID(this.<#= canonicalName #>GUID, <#= constName #>ID));
            commands.Add(<#= canonicalName #>);

<#
}
#>
            // Result

            return commands;
        }

        #endregion
    }
}

Step 5: Complete the CommandSet partial class.

After saving the file, you should now have a few entries in the Error List. Now you need to complete the generated partial class with the variable part of the menu model. Add the file Language1CommandSet.cs file to the CustomCode folder and add the handlers for the menu visibility and click actions:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.ComponentModel.Design;

namespace Company.Language1
{
    /// <summary>
    /// Custom code for Language1CommandSet.
    /// </summary>
    internal partial class Language1CommandSet
    {

        #region Event Handlers

        /// <summary>
        /// Occurs when one of the menu items is being displayed.
        /// </summary>
        /// <param name=”sender”>Sender of this event.</param>
        /// <param name=”e”>Event arguments.</param>
        internal void OnPopUpMenuDisplayAction(object sender, EventArgs e)
        {
          // Control availability of the command

          MenuCommand command = sender as MenuCommand;

          if (command != null)
          {
              // Check what command it is

              if (command.CommandID.Guid.Equals(this.cmdImportDbSchemaGUID))
              {
                  command.Enabled = true;
                  command.Visible = true;
              }
              else if (command.CommandID.Guid.Equals(this.cmdImportClassesGUID))
              {
                  command.Enabled = true;
                  command.Visible = true;
              }
          }
        }

        /// <summary>
        /// Occurs when one of the menu items is clicked.
        /// </summary>
        /// <param name=”sender”>Sender of this event.</param>
        /// <param name=”e”>Event arguments.</param>
        internal void OnPopUpMenuClick(object sender, EventArgs e)
        {
          // Get the current menu

          MenuCommand command = sender as MenuCommand;
          if (command != null)
          {
              // Check what command it is

              if (command.CommandID.Guid.Equals(this.cmdImportDbSchemaGUID))
              {
                 this.ImportDbSchema();
              }
              else if (command.CommandID.Guid.Equals(this.cmdImportClassesGUID))
              {
                   this.ImportClasses();
              }
          }
        }

        #endregion

        #region Private Methods

        private void ImportDbSchema()
        {
            // TODO
        }

        private void ImportClasses()
        {
            // TODO
        }

        #endregion
    }
}

Your project should now look like this:

Step 6: Test it.

Transform All Templates. Copy the contents of the generated Commands.vsct file into the file with the same name located in the DslPackage project root.

Run the solution. Right-click the Designer surface. The two commands defined in the model are available:

Next steps:

1. Simplify and improve the menu entry model configuration. Maybe by creating an EntryDefinition class with a custom constructor.
2. Find a way to update the project Commands.vsct directly without having to copy the contents from one file to the other.