Criteria

When considering these JavaScript build tools, I bring a certain perspective and set of expectations. I’m a .NET web developer who has been using Visual Studio for a long time. In the past I have worked in Java and C++ and used a number of IDEs. I have some experience with a wide range of build tools including make, msbuild, rake, nant, and psake. That experience has led me to prefer a polished tool with lots of convention over configuration. To me, a build tool exists so I can work on my project and should stay out of my way. I should have to spend a minimal amount of time configuring and debugging the build. Most IDEs are pretty good in that regard. You add a file and the IDE just does the right thing based on the file extension (one only occasionally needs to set a few options).

Being a professional web developer, I also expect that a web focused build tool will have lots of support for the complex asset pipelines of modern web apps. This is an area where Visual Studio definitely falls down. This is also an interesting and unique requirement for build tools. Traditional tools like “make” rarely had to deal with situations like this. Compiling a C++ app is mostly about finding all the source code, compiling and linking. There was rarely much more to it than that. For web applications, there are many different kinds of files that must each pass through their own long pipeline of transformations and combinations. Additionally, the result isn’t a single executable, but a large assemblage of files. That makes the need for good file clean-up much more important. Finally, developer expectations have risen since the original days of make tools. Then we were happy to get an executable when we ran the make command. Now, we expect file watching with incremental rebuilds and live browser reloads.

Here is a brief list of the steps/actions I expect will be present in most JavaScript web app builds. Of course, they will vary from project to project and by developer taste.

• Transcompiling JavaScript: CoffeeScript, Dart, Babel, Traceur etc.
• JavaScript Transforms: wrapping in modules or ng-annotate etc.
• Bundling/Concatenation: combining of scripts and styles into multiple files
• Minification: scripts, styles and html
• Source Maps: for both scripts and styles
• CSS Preprocessor: Less, Sass, Stylus etc.
• Style Transforms: Autoprefixer, PostCSS etc.
• Cache Busting: renaming files with a hash to prevent incorrect caching
• Image Optimization
• Compiling Templates: Mustache or HandlebarsJS, Underscore, EJS, Jade etc.
• Copying Assets: html, fav icon etc.
• Watching for Changes
• Incremental Rebuild
• Clean Build: deleting all files at start or preferably cleaning up files as needed
• Injecting References: generating script and style tags that reference the bundled files
• Build Configurations: separate Dev, Test and Prod configuration, for example to not minify html in dev build
• Serve: running a development web server
• Running Unit Tests
• Running JavaScript and CSS Linters: jshint, csslint etc.
• Dependencies: handle dependencies on npm and Bower packages, Browserfy etc.

Findings

So what did I find? There were some common failings in all the build tools. One was the presumption that npm and Bower package restore were outside the scope of build. Unlike with an IDE such as Visual Studio that automatically runs Nuget package restore as needed, none of the JS build tools attempts to. This was often impossible to work around due to the fact that the build tool required it be one of the npm dependencies of your project and would not run until it was restored. Several of the tools did have plug-ins supporting Bower package restore. However, almost all examples I saw assumed one would first run npm install then bower install before building. They didn’t even setup bower install as an action triggered by npm install. Another shortcoming of all the tools is the handling of dependencies. This isn’t entirely their fault since the JavaScript ecosystem is still trying to figure out the issue. Apparently, npm packages work fine in Node.js (though the reason for the deep paths and duplicate packages created in node_modules are beyond me). But front end packages are not nearly so well thought through. Unlike a code library, a front-end package has lots of different assets which may be in many different languages, each of which has its own way of referencing one another. Furthermore, depending on the project toolset, the build might need dependencies either before or after compilation. For example, some projects needs the less files of bootstrap, while others need the compiled css. So how is one to express and control how those assets are to be brought into the project? Bower isn’t the answer if you ask me (at least not yet). Referencing a front-end package needs to become as simple as referencing an assembly or Nuget package is in .NET. Until it is, setting up builds involving dependencies will always be a pain. With that understanding of the shared issues with JavaScript build tools, let’s now consider each tool in turn.

Grunt

Grunt seems to be the starting point for many people when it comes to JS Build Tools. It certainly appears to be the most widely used one. However, the GruntJS.com website proudly proclaims Grunt as “The JavaScript Task Runner” and I think that is a good description. Grunt almost doesn’t qualify as a build tool. It really is a system for running a sequence of commands/tasks. Each Grunt task is a plug-in that represents a tool one might run from the command line. Each plug-in defines what its configuration in the Gruntfile is named and what the configuration options are. The fixed section names mean one can’t organize or name the configuration in a way that makes sense to them. Though there are many commonalities in plug-in configuration, in practice you are forced to read the docs of each plug-in to have any idea how to configure it. That means the otherwise acceptable docs are marred by the often poor and cryptic documentation of individual plug-ins. Since each plug-in is essentially a command line tool, they each take a set of input files and a set of output files. To chain together the long asset pipelines commonly needed requires one to manage an ever expanding set of intermediate temp files. File watching and live reload are manually configured by specifying tasks to run when files matching file name patterns are modified. The server configuration options are still totally opaque to me. Incremental builds must be manually setup using grunt-newer. In practice, that means having a deep understanding of how each task makes use of its various input files and which of the cascading temp files will need to be rebuilt. All of this manual configuration leads to enormous, inscrutable configuration files. If you need to run a couple tasks, grunt will be fine for you. If you need a real web development build tool. I strongly suggest you look elsewhere.

Gulp

Gulp is the hot new tool. It is rapidly gaining in popularity and there are some justifiable reasons for this. It recognizes and attempts to handle the primary unique requirement of web build tools, namely the long pipelines. In Gulp all plug-ins work on streams (well as of today there are still a number that don’t, which is very annoying). This streaming model allows Gulp tasks to be chained together in streams without worrying about or even producing intermediate temporary files. Gulp touts its high performance due to both streams and parallel task running. Yet, many users have complained that the parallel task execution is confusing and there are plans to extend the configuration to give more control over this. Personally, I found the parallel task execution very intuitive and feel the proposed cure is worse than the problem.

While streams are a compelling base for a build tool, I ultimately found Gulp lacking. Since the core configuration options are minimal, the attitude seems to be that little documentation is needed, but I found the documentation inadequate. Like Grunt, it was necessary to rely on the usually poor documentation of the plug-ins. This was somewhat mitigated by the greater consistency in plug-in usage versus Grunt. The Gulp authors believe in opinionated software and have some strong opinions when it comes to Gulp plug-ins. Normally, I appreciate opinionated software, but in this case I have to disagree with the authors’ opinions. They believe that a tool that already supports streaming should not be wrapped in a plug-in, but used directly. This runs counter to making builds as easy as possible to setup because it requires one to learn and adapt the tools’ unique APIs to the Gulp system. For a peek into the mess this causes, check out the GitHub issue over the blacklisting of the gulp-browserify plug-in. In it, dozens of users attempt to figure out the correct way of using Browserify. I don’t think any quite hit upon the difficult to find official recipe. But note, even that fails to support the important case of streaming files into Browserify. The whole issue is complicated enough that it takes an excellent but lengthy blog post to fully explain. Furthermore, the streaming model becomes confusing for things that don’t fit into it, such as cleaning. Source maps in particular are odd. I attempted to combine external source maps with cache busting with a manifest file and cleaning up old revisions and while the final config was quite short, it took me several hours and additional plug-ins to get working. Watching still requires manual configuration and incremental building is quite confusing to setup. Figuring out how to do anything is made worse by the plethora of plug-ins for doing the same task. Finally, while I don’t understand the issue myself, I found many discussions of needing to do extra configuration to get reasonable error messages out of Gulp.

Broccoli

Broccoli.js is a newer build tool that has been in the works for a while. It doesn’t seem to have much traction yet, but hopefully its inclusion in the ember command line tool will change that. I really like that the developer has thought through most of the issues facing a JavaScript build tool. Support for watch and incremental build is built in so that Broccoli “will figure out by itself which files to watch, and only rebuild those that need rebuilding”. It achieves that through a sophisticated caching system built on top of its core abstraction of file trees. Trees are Broccoli’s answer to Gulp streams and are easily chained. To simplify the system, they don’t run tasks in parallel like Gulp, but claim that it isn’t necessary because performance is still good. I highly recommend you check out the author’s Comparison With Other Build Tools (scroll down to section 5).

Despite all these excellent attributes, Broccoli is still very immature and in beta. It is lacking in the area of documentation. The read me boldly states windows support is spotty. The fact that there is little to no source map support was a big issue for me. It seemed to me that Broccoli wasn’t quite ready for prime time, but I’d really like to see it mature and supplant Grunt and Gulp.

Brunch

Brunch has apparently been around for quite a while and yet flown under the radar. Brunch relies heavily on convention over configuration and assumes your build pipeline will include most of the steps I laid out above. When you include a plug-in, it is assumed to apply at the logical point in the pipeline and requires little or even no configuration. This means that a config that might be 600+ lines in Grunt or 150+ lines in Gulp could end up being 20 lines in Brunch. Brunch is written in CoffeeScript and all the examples are in CoffeeScript, but it isn’t too hard to translate to JavaScript. It natively supports watching, and incremental compilation with no extra config whatsoever. There is direct support for Development vs Production builds with different configurations (other build configurations are easily setup). It even wraps JavaScript modules for you automatically (configurable).

With all that, you might think we had found our answer to JavaScript builds. However, there are always problems. Despite the lengthy Guide and the Docs, I still found myself wishing for clearer explanations and more examples. I was surprised to find no support for clean builds baked in. Hopefully, this will be addressed due to the issue I filed. I’m still trying to figure out the right way to run unit tests with Brunch. The real issue for me was node and Bower packages. Since Brunch attempts to manage your code more and even wraps JavaScript in modules, it doesn’t seem to play well with packages. They claim support for Bower, but this wasn’t well enough documented for me and still didn’t seem to quite be right. It appears these issues are because Brunch pre-dates the widespread use of npm and Bower and they are looking at how they might solve these issues. In the mean time, be prepared for some awkwardness around that and around Browserify. Lastly, as is common with systems focusing on convention over configuration, if you stray off the established path you might have some issues. For example, I am still trying to figure out exactly how I can get cache busting to apply to assets like images if I am compiling the templates that reference those images.

Conclusions

I am disappointed by the state of JavaScript build tools in 2015. They are very immature and don’t seem to be well informed by the long history of build tools for other environments and platforms. Additionally, their authors don’t seem to be sitting down with the goal of really working through all the issues that come up in a build tool and making sure they are handled and the tool is easy to use. Only two, namely Brunch and Mimosa seem to have set out with the goal of making most builds very simple to setup and not requiring lots of boilerplate configuration.

So which tool would I recommend? Personally, I am still working with Brunch to see if it will work for my project. It would be at the top of my list right now (if you like the convention over configuration approach of Brunch, you could also check out Mimosa). However, if you don’t mind the large awkward configurations then you might want to select Gulp or even Grunt just for the size of the community around them and the selection of plug-ins (though I haven’t had a problem finding the Brunch plug-ins I need). I’m really interested in watching the development of Broccoli and hope it becomes a viable option in the future. The ultimate answer is likely to be a tool that doesn’t even exist today. Once package dependencies are ironed out correctly, I think we are likely to find all the existing tools don’t quite make sense anymore. Of course, there is no telling how long it will take to get to such a state.

It’s Just JavaScript, but Better

The example of CoffeeScript has clearly shown the advantages of an “It’s Just JavaScript” approach. A clear and direct mapping between the language and JavaScript makes it easy for the developer to understand what is happening and what they can expect. It aids early adoption when support for source maps may not be available yet. It provides a clear path to JavaScript interoperability. Importantly, this approach greatly simplifies the problem for an Open Source project that doesn’t have the resources to tackle a whole new platform approach the way Google has done with Dart. However, being just JavaScript doesn’t preclude deviations in semantics, not just syntax. CoffeeScript often shows an unwillingness to change JavaScript’s semantics even when it would be easy to do so and provide significant improvements to the developer. Instead we should strive to improve on JavaScript semantics when possible. That’s why, rather than “It’s Just JavaScript” or even “It’s Just JavaScript, the Good Parts” I propose “It’s Just JavaScript, but Better”.

Proposed Syntax

In syntax, “boring” is good. Deviating from syntax that is widely known without a good reason will probably just confuse people and hurt language adoption. I propose a largely C-style syntax with proper variable scoping. That is to say, you can’t use a variable or function before it is declared or outside the scope it is declared in. C# has carefully considered a lot of the issues and won’t, for example, allow one to shadow a variable or parameter in a method with another in the method. Scoping should just work, not cause any surprises and protect against unintended behaviour. Semicolons should be required statement terminators and white-space shouldn’t be significant. Parenthesis, commas and curly braces are required in the same places languages like Java, C# and JavaScript require them. All of that creates a language that is clear, unambiguous, easy to read and fast to compile.

forkeys(var key in obj)  // loops through Object.keys(obj)
{
  var inner = obj[key];
  inner.method(outer); // error can't access outer before it is declared
}
inner.method(); // error can't access inner that is not in scope
var outer = 3;

To improve on JavaScript, each operator should have a single result type regardless of the types passed to it. This rule is followed in other languages with implicit conversions like VB.NET. Although the flexibility of JavaScript’s boolean operator leads to cute idioms like using || for null coalescing, it does nothing for readability or bug prevention. Boolean operators should always result in a boolean value.

var x;
x = false or 5; // x == true;
x = "5" + 5;    // x == 10
x = 5 & 5;      // x == "55"
x = 1 == "1";   // false (== compiles to ===)
x = null ?? 5;  // x == 5, ?? returns the right side if the left is null or undefined
x = 4?;         // x == true, ? checks value is not null or undefined
// the existential operator '?' can also be combined
x = something?.method?(); // simply returns null rather than throwing an exception on null

To increase safety, all code should compile to JavaScript strict mode and be wrapped in an IIFE. Additionally, all uses of global scope should be explicit.

::console.log("diagnotistic message"); // use :: to access global
use $; // pull global variables into the local scope explicitly
$('#myButton').click(); // because of 'use' the '$' doesn't need '::' before it

Syntax Matters

The words you use for something matter. Words and syntax can either aid in comprehension and remind us of how things work, or can obscure meaning and consistently mislead. Having syntax for something can change how you think about it, how often you use it and ultimately how you program. The first time I remember being truly struck by that was when anonymous inner classes were added to Java. For those not familiar with Java, that feature allows one to create an unnamed “class” inside a method and instantiate an instance of it that has access to the containing scope. It is very much like being able to declare an object literal inside a function in JavaScript and have access to the function closure. In Java, there are enough restrictions on anonymous classes to make for a straight forward translation to Java without that feature. In fact, that is how the compiler implements it. Nevertheless, this feature provided syntax for something and changed how most people wrote Java code even though it didn’t strictly add any capabilities. It was always possible to do before, but was so awkward that it was rarely done. For JavaScript the situation is a little different. With JavaScript there is certain semantics that we will not be able to change without too great a performance or complexity penalty. In those cases, we need to come up with syntax that clarifies the semantics of JavaScript.

Proposed Syntax

I have repeatedly raised the issue of the confusing semantics of this in JavaScript. Attempts to improve JavaScript have generally made no attempt to address this issue. One of the two pillars of my approach is a new syntax for this. In other object oriented languages, this always refers to the current instance of the class lexically containing the occurrence of this. In JavaScript, it actually has a significantly different meaning, but the value of this ends up being equal to what it would have been given the more common definition in many situations. That is the root of the confusion. In order to create a syntax that clarifies this, we must first ask what are the semantics of this in JavaScript. The value of this can be determined in four different ways. Namely, through invoking a function after the . operator, invoking the function with a value for this using call or apply, or by invoking it as a constructor function using the new operator. Additionally, it can be fixed to a particular value by calling bind. The value of this could be better termed the context of the function call. A further confusion arises when functions are nested and have different contexts. In that situation, it is far too easy to accidentally reference the wrong context with this, because one forgets that the context of the nested function is different from the context of the outer function. This mistake is so frequently made because the more common definition of this is lexically based and the nested function is lexically nested, so it seems it should have the same context. I propose that both of these issues can be clarified by making the context a special named parameter of the function. Perhaps listed before the other parameters and separated from the rest by a dot rather than a comma, since the most common way of establishing the context of a function is with the dot operator.

use $; // use jQuery
$("li").map( (item . index) -> // jQuery passes each item as the context to the function
  {
    return cssClass -> item.hasClass(cssClass);
  });

You see above that I have also shortened the verbose function syntax to simply ->. The arrow is widely recognized as a function operator and is a proposed addition to ES6. I propose that it be the only function declaration syntax. Additionally, the return line shows that the parenthesis around the argument can be omitted when there is a single argument and the body can be shortened when the function returns a single expression. This functionality is like lambda expressions in C#. So the return line is equivalent to return (cssClass) -> { return item.hasClass(cssClass); };. Notice how there is no ambiguity over the context in the inner function because the context of the outer function is clearly and uniquely named. The typical mistake in JavaScript would be to write that line as return function(cssClass) { return this.hasClass(cssClass); };, but that would be incorrect because this does not refer to the context of the outer function inside the inner function.

A fully defined JavaScript alternative would probably have many other niceties like splats, array slicing, string interpolation, date literals, modules, default parameters, a call operator (perhaps ..) and some kind of asynchronous programming support like C#, but built on top of promises and callbacks.

Embrace Prototypes

JavaScript doesn’t have classes. Instead it has object prototypes and constructor functions. Yet, many developers yearn for classes and try to extend JavaScript with class like functionality. However each implementation of classes in JavaScript is different and many are incompatible with each other. JavaScript developers can’t even agree on whether/how to use the new keyword and declare constructors. As many authors, including Douglas Crockford, have noted, it is much simpler to embrace the prototype based nature of JavaScript and avoid not only classes but constructor functions by using Object.create(...) to directly setup prototype chains. David Walsh has a great three part write up on this (part 1, 2, 3). By accepting this simple truth and embracing it we can greatly simplify both the semantics and syntax of our language. Following the principle that syntax matters we need a syntax that clarifies this. We can further simplify our language if we accept that JavaScript doesn’t have private properties (at least without introducing a reasonable amount of overhead).

Proposed Syntax

I think the syntax for objects is important and while I propose a concrete syntax here, I think there is further room for improvement. A syntax for prototypical inheritance is the second pillar of my approach. The essential idea is that all object construction is through object literals, we simply need a way to specify the prototype object when declaring an object literal. Additionally, our syntax can be much clearer if object literals have a clearly distinct syntax from function bodies (this is needed to support the lambda expression style syntax described before).

var anObject = {**}; // an empty object, equivalent to { } in JavaScript

use console&.log; // The & is the bind operator used to bind the context
                  // so this is equivalent to console.log.bind(console)

var animal =
  {**
    // the constructor property of an object is special. When a
    // object is created with animal as it's prototype the
    // the constructor will be called on it (unless it has it's own
    // constructor).
    constructor = (animal . name) -> { animal.name = name; },
    legs = 4,
    eat = (animal . food) ->
      {
	    // Here we see the ability to embed expressions in strings
        log('{animal.name} ate {food} on {animal.legs} legs');
      },
    speak = -> { throw "speak not implemented"; },
  };
  
var dog =
  {*animal* // here we declare what the prototype is
    // We should call the animal constructor from our constructor.
    constructor = (dog . name) ->
    {
      // Notice we use the proto keyword to access it's
      // prototype.  When a function is called on proto, the
      // context is the original object (in this case dog)
      dog.proto.constructor(name);
    },
    // As in Javascript, this speak property will hide the prototype
    // speak property.
    speak = -> { log('Woof!'); },
  };
  
 var cat =
  {*animal*
    constructor = (cat . name) ->
    {
      cat.proto.constructor(name);
      // after calling the proto constructor, we can do our
      // own initialization
      cat.lives = 9;
    },
    eat = (cat . food) ->
    {
      if(cat.lives > 0)
        cat.proto.eat(food);
      else
        log('The {food} sits uneaten :(');
    },
    speak = (cat.) -> { if(cat.lives > 0) log('Meow!'); },
    oops = (cat.) -> { if(cat.lives > 0) lives--; },
  };

  // Now let's play with out pets

// This is how we pass parameters to the constructor
var myDog = {*dog("Skip")*};

var myCat =
  {*cat("Fluffy")*
    // There was already a horrible accident
    legs = 3,
    lives = 8,
  };
	
myDog.speak();					// prints "Woof!"
myDog.eat('table scraps');		// prints "Skip ate table scraps on 4 legs"

myCat.speak();					// prints "Meow!"
myCat.eat('a mouse');			// prints "Fluffy ate a mouse on 3 legs"

// Functions always return the context unless a return value is specified,
// so we can chain our calls to oops() (even return; will return the context)
myCat.oops().oops().oops().oops() // Someone has it out for fluffy

Note how using the = operator in object declarations increases consistency across the language. Unlike in CoffeeScript, JavaScript property attributes (introduced in ES5) should be considered.

var demo =
  {**
    hidden items = [], // not enumerable
    get count = demo. -> demo.items.length, // a getter
    readonly field = 42, // not writeable
    'a property name' = "something",
  };

Support for property attributes will have to be carefully planned. Additionally, it needs to be considered how private properties would be supported if they were added to JavaScript.

There are some other possible syntaxes for object literals. I welcome suggestions for this. Keep in mind that it has to be distinct enough that the compiler and programmer can distinguish it. The syntax can’t be ambiguous with existing operators or when used with the lambda form for -> described above (i.e. there must be something more than just { to start the object). It should also imply the creation of an object with a prototype (which is not the same as inheritance). Finally, it is worth considering how difficult it is to type including on international keyboards (many language avoid $ for that reason). I’ve come up with a few more considering those criteria.

// The ~ could be taken to mean 'like a', so that
var myDog = ~dog{name='skip'}; // is an object like a dog
var something = ~{prop=5}; // is a general object like all objects
var empty = ~{};

// Alternativly, the ~ could be moved inside the braces
var myDog = {~dog, name='skip'}
var something = {~, prop=5};
var empty = {~};

// Other operators could be used inside the brace instead
var myDog = {>dog, name='skip'}
var something = {>, prop=5};
var empty = {>};
// or

var myDog = {%dog, name='skip'}
var something = {%, prop=5};
var empty = {%};

Interoperate, don’t Imitate

The last principle that I propose guides how this new language should interoperate with JavaScript and whether it needs to be powerful enough to express every valid JavaScript program. I think interoperability with all JavaScript features is critical. We simply can’t predict how libraries will make use of JavaScript features and must make sure that they can’t break the user experience of our language. For example, I think it is very bad that CoffeeScript assumes property access is side effect free and idempotent (returns the same value repeatedly). Those are simply not true if the property is actually a getter. While no code written in CoffeeScript will have getters, libraries will increasingly use them as a larger percentage of installed browsers support them. Decisions like that could easily be the downfall of a language. What if the next great library, the next “jQuery” relies on them heavily? That could be the end of CoffeeScript, or at least a very painful transition for CoffeeScript users as the problem was fixed.

While interoperability is critical, that doesn’t mean it must be necessary to fully use every language feature. I have already proposed abandoning the ability to directly use the new operator or to easily declare constructor functions. It may be the case that not every property attribute or operator is important to have.

Proposed Syntax

While there should be no new operator. It would be possible to invoke constructor functions declared in other libraries by using them in place of object prototypes.

var birthday = {*Date("7/10/1980")*};

Another way the language can “Interoperate, not Imitate”, is to provide functionality that is similar to but not equivalent to JavaScript. For example, the typeof operator should be improved to at least return something more reasonable for the null value. Additionally, the instanceof operator should be replaced with something that checks directly against the prototype chain rather than falsely implying that there is such a thing as an object type tied to a constructor function.

What Now?

Having laid out these principles and proposed some concrete syntax based on them, the question becomes where do we go from here? I am seriously considering whether it is worth the effort of creating another JavaScript alternative. I have always been interested in programming languages and have the experience necessary to implement a compiler. However, I also know what a large commitment a project like that would be. Were I to start such a project, I would probably try to have an open dialogue process where people could provide input into language features and syntax and reasons for design decisions could be documented.

Gauging interest in another JavaScript alternative has been a major motivation in writing this series of articles. To that end, please take a moment to answer the question below. Furthermore, if you would be interested in contributing to such a language in any capacity whether it be coding, managing or design input please email me.

Signs Don’t Remove the Mines

Despite how helpful these tools can be, I don’t think they can ever really be considered an answer. Simply put, a sign doesn’t fix a problem. The problem still exists. If someone helpfully found every land mine in a minefield and put a sign next to them, the mines would still be there! The signs would certainly be beneficial, but an actual answer would be to defuse or remove the mines. Using a linter is better than nothing, but there is still increased cognitive load to avoid the pitfalls of JavaScript. The tool is just there to remind you when you slip up and teach you when you don’t know. Personally, I don’t need any more things to think about when I am trying to solve difficult programming problems. I want to focus as much as possible on the real business problem, not the difficulties of expressing it in code.

I recognize that this argument will sound abstract and like quibbling over semantics to some. They will say, if it lets me get the job done, it is a fix. I just can’t see it that way. To me, this is the most important part of why linters are not enough. Is it too much to ask for a programming language that doesn’t have such horrendous problems that I need a tool to help me avoid them? The very existence of such tools makes the case that JavaScript is a minefield.

Can’t Check Everything

Even with a lint tool, there are many things about JavaScript that won’t be checked. Additionally, even for issues that these tools address, JavaScript embedded in script tags in pages or in event attributes is generally not run through them. As an example of what is not addressed, tools do nothing to mitigate the confusing nature of the semantics of this.

(function () {
    "use strict";

    function Car(model) {
        this.model = model;
    }

    Car.prototype.drive = function () {
        return alert("You are driving a " + this.model);
    };

    Car.prototype.delayed = function () {
        return function () {
            return this.model;
        };
    };

    var myRide, letsDrive, getModel;
    myRide = new Car("BMW");
    letsDrive = myRide.drive;

    letsDrive(); // alerts "You are driving a undefined"

    getModel = myRide.delayed();
    alert("Delayed model: " + getModel()); // alerts "Delayed model: undefined"
}());

This code passes the strict JSLint tests with a clean bill of health, but contains what are probably developer mistakes. I’m sure someone with more JSHint/JSLint experience could come up with many more examples. This is not to knock these tools, it just isn’t possible to check for these things. A real answer would address these issues, either by changing the semantics or syntax of JavaScript.

Code Reviews

One commenter believed that “practices which enforce code review of every check in can help catch almost all of the silly coding mistakes we make that JSlint or JSHint might miss.” Like the use of linters, I believe that code reviews are good practise, but again they don’t fix the problem. They are just a strategy for mitigating it. Pilots have co-pilots there to double check them for the same reason we do code reviews, but that doesn’t fix a poor cockpit design. Instead, controls that make it easy for pilots to make dumb mistakes are redesigned by plane manufactures to fix the underlying issue. That way the pilot and co-pilot can spend their time focused on getting everything else right, making the flight safer.

Don’t Code Without One

If you are writing JavaScript code, I strongly recommend you use a linter. On my current project I have dropped CoffeScript in favour of JavaScript with JSHint. However, I don’t think that has fixed the JavaScript minefield. Furthermore, it has added issues with false warnings and littering our code with JSHint directives that detract from the focus of the code.

In this series, I have considered the most popular responses to the JavaScript minefield. I haven’t found one that truly addresses my problems with the JavaScript language. Of course, there are many more languages out there and I could be missing a great one. Though, I haven’t heard of any that meet what I see as the high level criteria of an answer and address the specific issues of JavaScript. In my next post, I will begin to explore what an answer might look like.

Ambiguous Code

I quickly found that there were situations where I couldn’t predict what a given chunk of CoffeeScript would compile to or couldn’t figure out how to write the CoffeeScript for the JavaScript I wanted. The problem arises because in CoffeeScript parenthesis, curly braces and commas are often optional and white-space and indention replace them. Frequently code I thought should compile didn’t. Consider this valid code.

func 5, {
   event: -> 45,
   val: 10}

Now, if the event function needs to be expanded, one would logically think it could be changed like this.

func 5, {
   event: (e) -> 
     if e.something
       36
     else
       45,
   val: 10}

However, that doesn’t compile. You have to drop the comma or move it onto the next line, before val.

How about something like func1 1, func2 2, 3? Does 3 get passed to the first or second function? I still frequently forget one place where parenthesis are not optional, leading to unexpected behaviour. The statement x = f -> f looks like it should assign x to the identity function, but it doesn’t. Instead, x is assigned the result of calling f with a function of no arguments returning f. Or how about this, a + b adds a and b while a +b calls a with the argument +b. There are lots of other examples of ambiguous and confusing syntax in CoffeeScript. For more examples, check out My Take on CoffeeScript by Ruoyu Sun and this Gist from Tom Dale.

Readable, Think Again

Ambiguous code is only one part of what makes CoffeeScript difficult to read. Everything is an expression (returns a value) and it has lots of control flow and operator aliases. All of which encourages very English sentence like code. However, that often makes the code less readable, not more. The human mind is good at understanding logic in symbols; English is not good at expression logic. As an example, consider the line eat food for food in foods when food isnt 'chocolate' from the CoffeeScript tutorial. The declaration of what food is occurs in the middle of the line and doesn’t even look like a variable declaration. Furthermore, until you finish reading the line it isn’t clear which foods will be eaten. That code could easily be worse if unless eat is undefined was added to the end, making the whole line conditional. One wouldn’t realize it was conditional until reading the end. Imagine if the expression before the for had been a complex multi-line method call with logic in it. Ryan Florence digs deeper into these issues in his post A Case Against Using CoffeeScript. Suffice it to say, many of the features added to CoffeeScript with the intent of making it “readable” actually have the opposite effect.

Variable Capture

In addition to the these confusions, CoffeeScript actually creates new mines that aren’t present in JavaScript at all. As explained by Jesse Donat in CoffeeScript’s Scoping is Madness, in CoffeeScript’s zeal for terseness and “simplicity” it has actually created a major pitfall around variable declarations. In CoffeeScript there is no variable declaration syntax equivalent to var in JavaScript. Instead, all variables are declared in the scope they are first assigned in. This means it is easy to accidentally change the scope of a variable and not even realize it.

To see how this would happen, imagine you are in a hurry to implement the next feature. Unbeknownst to you, the following code is near the bottom of the file you are about to modify.

innocent = (nums) ->
	lastValue = 1
	for num in nums
		doSomething lastValue, num
		lastValue = num
	lastValue

Now, near the top of a code file you add these lines. So that the file is now.

value = 42
lastValue = null
changeValue = (newValue) ->
	lastValue = value
	value = newValue


# many pages of code here


innocent = (nums) ->
	lastValue = 1
	for num in nums
		doSomething lastValue, num
		lastValue = num
	lastValue

Did you catch the error? Originally, lastValue was local to the innocent function, but now it is global to the file and the innocent function actually modifies it. We now have a bug waiting to happen when someone calls innocent then checks the value of lastValue. Keep in mind there could be multiple screens of code between these two code segments.

Not Far Enough

Certainly, a language that is “just JavaScript” won’t be able to fix everything. Yet, despite radically altering the syntax of JavaScript and claiming to only expose the good parts of JavaScript, in some ways CoffeeScript doesn’t go far enough in fixing the issues of JavaScript. For example, the + operator is still both numeric addition and string concatenation. That is frequently listed as one of the bad parts of JavaScript. Why no provide separate operators for the two? Likewise, the typeof operator “is probably the biggest design flaw of JavaScript, as it is almost completely broken” according to the JavaScript Garden, but CoffeeScript brings its behaviour over unchanged. Instead CoffeeScript could have altered the meaning of typeof to something that was more useful, for example the typeOf() function recommend by Douglas Crockford.

Classes Are an Illusion

Many developers wish that JavaScript had classes. This has led to numerous alternative ways to emulate class like functionality in JavaScript and libraries that embody those various approaches. Many of the different approaches don’t interact well with each other. There is even debate on whether to use constructors requiring the new keyword or to make everything a factory function. CoffeeScript has a class keyword that makes it easy to create classes. However, since “it’s just JavaScript” they are just one particular emulation of classes on top of JavaScript. Consequently, they may not play well with your library of choice. More troublesome is that they make other language features more confusing. In particular the this keyword. When creating a class you can’t help but feel that this should refer to the class everywhere within the body of the class. That is what it would mean for something to be a class. But the actual semantics of this are unchanged. For example, if you declare a class with methods using this.

class Car
  constructor: (@model) -> 

  drive: () ->
    alert "You are driving a " + this.model;

  delayed: () ->
    -> this.model

Because Car is a class, one expects this to always refer to the current car object, but instead it behaves like regular JavaScript.

myRide = new Car("BMW")
letsDrive = myRide.drive

letsDrive() # alerts "You are driving a undefined"

getModel = myRide.delayed()
alert("Delayed model: " + getModel()) # alerts "Delayed model: undefined"

Ultimately, JavaScript isn’t a class based language and classes never quite work right in it. I believe it is a mistake to push the language in that direction. For a more in-depth discussion of why this is the case, see JavaScript Doesn’t Need Class by Ian Elliot.

I have been writing JavaScript for 8 years now, and I have never once found need to use an uber function. The super idea is fairly important in the classical pattern, but it appears to be unnecessary in the prototypal and functional patterns. I now see my early attempts to support the classical model in JavaScript as a mistake.

And So…

CoffeeScript started from a strong position and good philosophical approach to the problem of the JavaScript Minefield. Unlike TypeScript, it was willing to break with JavaScript and fix its issues. Unlike Dart, it was intended to remain true to JavaScript and be fully interoperable. Realistically, the issues I have raised with CoffeeScript don’t come up every day. Ultimately though, they are enough to say we need something better. I hope to share some thoughts soon on what such a solution might look like. However, based on feedback in the comments, we’ll need to discuss JavaScript linters first.

Why not ASP.NET Web Optimization

Ever since Microsoft released the ASP.NET Web Optimization framework, now available as a NuGet package, it has been the default choice on the .NET platform for bundling and minification of scripts and css. Indeed, I initially started with it. However, I quickly ran into a number of issues with it, all of which were made more challenging by the lack of documentation.

It provides support only for minification of *.js and *.css files by default and no support for languages like Less, Sass, TypeScript or CoffeeScript. I’ll be using Less, and possibly CoffeeScript. It seems the encouraged workflow for such languages in Visual Studio is to use the Web Essentials extension to compile them. However, with that, the compiled js or css files are included in the project and need to be committed to source control. I have found that to be problematic when doing merges in a DVCS. Also, it just seems like the wrong thing to do. Like committing the compiled assemblies of my .NET code. Even using third party or custom built extensions, it generally isn’t possible to mix languages in a single bundle. For example, I might want to combine some JavaScript libraries into a bundle containing CoffeeScripts I have written.

More serious than that, ASP.NET Web Optimization provides no support for compiling or embedding templates. In the official EmberJS template from Microsoft they use a beta version third party extension to provide template compilation for Ember.js and provide a 117 line class for embedding templates outside the the web optimization framework. The third party library appears to be abandoned, with 8 months since the last commit and still in beta as of March, 2014. Even with all that, the EmberJS template’s solution doesn’t switch between embedding and compiling based on whether the web optimization library is in debug mode, leading to possible problems.

Introducing Cassette

The Cassette library preceded Microsoft’s ASP.NET Web Optimization library and is arguably the primary alternative to it. If your familiar with how Web Optimization approaches bundling it may take a while to become accustomed to the Cassette approach. They use the term “bundle” somewhat differently, which is confusing when you are first learning. I recommend you read the “Getting Started” and “Assets and Bundles” sections of the Cassette v1 docs before switching and reading the the v2 docs. Those sections of the v1 docs explain basic concepts not explained in the v2 docs. Until I read them, Cassette wasn’t making any sense to me.

The main difference between the approaches is around what a bundle is. In the Web Optimization library, a bundle is a group of files that will be minified and combined into a single file. Referencing that bundle is including a reference to the combined file. Whereas, in Cassette a bundle is more like a group of files that work as a single dependency. Meaning that you would never want one file out of the bundle separate from another. Referencing bundles is then stating what the page’s dependencies are. The bundles are then minified and combined into three separate files for css, scripts and templates. For templates, Cassette supports the idea of embedding the templates rather than loading them from a separate file.

At the top of your cshtml page you reference any individual assets or bundles the page depends on:

@{
    Bundles.Reference("Scripts/jquery.js");
    Bundles.Reference("scripts/app"); // the main application bundle
    Bundles.Reference("Scripts/page.js");
    Bundles.Reference("Styles/page.css");
}

Then in the page body, you indicate where the html to include scripts, templates and style sheets should be rendered. This will then group the dependencies by type and, depending on whether Cassette is in debug or production mode, output either debug friendly assets or minified, compressed, cached, versioned assets.

<!DOCTYPE html>
<html>
<head>
    <title>Web App</title>
    @Bundles.RenderStylesheets()
</head>
<body>
    ...
    @Bundles.RenderScripts()
    @Bundles.RenderTemplates()
</body>
</html>

Cassette is a powerful library with many more features and options. Be sure to check out the documentation for the details. Additionally, the author of Cassette has created NuGet packages for drop in support for Less, Sass, CoffeeScript, TypeScript and more. One of those NuGet packages is for pre-compiling Mustache Templates using Hogan.js. Unfortunately, this doesn’t work for Ember.js, because it needs its own template compiler due to the get and set methods on Ember models. Never the less, we’ll see that all is not lost.

Configuring and Referencing an HtmlTemplateBundle

Setting up a bundle in Cassette for all your templates is very easy. Don’t install the Cassette.Hogan NuGet package since it doesn’t work with Ember.js. I choose to put all my templates in the app/templates folder following the example in the EmberJS template from Microsoft. The template uses the extension hbs for Handlebars templates, however I choose to use the also common extension handlebars, because it was more explicit. Since all the templates are in one directory, creating a bundle was an easy addition to my Cassette bundle configuration.

public class CassetteBundleConfiguration : IConfiguration<BundleCollection>
{
    public void Configure(BundleCollection bundles)
    {
        // Other configuration ...

        bundles.Add<HtmlTemplateBundle>("app/templates", new FileSearch()
        {
            Pattern = "*.handlebars",
            SearchOption = SearchOption.AllDirectories,
        });
    }
}

To reference the templates from my application page only required adding Bundles.Reference("app/templates"); to my bundle references. The templates were then automatically embedded in the page inside script blocks in place of the @Bundles.RenderTemplates() call right before the close body tag.

Giving Templates a “data-template-name”

There were some problems with the embedded templates at this point. The script blocks were being generated with an id attribute based on the path of the template file and the type attribute was set to “text/html” instead of “text/x-handlebars”. While there is some confusion over this, I believe that the data-template-name attribute is the preferred way of identifying your ember templates, rather than the id attribute. The reason is that nested route templates have names separated with ‘/’, but it is not valid to have the ‘/’ character in an html id. Fortunatly, Cassette is based around a very flexible pipeline model, making it easy to customize. After reading some of the documentation, poking around the source and reading some some code from the Cassette.Hogan package, I came up with a simple solution.

Cassette allows the bundle pipeline for any bundle type to be easily modified by implementing the IBundlePipelineModifier<T> where T : Bundle interface. All bundle pipeline modifiers are picked up automatically. Fixing the issues was almost as simple as setting the content type of the html template pipeline and swapping out the implementation of how templates were wrapped in a script block.

public class SetupHandlebarsPipeline : IBundlePipelineModifier<HtmlTemplateBundle>
{
    public IBundlePipeline<HtmlTemplateBundle> Modify(
            IBundlePipeline<HtmlTemplateBundle> pipeline)
    {
        // Set the content type
        var index = pipeline.IndexOf<ParseHtmlTemplateReferences>();
        pipeline.Insert(index, new AssignContentType("text/x-handlebars"));

        // Replace how the scripts are wrapped
        index = pipeline.IndexOf<WrapHtmlTemplatesInScriptElements>();
        pipeline.RemoveAt(index);
        pipeline.Insert(index,
            new WrapTemplatesInEmberScriptElements(new HtmlTemplateIdBuilder()));

        return pipeline;
    }
}

The new WrapTemplatesInEmberScriptElements class is basically the same as the old WrapHtmlTemplatesInScriptElements with the id= replaced by data-template-name=.

public class WrapTemplatesInEmberScriptElements : IBundleProcessor<HtmlTemplateBundle>
{
    private readonly IHtmlTemplateIdStrategy idStrategy;

    public WrapTemplatesInEmberScriptElements(IHtmlTemplateIdStrategy idStrategy)
    {
        this.idStrategy = idStrategy;
    }

    public void Process(HtmlTemplateBundle bundle)
    {
        foreach(var asset in bundle.Assets)
        {
            asset.AddAssetTransformer(new WrapTemplateInEmberScriptElement(bundle, idStrategy));
        }
    }
}

public class WrapTemplateInEmberScriptElement : IAssetTransformer
{
    private readonly HtmlTemplateBundle bundle;
    private readonly IHtmlTemplateIdStrategy idStrategy;

    public WrapTemplateInEmberScriptElement(
            HtmlTemplateBundle bundle,
            IHtmlTemplateIdStrategy idStrategy)
    {
        this.bundle = bundle;
        this.idStrategy = idStrategy;
    }

    public Func<Stream> Transform(Func<Stream> openSourceStream, IAsset asset)
    {
        return () =>
        {
            var template = openSourceStream().ReadToEnd();
            var scriptElement = String.Format(
                "<script type=\"{0}\" data-template-name=\"{1}\">\n{2}\n</script>",
                bundle.ContentType,
                idStrategy.HtmlTemplateId(bundle, asset),
                template
            );
            return scriptElement.AsStream();
        };
    }
}

An Exception

If you try the above code with version 2.4.1 or prior of Cassette, you’ll get an exception when it’s not in debug mode. I didn’t notice this issue until a week after writing the code above when I deployed to the test environment. The switch between production and debug mode can be confusing in Cassette, because it isn’t your first thought when something works on the developers machine but not the deployment environment. It took me at least an hour to track down the problem. In production mode the exception you get is KeyNotFoundException from deep inside Cassette around bundle cache code. Turns out that setting the content type to "text/x-handlebars" causes Cassette to not know what extension to give the cache file. That seems to be poor design choice to me, but essentially the fix is that "text/x-handlebars" needs to be added to a list of known content types. I have submitted a pull request to do this, that will hopefully be accepted soon, so this won’t be a problem in future versions. Until then, you can work around this by adding the following hack to the beginning of your CassetteBundleConfiguration.Configure() method.

// Hack so we can use type="text/x-handlebars" in release mode
var bundleType = typeof(Bundle);
var field = bundleType.GetField("FileExtensionsByContentType",
                                 BindingFlags.Static | BindingFlags.NonPublic);
var fileExtensionsByContentType = (IDictionary<string, string>)field.GetValue(null);
// Sometimes the config is run again and it is already there
if(!fileExtensionsByContentType.ContainsKey("text/x-handlebars"))
	fileExtensionsByContentType.Add("text/x-handlebars", "htm");

With that, my Ember.js templates where embedded correctly into the page and developers could begin work with a clean separation of the templates into individual files. Obviously, before production release I would like to be able to enable compiled templates when Cassette is not in debug mode, but that challenge can wait for another day.