Adapt Learning Community: Node.js style require in Adapt

Skip to main content

I've recently been looking at using browserify to use Node.js style require on the client.

This makes modularity very easy, and solves many issues with data being leaked to the global scope. I've put together some basic applications with this technology using Backbone.js and jQuery, making use of Backbone routers and ajax request for JSON. All data is stored in a closure with no way to get access to it.

Using this would also set us up for having adapt core, components and plugins all stored on npm (node package manager) and with a package.json we can set dependencies ensuring a developer need only run a single `$ npm install` and have everything that is required.

This also ties in with running the Authoring tool on a Node.js server

If anyone would like to see some example code in this style, let me know

Fabien

Hey Fabien,

Thanks for sharing that - it's a really interesting approach. It could solve some of Daryl's headaches regarding modularity and data access. It also gives us an out of the box approach regarding the question of what a contributed module should look like - i.e. it looks like a node module!

On a tangentially related note, Rob Moore shared an interesting video with me that gives a nice overview of achieving modularity with a one page javascript app:

It gets a bit specific regarding Backbone.js and Marionette.js, but I think the broader ideas are very pertinent to this discussion.

Dennis

Show parent | Reply

There is a grunt task for browserify too, so you can automate the whole thing in the build process.

With regard to npm, I think its a great idea, I suggest that we tag plugins with keywords e.g. "adapt", "plugin" so that the Authoring Tool can search for and find plugin in the registry.

One caveat on npm, do you know if its possible to control where the package is installed? i.e. node_modules
So far we have discussed putting plugins in a different folder in the folder structure so this might screw things up a little.

What do you think?

Edit: I've been using bower on another project for client-side dependencies, you can specify a location for the components (default is 'bower_components' but i prefer to use 'components'), you can reference the git repo directly without having to publish to a registry and it has a javascript api too.

Show parent | Reply

Yeah, that'll make things a lot easier, it also supports source mapping for when we're developing it.

We could use a similar system to the one grunt uses, all grunt plugins are named grunt-[pluginName] and plugins maintained by the core team are named grunt-contrib-[pluginName].

Unfortunately we can't force node modules to be installed anywhere other than node_modules, however as this is going to be an end result is that so much of a problem? If a developer was to add a plugin manually they could either use
npm install adapt-reallyCoolPlugin or drop it into a js folder and require it using a relative path.

Bower is pretty interesting however to use browserify with bower I believe you have to use a module called debowerify. Referencing just the git repo is very appealing

Show parent | Reply

I think that having plugins in node_modules may be an issue if the plugin has any css or handlebars that needs to be packaged with it.

Referencing CSS from node_modules would feel wrong to me.

I don't have a answer on this but I'm thinking that a custom grunt-task to extract the CSS/SASS or HBS from the plugin before compilation and minification would work.

Show parent | Reply

I think that we could use the "scripts" attribute in the package.json to compile any templates for us on install of a plugin, i don't see a problem with a plugin referencing its template as a relative path from inside its package.

e.g. require("./template") // i think this is completely fine

I can see what you mean about the css and think that a custom grunt task would be suited to this, especially if using browserify, which concat's all scripts into one, we would have simple one css and one js file. very elegant imo

Show parent | Reply

Hey,

Ok, so after much 'umming' and 'arrring' and finding out about a few more dependancy loaders (what convinced me was this article - http://dontkry.com/posts/code/browserify-and-the-universal-module-definition.html) I've realised that there are so many ways of defining modules and the way browserify does this is clever. I like the way it integrates with npm, but maybe this could be better solved with a grunt task.

We could look at keeping all our node_modules in our src folder. CSS and Handlebars templates could also come down with them. But like Chris said, I think a custom Grunt task to extract them would be best. Especially as we'll be using SASS for our CSS preprocessor and Grunt has some great tasks for this. This way our modules don't have to reference templates and css files.

I read somewhere that keeping your core code separate to your plugins is a better approach as core code shouldn't be changing that often. So having something like one CSS file and one JS might not fit so well. (I believe one of the issues is the compiling time).

Thanks,

Daryl

Show parent | Reply

I think the git repo wouldn't store any node_modules folders, just package.json, the developer can be expected to run npm install IMO. In terms of keeping files separate, core code would always be kept separate from plugins, but at build time i see no reason to keep the two separate, they're never going to edited or modified in that state.

I would also suggest splitting the core up into separate npm modules, with an main adapt-core module that is dependant on the rest.

Reading about the plugin architecture of eclipse, there were many comments on having the core as small as possible with everything you can manage to be a plugin. This idea I believe could take Adapt very far, as starting the main code base with half it's features as plugins ensures that we as developers will make it as easy as possible to add plugins, I believe eclipse takes this to the extreme whereby its core is also a plugin, I don't see that as necessary here.

Note, by above I'm am not suggesting that Adapt be anything other than what it is and i expect it to be rendered useless if any of the core plugins/modules are not present, however I think that architecturing it in this way would solve many headaches in the future that we just wouldn't have encountered - it also gives us countless examples to show how plugins works and would ease developers into the core make-up of the product quickly and with a step by step plan they could follow (the dependancy tree).

Fabien

Show parent | Reply

Hi,

I'm mindful of staying out of the depth of this discussion but would like to comment from a high-level vision perspective prompted by the phrase 'the developer can be expected to...'.

From a very high level view, we are committed to driving for a low barrier of entry for developers as well as end users. This is so that we can achieve mass adoption. A lot of this also comes down to for how long we can keep new developers, i.e. people who literally just want to have a play and see where they can get in five or ten minutes. If they can't achieve much in that time, we may have lost them forever.

And when referring to developers at large, I'd like us to keep in mind that even technical people may well have a far lesser knowledge of the libraries we use and the code we have written to start with.

So in summary, let's make this really easy for them, avoid lots of intricate setup tasks to get started and make sure the documentation is good. I'd be a willing guinea pig to try and tell you if it's easy enough.

Apologies if you're keeping all of this in mind already.

Thanks, Sven

Show parent | Reply

Was playing around with creating some npm modules, and noticed that dependencies can be listed as a git url followed by a "commit-ish"

Quote their docs : The commit-ish can be any tag, sha, or branch which can be supplied as an argument to git checkout. The default is master.

This means that the authoring tool can specify versions of adapt it works with via a tag. as well as components only working in adapt 1.x etc..

This idea seems very powerful, versioned dependencies will be very important if this is to be a long running project

Show parent | Reply

Hey,

With regards to locking away data and modularity I believe the new Adapt API will stop this. The talk Dennis posted is great for picking up the things Adapt should have been when we first started developing it. Unfortunately time constraints never allowed us to properly think about the architecture as we were prototyping at the same time.

So the new API forces a modular approach to declaring extensions/views/models/collections and routers whilst maintaining a level of security (a level we agreed would "be enough"). It also forces developers to code to our standards with a simple API. It follows simple approaches like asking for data and not taking (all covered in the talk).

However what I liked about "browserify" is the dependancy management it contains. One thing I was hoping the Editor to control is dependancy management. Every plugin should contain some sort of dependency list, whether this be in a json file or in it's javascript files. (I prefer a json file as it can contain css and template files too). With an editor being able to select or deselect plugins it should be able to find it's dependancies and warn the user of these (maybe the user should be given the chance to decline the extra installs and decide upon a different plugin altogether?). Whether this comes from "browserify" or "grunt-concat-in-order" or another system, this is something that Adapt doesn't not currently have.

I guess a question for Dennis and Rob - would it make sense to move the dependancy management into the editor?

(Sorry for any spelling mishaps - sat here ill slightly bored.)

Thanks,

Daryl

(A thing to consider when minifying all of our javascript files together is that some LMS's don't allow single line minification above a certain character size. - jQuery has fallen over in two LMS's so far when fully minified).

Show parent | Reply

I think the level of modularity you would get from browserify is far more than what the proposed API would be able to do. Also locking away the global ADAPT namespace at compile-time would mean we wouldn't have to worry about any security as it is completely inaccessible after a project has been built.

I think using an approach with browserify would also make it easier for developers to code how they want, a backbone view for a component is no longer necessary, one could use any object with the right interface and it'd just work.

Also, Chris you may be able to answer this, are bower components able to list their dependencies? In the future we could have canvas components required a dependency on say ImpactJS, using npm or bower like this could make things very easy. Developers could start using their own utility libraries, linking to them in a bower.json or package.json file

Show parent | Reply

The API for npm supports listing dependencies in a tree structure... the same as doing "$ npm ls".
Not sure about bower. Worst case I guess you could just load the JSON file as a object, and iterate over the dependencies.

Like you said though, if a plugin has a package.json in its root with dependencies then npm will automatically install these anyway.

If the Authoring Tool is built in Node.js then using the npm API is supported by default.

Show parent | Reply

Yeah, I was thinking more for supporting Bower, i guess it's not too much trouble to iterate through JSON..

I know that npm doesn't install dependencies twice, we'd have to think about that if rolling our own for Bower. I'm much more in favour of using npm, although certain libraries will need to be shimmed. I had an issue with using backbone like this but you simply have a backbone.js in a libs folder

var Backbone = require("backbone")
Backbone.$ = require("jquery");

module.exports = Backbone;

and then require("./libs/backbone") throughout the app

Show parent | Reply

Hi Daryl,

There appears to be a bit of conflation at the moment between the scenario of a developer extending or programming into the Adapt framework and a "Course Creator" simply using the Adapt authoring tool and selecting components for inclusion in a course.

I think dependency management clearly belongs with the authoring tool in the second scenario, but this thread raises some interesting approaches to dependency management within the Adapt framework for contributed plugins. In the end, the authoring tool will have to adapt to whatever approach you decide to go with - either rolling your own package.json-esque plugin management approach, or working with one of the solutions proposed here by Fabien and Chris.

Generally, however, I would favour a solution which is as simple as possible and over which we (you) have tighter control.

Den

Show parent | Reply

Hey,

I guess this is how I see it and maybe I am getting a bit confused.

1) As a content editor I am able to select a plugin - Let's say I'm creating an extended version of the Multiple Choice Question. As I select this, the editor looks through it's dependancies - where ever that may be and goes:

"Oh I need the ComponentView, QuestionView, my new shiny template, my styling but also the mcq styling". The editor is then able to determine where each script and css should be concatenated before or after to allow dependencies to be loaded first. It would also allow all my templates to be put into one big templates.js file. Also if it contains any partial template files to put those in it's own partials.js

2) Like the content editor - we also need a process that will allow a developer to create a new extension and without putting it through the editor be able to run a task that loads those dependencies in order like above.

I've only touched on browserify so don't know too much about it. But what I read is that it can handle dependencies in this kind of manner.

But while we're discussing issues like this we should also take in to consideration that we need a dynamic script loader based upon conditionals. Now this could happen before plugin loading is done and we could use Modernizr yepnope. But we also need to consider creating an API for Adapt where people have a familiar syntax (thinking of jQuery/Zepto/Google maps) to tap into. This creates a low barrier of entry for developers coming from a web designer/developer role.

Hope this all makes sense.

Thanks,

Daryl

Show parent | Reply

Feel free to shoot me down, but I would not have thought that plugin creation would fall into the scope of the Authoring Tool.

I can see that it would need to display a list of available plugins and install them into a Project/Course, but I would expect plugins to be created by a developer using standard web development tools.

Show parent | Reply

I agree with that, Chris.

Show parent | Reply

Hey,

Sorry Chris I think my post came across wrong. I said that in number 2 that there should be a way for a developer to create a plugin and compile the files in the correct order without using the editor. I agree, all development should happen outside of the editor.

Have you done much front end dependency loading? What we need to do is pick a solution that can work across both the editor and on a developer kit. Which makes a lot of sense using something node based, whether this be browsify or bower.

Is there any chance you could share some of your loading/grunt processes.

My initial thoughts were to have a config.json for each plugin that would list it's dependencies including css, templates, partials and javascript files. Using some sort of Grunt process we can find the dependencies and concat those first.

I think we broke the final js files into these:

adapt.js - all main core code

components.js - all components in here

theme.js

menu.js

extensions.js - all extensions in here.

However there are other files to factor in. For instance json2.js, which version of jQuery to use, swfObject.js and a few others that are loaded based upon what browser the user is on. This can also be used to provide a smaller library footprint on mobile devices. We have to bare in mind that we need performance as well - so any advice on that too would be great?

So, I have a few questions -

Does browsify solve all of these?

Or are there other work arounds we haven't yet thought of?

Should we be considering Bower?

We also need to load certain CSS files in order - is this possible?

(Not sure about this one but we might need to load templates in certain orders too - mainly partials)

I need to pull together enough information so we can make a decision on moving forward.

Thanks,

Daryl

Show parent | Reply

Maybe it would be useful if I just describe what I'm currently using in my build process for front-end dev and why. I'm constantly evolving the tools I'm using but some the general functions are the same even if the tool is different.

Package Managment - bower & npm

(I'll use npm for my dev dependencies, and bower for my client dependencies; I find having them in separate folders makes it easy to manage deployment)

Conditional Module Loader - Modernizr/yepnope.js (thanks to Daryl for that one)

Build Process - grunt

UI Framework(if required) - ember

My build tasks tends to be pretty much the same every time -

a test runner (grunt-mocha-test, grunt-contrib-watch)

a local server (grunt-connect, grunt-concurrent, grunt-open)

CSS pre-processor (grunt-contrib-less)

concat+minify (this gets complex but is the basically yeoman build tasks; grunt-copy, grunt-neuter, grunt-contrib-uglify, grunt-usemin)

I think that npm/yepnope/grunt are awesome and will save massive amounts of time along with browserify as the concat and uglify as the min process.

Bower is optional really, its nice because you can configure it easily and reference a git repo, but tbh I think if we can pull the non .js assets(cs/hbs/images) out of a plugin using a grunt task then we can just leave the plugins in node_modules.
If we get problems doing this then at that point we could consider Bower as an alternative.

Show parent | Reply

Thanks Chris,

Nicely summed up. I liked how you've given us your workflow - it has a sense of this works and works well. But also to bring Sven's comments in about making it as easy as possible for a developer to get involved - I think aiming for a process that is as simple as running Grunt in terminal or git bash would make everything easier.

It's got me thinking about Sass and Less now - Sass needs Ruby installed and seems to be the most powerful (can run as a grunt command but still needs Ruby installed). Now on a PC machine ruby doesn't come preinstalled. Where as if we go with a node based css preprocessor like Less this becomes a lot simpler.

I think as Adapt developers we should aim for something that runs off one system - node. If this is the case I'd be up for considering Less over Sass. How has Less worked for you?

Thanks,

Daryl

Show parent | Reply

At Sponge we use Less as the default, as we use Express.js as our webserver and it supports it straight out of the box via middleware.

Its a super-set of CSS so if you just want to use plain ol' CSS syntax then that is just fine, but if you want to refactor your styles to use variables and mixins then that's okay too.

The only "problem" I sometimes encounter is when new devs discover nesting rules and go a bit crazy and build really deep, over-specific rules are that not reusable.

I've never used SASS, but at a first glance the syntax is pretty much the same, just more $'s and less @'s.

I've just done a quick test of node-sass and it SASS works natively in Node, so no need for Ruby.

Show parent | Reply