I’ve spent the last three years working on single page applications of various shapes and sizes. I don’t like em,  this post isn’t about why but I will just say I like data to be exposed at the lowest level (HTTP) and not require Javascript to turn it into something useful.   All that being said I’ve been lucky enough to work with some clever folks and the end results have all been very interesting and pushed boundaries in their own way.

At Full Frontal I enjoyed listening to Nicholas Zakas talking about Scalable Javascript Application Architecture. In many ways he described the architecture we used for the Volkswagen Configurator. Earlier in the day Phil Hawksworth had been talking about Excessive Enhancement.

It struck me that while most front end MVC frameworks make it easy to build Javascript applications, making them work as web applications (without Javascript) becomes much more difficult. This is largely due to the fact that you define and build your application, templates/page structure using the browser’s Javascript interpreter.

I’m of the view that if you architect something properly you can get the same full experience provided by Javascript applications but still have a reliable fallback for those who do not have Javascript enabled. That is what I call a proper web application.

In the back of my mind I’ve been thinking about Charlie Robbins excellent post: Scaling Isomorphic Javascript Code which talks elegantly about why MVC might not be the best pattern in an environment where you can execute Javascript on the server. He suggests the Resource-View-Presenter pattern.

It seemed that the way we currently split our frameworks between front and back end reduces reuse of configuration/code and actually encourages duplication.

I wanted to try defining all of these things on the server, so they could be consumed on the server and client.

Sharing URLs between client and server

Allow me to deviate for a moment.

On large applications it’s common practice to split the code and teams between front and back end developers.  This often results in duplication and unnecessary bugs.  A common example being maintaining URLs in two places.  In the browser we might a have a URLs object:

namespace.urls = {
     login: '/login',
     user: '/user/:username',
     ....
}

And then on the server the same URLs would be defined possibly using a different syntax.  Changing one does not change the other, and with split teams this can result in unnecessary bugs.

With Node.js it’s particularly easy to share exactly the same URL object on the client and server. If you ensure your HTML links are populated from the same URL object your application will continue to function when URLs change.

This is what I’ve been doing in some of my Node apps:

(function(exports){
    exports.HOME = '/';
    exports.LOGIN = '/login';
    exports.USERS = '/users';
    exports.USER = '/users/:user';
    // also add URL functions here that can be shared between client and server.
    exports.build = function(str, tokens) {
        for(token in tokens){
            str = str.replace(':'+token, tokens[token]);
        }
        return str;
    };
})(typeof exports === 'undefined' ? namespace.urls={} : exports);

That allows it to be used as a Node module using require(‘./urls.js’) and when served to the browser the URLs are available at namespace.urls.

I’ve been using Express.js, the parameter sytax seems to work nicely with Levi routes on the front end.

This seems like common sense. It’s a fine example of DRY.  Define as much as possible in server-side JS and then allow it to be consumed by the client-side JS reusing as much logic as possible.

Reusable modules

TiddlyWiki has the concept of plugins which are essentially just a chunk of HTML/CSS/JS relevant to a particular piece of functionality which could be added to HTML using a special syntax:

<<pluginName>>

On the Volkswagen Configurator we also have the concept of UI’s which were reusable chunks of HTML/CSS/JS which could be appended into any DOM node.

Both are essentially variants of the Module Pattern with added support for HTML and CSS as part of the module.  Both methods work really nicely just so long as you’ve got Javascript turned on.

I started to build a simple Node.js app which could parse a modules folder and serve the resources at appropriate URLs. See the following folder structure:

Generated the following URLs:

/contact.css
/contact.html
/contact.js
/content.css
/content.html
/content.js
/flickr.css
/flickr.html
/flickr.js

You will notice there is an app.js file in the flickr folder which contains the server-side JS required by the module. Later this will provide a getData method to asynchronously generate a templating object which can populate the HTML.

Define your views on the server

In browserland it’s really easy to mess around with the DOM. You can completely transform a page, especially when you start appending modules of HTML/CSS/JS to DOM nodes. The problem comes when you want to show the same view to something that doesn’t understand Javascript, an RSS feed or search engine for example.

I decided to create a view specification which could be read by the server and also served to the browser. Doing so should make it really easy to render the exact same HTML with or without Javascript enabled.  Using Sizlate I started with this (it has since changed):

var views = [{
    url: '/user/:user'
    view: 'userpage',
    modules: [
        'photos',
        'login',
        'timeline'
    ]
 }];

In the following example I have left out the JS and CSS files.  The view is just a folder containing a HTML file of the same name and an optional app.js (as with the modules). It assumes the view HTML file contains a  tag with a id corresponding to each of the modules specified for the view.

I like this simple approach but in order for it to scale I may soon need to start using HTML data attributes instead of ids. This is what the folder structure looked like:

project/views/userpage/userpage.html (the view HTML file) contains:

<html>
<body>
    <nav>...</nav>
    <div id="photos" /> 
    <div id="login" />
    <div id="timeline" /> 
</body>
</html>

The end result would look something like this:

<html>
<body>
    <nav>...</nav>
    <div id="photos">
        .. contents of /modules/photos/photos.html appended in here
    </div> 
    <div id="login">
        .. contents of /modules/login/login.html appended in here
    </div> 
    <div id="timeline"> 
        .. contents of /modules/timeline/timeline.html appended in here
    </div>  
</body>
</html>

Concatenation vs Caching

At this stage I was able to build a sample app with my re-usable modules and a view. I thought it would be useful to concatenate all the JS and CSS files together for each view. It turns out I was wrong.

In a situation where you’re reusing modules across multiple views concatinating per view makes no sense because you end up serving the same CSS across multiple URLs. The CSS/JS will be cached per view, not per module.

I decided it was actually better to use the URLs generated for the CSS/JS by the modules so they can be cached at the most granular level.  Both methods will be possible.  Currently CSS modules are served inline with the module HTML (not the document HEAD).  There will be some work to ensure that all CSS LINK tags are moved to the document HEAD before the view is rendered. JSDOM should make that quite easy.

History API

With views and modules defined on the server I’ve started to put together a front end framework which can consume the same application specification and make the whole experience a bit more pleasant.  I’m currently experimenting with generating popstate listeners from the specification which can then fire off the default/custom transitions between views.

Whats going on here?

Essentially I have started building a framework for mixing up reusable modules of HTML/CSS/JS in ways usually associated with Javascript applications but with progressive enhancement as one of its core values.

The functionality described above will almost certainly change.  What I have built so far is a simple proof of concept to test the best way to define views in this way using Sizlate.

At the moment I’m working on a sample app pulling in data from Flickr to demonstrate how it might all fit together in the real world.  Its pretty messy and requires lots of work but you can see the code here.

I thought it was worth blogging about my approach just to get some feedback. Please do let me know what you think. All idea’s, contributions, criticisms welcomed.

I’m going to be talking about Sizlate at the London Node user group on the 25th January at the Forward Offices in Camden, London. Please register here if you would like to attend.

One of my more interesting experiments has been Sizlate.

On projects at work I often find myself doing things like this:

 domNode.find('div.class').html('<b>INSERT SOME STUFF HERE</b>'); 

It’s a really powerful way to populate HTML. What I really like is that there is no need to add any crazy syntax into my templates.  Templates are just valid HTML and the point of insertion is specified by the jQuery selector.

From the developers point of view this is really simple, it does however introduce problems when Javascript is not turned on. I found myself wondering how this technique might be transferred onto the server.

After some experimentation I came up with Sizlate. A HTML templating engine for Express.js.

Its pretty easy to get jQuery running on Node.js but I decided that  jQuery wasn’t  a good fit for my use case.  Sizzle is the selector engine used by jQuery, I decided to investigate using Sizzle to provide the selector functionality. It turned out that this works quite nicely using the JSDOM project.

To use Sizlate you simple need to register it as your templating engine:

var sizlate = require('sizlate'); 
app.register('.html', sizlate); 

And then just call res.render as you would normally with Express:

 res.render('template.html',  { selectors: { 'a': 'hi there' } }); 

That’s the most basic example. On github I have provided an example of  passing sizlate an object allowing more complex data structures to be used. There is also an example using partials.

At the moment Sizlate only works on the serverside but it should be quite easy to get it working in the browser.

Feel free to have a play and let me know if you have any feedback.

Sizlate is available as a NPM package and can be installed using the command:

 NPM install sizlate 
 
For more details please read the readme on github.com.

background: #FF8D2C; /* for non-css3 browsers */
/* For WebKit (Safari, Google Chrome etc) */
background: -webkit-gradient(linear, left top, left bottom, from(#FFFFFF), to(#FF8D2C));
/* For Mozilla/Gecko (Firefox etc) */
background: -moz-linear-gradient(top, #FFFFFF, #FF8D2C);
/* For Internet Explorer 5.5 - 7 */
filter: progid:DXImageTransform.Microsoft.gradient(startColorstr=#FFFFFF, endColorstr=#FF8D2C);
/* For Internet Explorer 8 */
-ms-filter: "progid:DXImageTransform.Microsoft.gradient(startColorstr=#FFFFFF, endColorstr=#FF8D2C)";

When building a house the foundations are fundamental to it’s structural integrity. Without good strong foundations the house is weak and liable to fall over at any moment.

Things on the web also have foundations in the form of HTML and URIs.  JavaScript can then be layered on top to improve the experience. It’s called progressive enhancement and people seem to be forgetting about it these days.

It’s pretty simple, build your HTML and CSS  first and then override the default behaviours with JavaScript.  This will ensure you are building on solid foundations.

When generating HTML on the server, you can easily re-use it with JavaScript. Its far better than generating the HTML in the browser (with JavaScript) and either ignoring Search engines or having to duplicate your logic on the server for SEO, accessibility and things like RSS feeds.

Here are some rather sweeping statements:

1. JavaScript should NEVER be used to process data in the browser.

2. JavaScript should rarely be used in the browser to generate html (sharing code with server-side JavaScript is acceptable).

I did warn you they were rather sweeping.

I’ve heard it said that if you want to provide an app/flash like experience you need to use JavaScript to render your pages: You need to build single page JavaScript apps.

history.pushState() tells us otherwise. You can read about it here.

Basically it makes it possible for what we now call single page web apps to exist across multiple pages while still providing nice page transitions (no page refresh).

history.pushState() – A Fallback

History.pushState is all well and good but it’s only available in WebKit and Firefox(4) at the moment.  Maybe that is why people are seeing hashbangs as an alternative solution. Personally I would rather fallback to a fragment identifier (#) only in situations where history.pushState is not available.

There would need to be a bit of JavaScript at the top of each page redirecting users to the appropriate fragment identifier in browsers that do not support pushState.

So when pushState is not available:

http://simonmcmanus.com/page.html

might redirect to :

http://simonmcmanus.com#page.html

which would then go and fetch the contents from

http://simonmcmanus.com/page.html

If a page is loaded with the fragment identifier in a browser that supports pushState the hash should be removed and pushState used.

Summary

When you require JavaScript for templating and data processing you are on a very slippery slope to writing JavaScript applications.

We have for a long time been able to obfuscate our data with technologies like Flash, I for one have avoided these technologies because I believe that when we publish data properly on the web it becomes more re-usable, findable, accessible and actually has far greater potential.

By all means use JavaScript, but please don’t rely on it.

The following examples only work on the latest webkit browsers (Chrome and Safari). In Chrome you will need to enable “GPU Accelerated Compositing”. Goto about:flags turn on “GPU Accelerated Compositing” and then restart Chrome.

I started off with one of Paul Hayes early experiments which later progressed into his animated cube demo.

Here is what I came up with.

Screenshots:

This is what I did:

Step 1: Create the 3d space

Wrap two divs round the elements to appear in 3D :

<div id="perspective">
   <div id="rotator">
      <!--original HTML in here-->
   </div>
</div>

Set the following CSS:

.perspective { -webkit-perspective: 2400; }
.three-d { -webkit-transform-style: preserve-3d; }

Step 2: Setting translateZ

Set translateZ for each item to appear 3D:

-webkit-transform: translateZ(3em);

In my example I set it for the following items:

h1,
.item,
.twitter-bird.sign,
.vcard{
   -webkit-transform: translateZ(3em);
}

In this case each z-index will be represented by 3ems.

Step 3: Making it move

Define the animation :

#rotator {
   -webkit-animation-name: rotate;
   -webkit-animation-duration: 15s;
   -webkit-animation-iteration-count: infinite;
}

And then specify the steps of the animation:

@-webkit-keyframes rotate {
   0% {
      -webkit-transform:rotateY(0deg);
   }
   5% {
      -webkit-transform:rotateY(0deg);
   }
   30% {
      -webkit-transform:rotateY(-40deg);
   }
   50% {
      -webkit-transform:rotateY(85deg);
   }
   55% {
      -webkit-transform:rotateY(85deg);
   }
   90% {
      -webkit-transform:rotateY(0deg);
   }
}

demo here.

Step 4 ..  Perspective

To give a zoom effect I created a second animation:

#perspective {
   -webkit-animation-name: perspective;
   -webkit-animation-duration: 15s;
   -webkit-animation-iteration-count: infinite;
 }

@-webkit-keyframes perspective {
   0% {
      -webkit-perspective: 2400;
   }
   5% {
      -webkit-perspective:2400;
   }
   30% {
      -webkit-perspective:650;
   }
   50% {
      -webkit-perspective:2000;
   }
   55% {
      -webkit-perspective:2000;
   }
   70% {
      -webkit-perspective: 2400;
   }
}

final demo

That’s what I’ve done so far. Comments and modifications welcomed.

Happy New Year

First off we need to get all the tiddlers required for CKEditor, the easiest way to do that is to cook the ckeditor vertical recipe.

For your convenience I’ve uploaded the TiddlyWiki file it produces here.

Now we need to download the CKEditor files, You can download them here.

Finally you just need to change the location set in the MarkupPreHead tiddler so it points to the CKEditor folder you just downloaded. In my case:

<script type=”text/javascript” src=”./ckeditor/ckeditor.js“></script>

If you are using cook you will need to save the MarkupPreHead tiddler from within the TiddlyWiki file for it to be set correctly. I have already done so in the example provided.

You may also want to change the values in the “CKEditorSettings” tiddler.

Save your changes, refresh the browser and that should do it.

Update:

As some folks experienced problems getting this setup I have provided a zip file of the output:

http://simonmcmanus.com/stuff/ckeditor/twCKEditor.zip

I have published a basic template which make it clear how to create a plugin that adds an item to the CKEditor toolbar, when that button is pressed the users is presented with a simple dialog box containing two tabs which can interact with the parent editor.

The template can be found at :

http://simonmcmanus.com/stuff/tw_uploader/testPlugin.zip

curl http://svn.tiddlywiki.org/Trunk/contributors/SimonMcManus/tiddlyweb/tiddlydocs/install.sh >install.sh
sudo sh install.sh
cd tiddlydocs
twanager server 127.0.0.1 8080

In your browsers then goto :

http://127.0.0.1:8080/recipes/tiddlydocs/tiddlers.wiki

The essential dissatisfaction I have with the parliament website is that the information is not being published for re-use. In this paper I will explain what I mean by this, why I believe it and offer some alternative solutions. I would be more than happy to come and discuss this with you further and would appreciate any feedback that you might have.

Executive Summary

Websites like Wikipedia demonstrate how conversations can take place around information. For each article there is a discussions tab which allows readers and authors to discuss the articles. If you would like the same thing to occur around your meeting transcripts and legislation you need to change the format to make the data referenceable, commentable and easily queried by a programming language.

I believe there are five steps to opening up Parliamentary data:
1 . Data Copyright
2 . Making raw data available
3 . Marking up data with semantic information
4 . Making data linkable
5 . exposing data through APIs

each of these parts will now be discussed.

::: 1 . Data Copyright

Publishing data online holds little value if the data has not been licensed for reuse. I suggest all parliamentary publications be made available under a creative commons copyright, so that anyone can republish commercially or otherwise.

::: 2 . Making Raw data available.

It is important that the raw data dumps which make any application possible are also available to the public. The data should be available be with no style information, no scripting nothing but pure, unadulterated data. While there is value in Parliament building websites/applications it is far more important that developers have equal access to the original data so they can build other applications without the unaffected by the preconceptions of parliament.uk developers.

Currently most parliamentary publications are in PDF form. This causes a number of problems :
1 .. Individual pages and sections are not indexed by search engines.
2 .. It is difficult to programmatically extract data from a PDF.
3 .. It is not possible to reference particular sections of a document.

::: 3 . Marking up data with semantic information

Theyworkforyou.com have put together a basic template of how parliament can improve the semantics of parliamentary publications. More details of their suggestions can be found at the following address :

http://www.theyworkforyou.com/freeourbills/techy

I fully endorse these suggestions. If followed, they would make it a great deal easier for developers like me to build new and richer interfaces because it makes the data more meaningful.

::: 4 . Making data linkable

When writing the paper it was particularly difficult to find the references from transcripts of your Committee’s meetings. It was sent to me in the following form :

“The transcript of the meetings the Committee has had as part of its inquiry are available here:

http://www.publications.parliament.uk/pa/ld/lduncorr.htm#info

(See in particular questions 78, 85 and 86 of the 1 April meeting).”

Finding the information required the following steps to be taken:
1 .. clicking the above link

2 .. Searching for the meeting on the 1 April
3 .. clicking another link which downloaded a big PDF file
4 .. Waiting for the entire document to download

5 .. Searching the pdf for question 78
6 .. Searching the pdf for question 85
7 .. Searching the pdf for question 86

I would like to see an implementation where clicking the following three URLs would take you straight to view each question, allow you to read its answer and comment against either.

1 .. http://www.publications.parliament.uk/InformationCommittee/PeopleAndParliament/Meetings/1April09/Question78
2 .. http://www.publications.parliament.uk/InformationCommittee/PeopleAndParliament/Meetings/1April09/Question85
3 .. http://www.publications.parliament.uk/InformationCommittee/PeopleAndParliament/Meetings/1April09/Question86

If the information is published in HTML files which are being indexed by Google the bills will be findable in google and extend your outreach to every single user of google.

The simplest way to expose data on the web is to break it down into small individually addressable sections each of which has a unique URL. These URLs can then be sent round in emails, added to a user’s favorites or programmatically interrogated.

::: 5 . Exposing data through APIs

An Application Programming Interface (API) provides developers an interface for interacting with a data set easily. By making it possible to programatically search legislation and comment against a particular section from a remote site, it becomes much easier for people to build new interfaces for the available data. A good API would make it really easy to build new ways of browsing, searching and commenting on legislation.

Data should be exposed so that it can be presented in ways never expected by those collating the data. It is through this approach that you help people to view and, most importantly, interact with both Houses regarding proposed legislation.

A good API will make data available in a number of different formats. HTML, XML and JSON are a good starting point. From the earliest possible opportunity any code being used to expose data should be open sourced so that developers can extend the existing code base without needing to start from scratch. Not only does this allow people to build things more quickly, it allows developers to extend functionality and form a community of developers working together to improve the nations data infrastructure.

If Parliament wants to engage with people it will be a great deal easier on sites they already visit rather than the parliament.uk site. You cannot expect to engage the majority of the electorate at parliament.uk. It needs to be made particularly easy to integrate the goings on of both Houses into any website so that useful (relevant) data can be pulled in about a given subject.

A site about digital rights and copyright should be able to make a call to the API which looks for any recent mentions of “Digital Rights” and “Copyright” and can then embed the results in its own site. I also suspect that providing functionality to comment against the results would massively increase the potential of both Houses to engage with the electorate.

Below I have put together a general criteria for exposing data on the web :
1 .. The data should be indexable and discoverable in google/other search engines.
2 .. The data should be exposed using open data formats (HTML, oof)
3 .. The data should be licensed for re-use, even by commercial organisations.
4 .. It should be possible to browse the data in a web browser.
5 .. Make the original data set available.
6 .. Maintain a consistent interface for developers to build against.
7 .. Any code used to abstract away from the data should be open sourced.
8 .. Any semantics that can be added to data are beneficial.
9 .. There should not be separate systems for MPs and the public. Whatever system MPs use to look up bills, track their progress through parliament, find out amendments etc should be available to the public.

The following criteria are not essential but I suspect could have a major effect on parliaments ability to interact with the people online :
1 .. the information should have a plain English summary. Often bills are full of jargon that makes them incomprehensible to many. A good example is the creative commons licenses. These have a plain English version and a legalspeak version.
2 .. Electronic book support. All bill, minutes etc should be available in a non-proprietary electronic book format (eg epub) for download.

Please note all that all the above should be possible for very little cost. All the software required is available for free with open source software licenses. The primary cost should be for one or two developers who work with the community to expose data based on user/developer feedback.

“Come up with a way for my team to collaborate with people outside the organisation on large Word document. At the moment, we tend to email them between us, which can degenerate into a nightmare.”

Our approach was to think about breaking these large documents down into small sections which could be assigned to individuals for authoring and/or review. These sections could then be reassembled into a single document for printing.

To build this proof of concept we used TiddlyWiki and jQuery. The latest release of TiddlyWiki (2.5) includes jQuery (1.2.6) so this provided a fine opportunity to explore the new potential.

So here is the demo,

The online demo does not demonstrate the iGoogle integration of TiddlyDocs.   The iGoogle integration showed how we could provide a personalised view of the sections assigned to each user.  While we used iGoogle for the original demo this portlet just contains HTML so could be embedded into any portal framework which supports HTML.

PSD wrote some XSL-FO to translate the generated HTML into a printable PDF document. Paul’s code can be found here.

Features :

Drag and Drop your Table of Content

Allows users to change the order and hierarchy of the document table of content

Easily assign sections to Users

Allows each section to be easily assigned to a pre-populated list of users.

Full WYSIWYG text editing with FCKEditor

Allows users to edit the document content in a familiar word style environment.  Also allows content including images to be pasted in the textarea from the clipboard.

iGoogle Integration of Personalised Task Lists

The iGoogle integration allows each users to have their own view of all the tasks assigned to them. This can be used in various portal frameworks.

Easy view of the documents status

The bar to the right of the table of content shows the status of each section. When the bar is completely green the document is complete.  This example demonstrates that only one of the sections has been completed.

Each Section Allows for Comments

Each section allows multi-threaded comments which will not be printed in the final document.  This allows for conversations to take place around each section of content.

The Components :

We were able to build this proof of concept so quickly by reusing open source code that is available in the TiddlyWiki and jQuery communities.  Below I talk about the components that make up TiddlyDocs.

SplitView Plugin

This is one of the plugins I wrote specifically for the demo. The macro accepts one parameter which is the name of another tiddler to store the document specification. The document specification is stored in list format, eg:

* Heading 1
* Heading 2
** Heading 2.1
** Heading 2.2
* Heading 3

Every time the document is re-arranged the specification tiddler is saved.

PrintView Plugin

This is the other plugin which I wrote specifically for the demo.  This plugin can read the document specification, collect the data from each of the required tiddlers, and then generate a static html file on the server which can then be converted to PDF for printing.

Single Page Mode Plugin

Provided by Eric of TiddlyTools, the singlePageModePlugin ensures only one tiddler is viewed at a time.

http://tiddlytools.com/#SinglePageModePlugin

Comments Plugin

Allows users to post comments against individual sections. These comments are presented as a threaded discussion, allowing rich conversations to take place around the content.  Thanks to Michael Mahemoff for this plugin who also helped with the demo.

http://tiddlyguv.com/CommentsPlugin.html

Value Switcher Plugin TeamTasks

Taken from Phil Hawksworth‘s TeamTasks, this allows the fields provided by TiddlyWiki to be used to store information about ownership and current status.

http://svn.tiddlywiki.org/Trunk/contributors/PhilHawksworth/verticals/TeamTasks/core/plugins/ValueSwitcherPlugin.js

from:  http://getteamtasks.com

FCKEditor

FCKEditor is a project to make a nice WYSIWYG editor from any HTML textarea.  I have to say a I have been very impressed with FCKEditor.

http://www.fckeditor.net/

TiddlyWiki FCKEditor Plugin

A TiddlyWiki plugin that makes FCKEditor work nicely with TiddlyWiki.

http://visualtw.ouvaton.org/VisualTW.html#FCKeditorPlugin

NestedSortable -(a  jQuery plugin)
This jQuery plugin provides the drag and drop interface for the demo.

http://code.google.com/p/nestedsortables/

This demo is hosted by ccTiddly but its been written to work on TiddlyWeb and any other server-side implementation of TiddlyWiki.  Over the next few months I will be working on TiddlyDocs to prepare it for easy consumption.

Updated : Michael Mahemoff has produced a screencast talking through the features of this demo :  http://vimeo.com/3109248