Speclate and Service Workers

I have alway been keen to provide a good offline experience for the London Node user group website. I’m pretty bad with directions and London is even worse with internet coverage. Knowing I can always get the map and address on my phone is very handy. We already use AppCache with the help of app-cache-nanny but with its upcoming depreciation I wanted to investigate how service workers might improve the whole experience.

Speclate is the tool we use to build the static LNUG website, it uses Sizlate which is a basic templating engine allowing data to be inserted into HTML using the Sizzle selector engine from jQuery.

In this example we take a HTML string and modify the innerHTML of the h1 element:

Sizlate can also take a DOM node when running in the browser:

Sizlate uses Cheerio to provide a fast DOM implementation at build time, in the browser it  needs something with jQuery like API to access to the DOM.

Speclate extends Sizlate with the idea of components and pages.  A spec.js file sits in the project root and defines the sites structure. Notably how the routes, pages, components and data all fit together. The exported Javascript object is used to combine the various HTML files that make up the pages. At build time it renders static HTML files for each page but it can also render just the parts of the page that change as users navigate around the site.

Here is a simple spec for a site with two pages:

Given that spec, three files need to be present:

  • ./pages/home/home.html
  • ./pages/contact/contact.html
  • ./pages/layout.html

Those paths are all relative to the spec.js file which should sit in the root of a project.

Here is a slightly more complicated site spec, it makes use of components and global selectors. Components are defined in the spec and only applied to the page (not the layout). The selectors are applied to the whole page after it has been appended to the layout which means you can modify things like the page title:

In this example it also uses the ‘contact’ component which needs to contain an li element and should be located at:

./components/contact/contact.html

You can see the full LNUG site spec here:

https://github.com/lnug/website/blob/master/spec.js

Application Shell – layouts

Speclate requires a pages/layout.html file which contains the outer layout for a site. The layout usually contains the nav, header and footer. For the LNUG site the markup in the layout.html file produces this:

screen-shot-2016-10-08-at-07-30-42

layout.html contains an element with id of container which the page content is inserted into. The first time you visit a site you’re served the layout as part of the complete page generated at build time. As you browse to routes defined in the spec, speclate-router reuses the layout and just updates the contents of the #container element with the new page and its components. By reusing the page layout we only render the parts of the page that change. This isn’t done with any clever virtual DOM diffing, we just know which parts of the page are going to change from the spec, and only change those parts. By not re-rending elements that don’t change it’s much easier to make smooth transitions between pages.

Service workers allow us to cache the layout.html file for all the routes defined in the spec. When a user hits a route it loads layout.html from the cache. The Javascript in layout.html triggers speclate-router to try and fetch the latest spec for the page and render it. This provides an application shell which can show the main nav to a returning visitor in under 300ms. It works well but I wonder if we could be even more efficient by only creating one cache entry for /pages/layout.html and referencing that when we intercept the fetch requests.

Partial updates

When making a change to the LNUG site using appCache, Speclate modifies a version hash in the appcache.manifest file. AppCache nanny detects the change and updates everything in the browser cache. The LNUG layout, pages, components and CSS files rarely change but the site gets updated several times each month with new information about speakers and their talks. That’s probably more often than most users visit the site.

Busting the whole cache each time we make a minor change is in not making efficient use of our user’s cache.

Long live the Service Worker

Service workers can intercept fetch requests which allows us to be far more efficient with the user’s cache. With Speclate 7.0 when you generate a site it creates a .json file in docs/speclate/api for each page defined in the spec. Each file contains just the information needed to render the page. For the current LNUG homepage the spec looks like this.

speclate-service-worker uses a network first strategy for requests to the speclate/api folder. Every time a user loads a page it tries to get the latest JSON page spec from the server. If the user is offline, it falls back to the cache. All other requests are served straight from the cache. By only loading a small spec file from the server we are able to present a fully rendered site with the latest data from the server over a throttled 2g connection in just over a second. By not busting the whole cache for regular updates we can keep the service worker until we want to make changes to the page, component and CSS files. This ensures we always serve the latest info while at the same time dramatically reducing the amount of data a returning visitor has to download. Speciate-Router and speclate-service-worker are both focused around loading the whole site into the service worker and rendering it from there (after first page load of course). This only works with very small sites but with some tweaks they could both handle partial loading.

Support

We don’t want to drop appCache, the LNUG analytics show there are users who don’t have service workers available. Here is the code we use to check if services workers are available, falling back to appCacheNanny when they’re not:

Note how the appCache nanny on ‘updateready’ sits outside of the else statement, this ensures returning visitors with appCache and services workers available are upgraded to use service workers.

Returning visitors are shown the layout while the server is checked for updates. This can result in the footer being visible and then disappearing to the bottom of the page when the content is loaded. That should be resolved with the addition of some nice CSS transition. (we have an open issue btw)

This implementation has an interesting side effect. If a user turns Javascript off after loading the site with Javascript and service workers enabled they will only see the layout. The page content will never load. All the Javascript resources are being loaded from the cache so the Javascript will never fail due to network conditions and we have static HTML files to revert to if the Javascript fails to load for some other reason.

Speclate ensures everything is defined and maintained once in the spec file which helps bring down the cost of these layered enhancements. We start off with completely static HTML files. If the browser supports history.pushstate client-side routing is provided. If the browser supports service workers returning visitors get a really fast experience on and offline. If they don’t, well, it’ll still work offline with appCache support.

If you would like to try Speclate I‘ve updated the example app to use service workers:

https://github.com/simonmcmanus/speclate-example

Please let me know if you have any comments, feedback or questions. Im also very grateful to all those who have blogging about service workers, notably, Jake Archibald, Jeremy Keith and Lyza Danger Gardner. Getting the LNUG site working with service workers was much easier thanks to your articles and sample code.🙂

 

 

 

 

Speclate

The London Node User group website is a static site hosted on github pages, it renders at build time, in the browser and works offline using Speclate.

The idea behind Speclate is you have a single file (a spec) which defines how your routes, pages and components fit together. A spec can be used to generate a static site and lots more too.  With the help of speclate-router we generate a router which swaps out the appropriate HTML elements to transition nicely between pages.

Let’s look at a simple example:

module.exports = {
     '/': {
         page: 'home',
         selectors: {
             title: 'Home',
             'a.home': {
                 className: 'active'
             }
         }
     },
     '/contact.html': {
         page: 'contact',
         selectors: {
             title: 'Contact',
             'a.contact': {
                 className: 'active'
             }
        }
     }
};

This spec will produce index.html and contact.html based on the contents of /pages/home/home.html and /pages/contact/contact.html and the values in the selectors object.

At the moment Speclate relies on a couple of conventions. /pages/layout.html needs to contain an element with an id of “container”for the pages to be appended to. This will move to config eventually.

I’ve created a simple example app. You can see the demo running on Netlify, and get the source code on GitHub. You can also read more about the spec format in the speclate readme.

Rendering

Underneath the hood Speclate uses Sizlate which makes extensive use of Sizzle, the selector engine from jQuery.  On the server, Cheerio provides a fast DOM implementation for generating the markup.

Rendering can be broken down into three steps. Firstly we replace the main page (using #container), then we apply the page spec, this will apply selectors and components to the page and lastly we apply the global selectors to the markup. This allows us to effect elements in the main layout such as the page title.

In the example there are three commands in the package.json to generate the site:

"scripts": {
    "markup": "speclate",
    "client": "browserify ./client/router.js > ./client/router-compiled.js",
    "build": "npm run client && npm run markup"
}

‘npm run markup’ will generate a functional HTML site. ‘npm run client’ uses browserify to generate the client-side router and ‘npm run build’ generates the whole site by running both commands.

Offline

Currently, the LNUG website works offline using an AppCache.manifest file. That file is generated from the spec like so:

speclate.site.appCache(spec, [
'/css.css',
'/images/lnug-logo.svg'
])

We use appCache Nanny to handle updates but the experience is still not great. The latest releases of Speclate now use the fetch API so it opens up lots of possibilities with Service Workers.

EnhanceConf Scholarship

 

One of the big motivations behind EnhanceConf  was education. Conferences are a great place to learn but the associated costs often make them difficult for many to attend. In the spirit of progressive enhancement, we want to make EnhanceConf available to the widest possible audience.

We have allocated a number of scholarship tickets for people who would like to attend but cannot pay for a ticket.

How to Apply

Send a short email to info@enhanceconf.com with a title of EnhanceConf Scholarship.

In your email please include:

– Who are you?
– What do you do?
– Why would you like the scholarship?

Who should apply:

The scholarships are available to anyone not normally able to attend a conference. If that might be you (even if you’re not in one of the groups below) then please apply.

Some of the groups we are especially keen to support include:

  • Ethnic minorities
  • Women
  • LGBTQIA+
  • Disabled people
  • Charity workers
  • Education / students
  • Open source projects
  • Industry newcomers

If you have any questions, just send an email to info@enhanceconf.com

Details: 

All emails will be treated in the strictest confidence, we will not announce who the scholarships go to but you’re welcome to say so yourself.

Successful applicants will get a standard conference ticket.

We are only able to cover the cost of the conference ticket so please consider your travel costs before applying.

EnhanceConf line-up

Over the last 6 months, with the help of some trusted advisors, I’ve been putting together the line-up for EnhanceConf.

The whole event is dedicated to progressive enhancement, but what does that even mean in 2016? Well, that’s the point of EnhanceConf. We get to spend the 4th of March in The Great Room figuring it out.

great-room

We are going to start the day thinking about why we do progressive enhancement. Nat Buckley will look at many of the assumptions we make in modern web development. Anna Debenham will then look at some of the unusual devices people use to interact with our sites/apps. Stefan Tilkov will provide an architect’s perspective.

In the second section, we are going to look at some real world examples. We’ve got Ola Gasidlo from Styla/Hoodie and Oliver Joseph Ash from the Guardian talking about their experiences making things work offline. Forbes Lindesay will be talking about building apps that can render on the server and in the browser.

After lunch, we are going to think a bit more about design and UX. Does progressive enhancement constrain design? Phil Hawksworth and Stephen Waller will debate. Jen Simmons will explain why 2016 will be the year that web layouts change and how to use these new techniques as an enhancement. Adam Silver will talk about embracing simplicity.

In the final section, we are going to look to the future. How can we build interfaces given how little we know about the people and devices using them? Robin Christopherson will be talking about inclusive design. Stephanie Morillo will talk about copy and what to think about when writing it. Aaron Gustafson will round off the event by talking about building for devices that don’t exist yet.

After each section, we are going to have 20 minutes of Q&A run by Jeremy Keith.

As if that wasn’t enough, the Planning Adaptive Interfaces workshop provides a unique opportunity to get training from Aaron Gustafson while he’s in London. We are giving a limited number of copies of Aaron’s e-book Adaptive Web Design with combined conference and workshop tickets.

Reserve your seat

Forcing Garbage Collection with Node.js and V8

I’m in the middle of looking for some memory leaks at the moment, in order to isolate them I wanted to confirm exactly how much memory was being used by a given line of code after garbage collection.

Fortunately V8 allows you to manually force garbage collection from within Javascript.

When you run your node script just add the option: ‘–expose-gc’

e.g:

node --expose-gc test.js

And then from within the Javascript just do:

global.gc();

That’s it, good luck finding those leaks!