intertwingly

It’s just data

Realtime Updates of Web Content Using WebSockets

Preface

You've seen web sites with stock prices or retweet counts that update in real time. However, such sites are more the exception rather than the norm. WebSockets make it easy, and are widely supported, but not used as much as they could be.

Examples provided for WebSockets typically don't focus on the "pubsub" use case; instead they tend to focus on echo servers and the occasional chat server. These are OK as far as they go.

This post provides three mini-demos that implement the same design pattern in JavaScript on both the client and server.

Quick Start

For the impatient who want to see running code,

git clone https://github.com/rubys/websocket-demo.git
cd websocket-demos
npm install
node server.js

After running this, visit http://localhost:8080/ in a browser, and you should see something like this:

header

  • one
  • two
  • three

Server support

The primary responsibility of the server is to maintain a list of active websocket connections. The code below will maintain three such sets, one for each of the demos provided.

// attach to web server
var wsServer = new websocket.server({httpServer: httpServer});

// three sets of connections
var connections = {
  text: new Set(),
  html: new Set(),
  json: new Set()
};

// when a request comes in for one of these streams, add the websocket to the
// appropriate set, and upon receipt of close events, remove the websocket
// from that set.
wsServer.on('request', (request) => {
  var url = request.httpRequest.url.slice(1);

  if (!connections[url]) {
    // reject request if not for one of the pre-identified paths
    request.reject();
    console.log((new Date()) + ' ' + url + ' connection rejected.');
    return;
  };

  // accept request and add to the connection set based on the request url
  var connection = request.accept('ws-demo', request.origin);
  console.log((new Date()) + ' ' + url + ' connection accepted.');
  connections[url].add(connection);

  // whenever the connection closes, remove connection from the relevant set
  connection.on('close', (reasonCode, description) => {
    console.log((new Date()) + ' ' + url + ' connection disconnected.');
    connections[url].delete(connection)
  })
});

The code is fairly straightforward. Three sets are defined; and when a request comes in it is either accepted or rejected based on the path part of the URL of the request. If accepted, the connection is added to the appropriate set. When a connection is closed, the connection is removed from the set.

EZPZ!

Client Support

The client's responsibitlity is to open the socket, and to keep it open.

function subscribe(path, callback) {    
  var ws = null;
  var base = window.top.location.href

  function openchannel() {
    if (ws) return;
    var url = new URL(path, base.replace('http', 'ws'));
    ws = new WebSocket(url.href, 'ws-demo');

    ws.onopen = (event) => {
      console.log(path + ' web socket opened!');
    };

    ws.onmessage = (event) => {
      callback(event.data);
    };

    ws.onerror = (event) => {
      console.log(path + ' web socket error:');
      console.log(event);
      ws = null;
    };

    ws.onclose = (event) => {
      console.log(path + ' web socket closed');
      ws = null;
    }
  }

  // open (and keep open) the channel
  openchannel();
  setInterval(() => openchannel(), 2000);
}

A subscribe method is defined that accepts a path and a callback. The path is used to construct the URL to open. The callback is called whenever a message is received. Errors and closures cause the ws variable to be set to null. Every two seconds, the ws variable is checked, and an attempt is made to reestablish the socket connection when this value is null.

First example - textarea

Now it is time to put the sets of server connections, and client subscribe function to use.

Starting with the client:

var textarea = document.querySelector('textarea');

// initially populate the textarea with the contents of data.txt from the
// server
fetch("/data.txt").then((response) => {
  response.text().then((body) => { textarea.value = body })
});

// whenever the textarea changes, send the new value to the server
textarea.addEventListener('input', (event) => {
  fetch("/data.txt", {method: 'POST', body: textarea.value});
});

// whenever data is received, update textarea with the value
subscribe('text', (data) => { textarea.value = data });

The value of the textarea is fetched from the server on page load. Changes made to the textarea are posted to the server as they occur. Updates received from the server are loaded into the textarea. Nothing to it!

Now, onto the server:

// Return the current contents of data.txt
app.get('/data.txt', (request, response) => {
 response.sendFile(dirname + '/data.txt');
});

// Update contents of data.txt
app.post('/data.txt', (request, response) => {
 var fd = fs.openSync(dirname + '/data.txt', 'w');
 request.on('data', (data) => fs.writeSync(fd, data));
 request.on('end', () => {
   fs.closeSync(fd);
   response.sendFile(dirname + '/data.txt');
 })
})

// watch for file system changes.  when data.txt changes, send new raw
// contents to all /text connections.
fs.watch(dirname, {}, (event, filename) => {
  if (filename == 'data.txt') {
    fs.readFile(filename, 'utf8', (err, data) => {
      if (data && !err) {
        for (connection of connections.text) {
          connection.sendUTF(data)
        };
      }
    })
  }
})

Requests to get data.txt cause the contents of the file to be returned. Post requests cause the contents to be updated. It is the last block of code that we are most interested in here: the file system is watched for changes, and whenever data.txt is updated, it is read and the results are sent to each text connection. Pretty straightforward!

If you visit http://localhost:8080/textarea in multiple browser windows, you will see a textarea in each. Updating any one window will update all. What you have is the beginning of a collaborative editing application, though there would really need to be more logic put in place to properly serialize concurrent updates.

Second example - markdown

The first example has the server sending plain text content. This next example deals with HTML. The marked package is used to convert text to HTML on the server.

This client is simpler in that it doesn't have to deal with sending updates to the server:

// initially populate the textarea with the converted markdown obtained
// from the server
fetch("/data.html").then((response) => {
  response.text().then((body) => { document.body.innerHTML = body })
});

// whenever data is received, update body with the data
subscribe('html', (data) => { document.body.innerHTML = data });

The primary difference between this example and the previous one is that the content is placed into document.body.innerHTML instead of textarea.value.

Like the client, the server portion of this demo consists of two blocks of code:

app.get('/data.html', (request, response) => {
  fs.readFile('data.txt', 'utf8', (error, data) => {
    if (error) {
      response.status(404).end();
    } else {
      marked(data, (error, content) => {
        if (error) {
          console.log(error);
          response.status(500).send(error);
        } else {
          response.send(content);
        }
      })
    }
  })
});

// watch for file system changes.  when data.txt changes, send converted
// markdown output to all /html connections.
fs.watch(dirname, {}, (event, filename) => {
  if (filename == 'data.txt') {
    fs.readFile(filename, 'utf8', (err, data) => {
      if (data && !err) {
        marked(data, (err, content) => {
          if (!err) {
            for (connection of connections.html) {
              connection.sendUTF(content);
            }
          }
        }) 
      }
    })
  }
})

The salient difference between this example and the previous example is call to the marked function to perform the conversion.

If you visit http://localhost:8080/markdown, you will see the text converted to markdown. You can also visit http://localhost:8080/ to see both of these demos side by side, in separate frames. Updates make in the window on the left will be reflected on the right.

No changes were required to the first demo to make this happen as both demos watch for file system changes. In fact, you can edit data.txt on the server with your favorite text area and whenever you save your changes all clients will be updated.

Final example - JSON

In this final example, the server will be sending down a recursive directory listing, complete with file names, sizes, and last modified dates. On the client, Vue.js will be used to present the data. We start with a template:

<tbody>
  <tr v-for="file in filelist">
    <td>{{ file.name }}</td>
    <td>{{ file.size }}</td>
    <td>{{ file.mtime }}</td>
  </tr>
</tbody>

And add a bit of code:

var app = new Vue({el: 'tbody', data: {filelist: []}});

fetch('filelist.json').then((response) => {
  response.json().then((json) => { app.filelist = json });
});

subscribe('json', (data) => { app.filelist = JSON.parse(data) });

The first line associates some data (initially an empty array) with an HTML element (in this case tbody). The remaining code should look very familiar by now. Because of the way Vue.js works, all that is required to update the display is to update the data.

The server side should also seem pretty familiar:

app.get('/dir.json', (request, response) => {
  response.json(stats(dirname));
});

fs.watch(dirname, {recursive: true}, (event, filename) => {
  var data = JSON.stringify(stats(dirname));
  for (connection of connections.json) {
    connection.sendUTF(data)
  }
})

Not shown is the code that extracts the information from the filesystem, the rest is the same basic pattern that has been used for each of these demos.

If you visit http://localhost:8080/filelist, you will see a table showing each of the files on the server. This list will be updated whenever you create, delete, or update any file. The server will push a new (and complete) set of data, and Vue.js will determine what needs to be changed in the browser window. All this generally takes place in a fraction of a second.

Vue.js is only one such framework that can be used in this way. Angular, Ember.js, and React are additional frameworks that are worth exploring.

Recap

By focusing on file system modified events, these demos have tried to demonstrate server initiated updates.

With comparatively little code, web sites can be prepared to receive and apply unsolicited updates from the server. The granularity of the updates can be as little as a single string, can be a HTML fragment, or can be arbitrary data encoded in JSON.

Reserving web sockets for server initiated broadcast operations can keep your code small and understandable. Traditional HTTP GET and POST requests can be used for all client initiated retrieval and update operations.

This makes the division of labor between the client and server straightforward: the server is responsible for providing state -- both on demand and as the state changes. The client is responsible for updating the view to match the state.


Achieving Response Time Goals with Service Workers

Blending cache and live responses in order to achieve response time goals.

...


Converting to Vue.js

I’m in the process of converting four Whimsy applications from React.js to Vue; and I’m taking a moment to jot down a list of things I like a lot, things I find valuable, things I dislike (but can work around), and things I’m not using.

On balance, so far I like Vue better than React.js (even ignoring licensing issues) or Angular.js, and am optimistic that Vue will continue to improve.

...


Badges? We don't need no stinkin' badges!

I found myself included in an IBM Resource Action ("RA").  I’m fine, nothing has changed.  I’m already working with a non-profit, namely the Apache Software Foundation, and find my work there to be very rewarding.

...


Service Workers - First Impressions

Created by potrace 1.13, written by Peter Selinger 2001-2015

Cache put and match worked right the first time; cache keys not so much. Authentication is a mystery. Outline of future plans, and a call for help.

...


FacePalm

Automated Publishing with Instant Articles

<description> A summary of your article, in plain text form.

<pubDate> The date of the article’s publication, in ISO-8601 format.

Related: plaintext, May Day, June Bug, Another Month, and numerous others.


Brief history of the ASF Board Agenda tool

the current implementation is a lot more fun to develop and easier to maintain than prior versions.  As an example, if it were decided that the moment the secretary clicked the ‘timestamp` button on the 'Call to order’ page, all comment buttons are to be removed from all windows and all comment modal dialogs are to be closed, this could be implemented using a single if statement as the event is already propagated, and a re-render is already triggered.  All that would be required is to change the conditions under which the comment button appears.

The board agenda tool has been tested on Linux, Mac OS/X, Vagrant, and Docker.  It contains a suite of tests.

...


Spartan Test Results

I replaced IE results with Spartan results in my urltests.  Other than the user agent string, nothing changed.

Following are selected examples where three out of four of the top browsers agree, identified by the odd browser out:


Ruby2JS 2.0

I’ve released Ruby2JS version 2.0.  Key new features:

The Whimsy Agenda rewrite-in-progress (previously based on Angular.js, now being rebased on React.js) can be used to explore both of these features.


React.rb updates

I’ve made a number of updates to the demos.  The tutorial demo has been updated to do server side rendering.  This means that it is able to be used by clients which either don’t support or have turned off JavaScript. 

The second demo is a calendar.  Unlike the tutorial which is a single file, this application is organized in a manner more consistent with how I expect projects to be organized.

...


DSL for JavaScript

W

Jeremy Ashkenas: “work towards building a language that is to ES6 as CoffeeScript is to ES5”… close, but—do it for [ES6+HTML+CSS], and you’ll win ;)

It occurs to me that there is a shortcut available.  Let a library like React replace [ES6+HTML+CSS].  Then build a DSL for that library.

...


Web Components

Brian Leroux: ES6 and Web Components

My take is that this talk lumps React in with others based on when it was introduced; but that it is fundamentally different from, say Angular.js as Angular.js is from jQuery.

...


Email addresses

I have been telling all non-IBMers to not use my ibm.com email address for years, but this advice is routinely ignored.  I’ve repeated the reaons behind why I ask this enough times that it makes sense for me to post the reasons in one place so that I can point to it.

...


React.rb

Having determined that Angular.js is overkill for my blog rewrite, I started looking more closely at React.  It occurred to me that I could do better than JSX, so I wrote a Ruby2JS filter.  Compare for yourself.

...


RFC 3986bis

URL parsers consume URLs and generate URIs.  Such URIs are not RFC 3986 complaint.  I’d like to fix that.

...


URL Work Status

I have test results that show that there is much work to be done.

The most likely path forward at this point is to get representatives from browser vendors into a room and go through these results and make recommendations. This likely will happen in the spring, and in the SF Bay Area. With that in place, I can work with authors of libraries in popular programming languages to produce web-compatible versions. This work will take the form of bug reports, patches, or — when required — authoring new libraries.

...


Ununzippable Modern.IE

I’ve downloaded the multi-part zip archive for IE11 on Win10 for VirtualBox on OS/X from modern.ie.  I’ve downloaded the single-file archive on both OS/X and Linux.  I’ve verified the md5 signatures for each.  Yet each time, when I try to unzip the result, I fail.

...


New PhantomJS and Capybara fan

While I’m clearly late to the party, I’ve already become a huge fan of capybara and phantomjs.  I’m now using both with my previously mentioned blogging software rewrite.

My original intent was to aggressively prune unnecessary function with the intent of producing a more maintainable result, but with the ability to have automated acceptance tests, this is now less of a concern.


Apple Apostasy

Looks like Why I quit OS X struck a nerve — it is currently down (see web archive).  Also good: Apple has lost the functional high ground.

I particularly like the comment that “It just works” was never completely true.  My experience is that when working with open source codebases, doing so on an Linux operating system comes much closer to “It just works” than doing so on any other.


Rack broke Sinatra

Not rack’s fault, but Sinatra hasn’t released in a while.  Problem has been known since July, and a fix was merged into master in August.  One possible workaround has been posted.  An alternate workaround:

module Rack
  class ShowExceptions
    alias_method :old_pretty, :pretty
    def pretty(*args)
      result = old_pretty(*args)
      def result.join; self; end
      def result.each(&block); block.call(self); end
      result
    end
  end
end