intertwingly

Preface

You've seen web sites with stock prices or retweet counts that update in real time. However, such sites are more the exception rather than the norm. WebSockets make it easy, and are widely supported, but not used as much as they could be.

Examples provided for WebSockets typically don't focus on the "pubsub" use case; instead they tend to focus on echo servers and the occasional chat server. These are OK as far as they go.

This post provides three mini-demos that implement the same design pattern in JavaScript on both the client and server.

Quick Start

For the impatient who want to see running code,

git clone https://github.com/rubys/websocket-demo.git
cd websocket-demos
npm install
node server.js

After running this, visit http://localhost:8080/ in a browser, and you should see something like this:

Server support

The primary responsibility of the server is to maintain a list of active websocket connections. The code below will maintain three such sets, one for each of the demos provided.

// attach to web server
var wsServer = new websocket.server({httpServer: httpServer});

// three sets of connections
var connections = {
  text: new Set(),
  html: new Set(),
  json: new Set()
};

// when a request comes in for one of these streams, add the websocket to the
// appropriate set, and upon receipt of close events, remove the websocket
// from that set.
wsServer.on('request', (request) => {
  var url = request.httpRequest.url.slice(1);

  if (!connections[url]) {
    // reject request if not for one of the pre-identified paths
    request.reject();
    console.log((new Date()) + ' ' + url + ' connection rejected.');
    return;
  };

  // accept request and add to the connection set based on the request url
  var connection = request.accept('ws-demo', request.origin);
  console.log((new Date()) + ' ' + url + ' connection accepted.');
  connections[url].add(connection);

  // whenever the connection closes, remove connection from the relevant set
  connection.on('close', (reasonCode, description) => {
    console.log((new Date()) + ' ' + url + ' connection disconnected.');
    connections[url].delete(connection)
  })
});

The code is fairly straightforward. Three sets are defined; and when a request comes in it is either accepted or rejected based on the path part of the URL of the request. If accepted, the connection is added to the appropriate set. When a connection is closed, the connection is removed from the set.

EZPZ!

Client Support

The client's responsibitlity is to open the socket, and to keep it open.

function subscribe(path, callback) {    
  var ws = null;
  var base = window.top.location.href

  function openchannel() {
    if (ws) return;
    var url = new URL(path, base.replace('http', 'ws'));
    ws = new WebSocket(url.href, 'ws-demo');

    ws.onopen = (event) => {
      console.log(path + ' web socket opened!');
    };

    ws.onmessage = (event) => {
      callback(event.data);
    };

    ws.onerror = (event) => {
      console.log(path + ' web socket error:');
      console.log(event);
      ws = null;
    };

    ws.onclose = (event) => {
      console.log(path + ' web socket closed');
      ws = null;
    }
  }

  // open (and keep open) the channel
  openchannel();
  setInterval(() => openchannel(), 2000);
}

A subscribe method is defined that accepts a path and a callback. The path is used to construct the URL to open. The callback is called whenever a message is received. Errors and closures cause the ws variable to be set to null. Every two seconds, the ws variable is checked, and an attempt is made to reestablish the socket connection when this value is null.

First example - textarea

Now it is time to put the sets of server connections, and client subscribe function to use.

Starting with the client:

var textarea = document.querySelector('textarea');

// initially populate the textarea with the contents of data.txt from the
// server
fetch("/data.txt").then((response) => {
  response.text().then((body) => { textarea.value = body })
});

// whenever the textarea changes, send the new value to the server
textarea.addEventListener('input', (event) => {
  fetch("/data.txt", {method: 'POST', body: textarea.value});
});

// whenever data is received, update textarea with the value
subscribe('text', (data) => { textarea.value = data });

The value of the textarea is fetched from the server on page load. Changes made to the textarea are posted to the server as they occur. Updates received from the server are loaded into the textarea. Nothing to it!

Now, onto the server:

// Return the current contents of data.txt
app.get('/data.txt', (request, response) => {
 response.sendFile(dirname + '/data.txt');
});

// Update contents of data.txt
app.post('/data.txt', (request, response) => {
 var fd = fs.openSync(dirname + '/data.txt', 'w');
 request.on('data', (data) => fs.writeSync(fd, data));
 request.on('end', () => {
   fs.closeSync(fd);
   response.sendFile(dirname + '/data.txt');
 })
})

// watch for file system changes.  when data.txt changes, send new raw
// contents to all /text connections.
fs.watch(dirname, {}, (event, filename) => {
  if (filename == 'data.txt') {
    fs.readFile(filename, 'utf8', (err, data) => {
      if (data && !err) {
        for (connection of connections.text) {
          connection.sendUTF(data)
        };
      }
    })
  }
})

Requests to get data.txt cause the contents of the file to be returned. Post requests cause the contents to be updated. It is the last block of code that we are most interested in here: the file system is watched for changes, and whenever data.txt is updated, it is read and the results are sent to each text connection. Pretty straightforward!

If you visit http://localhost:8080/textarea in multiple browser windows, you will see a textarea in each. Updating any one window will update all. What you have is the beginning of a collaborative editing application, though there would really need to be more logic put in place to properly serialize concurrent updates.

Second example - markdown

The first example has the server sending plain text content. This next example deals with HTML. The marked package is used to convert text to HTML on the server.

This client is simpler in that it doesn't have to deal with sending updates to the server:

// initially populate the textarea with the converted markdown obtained
// from the server
fetch("/data.html").then((response) => {
  response.text().then((body) => { document.body.innerHTML = body })
});

// whenever data is received, update body with the data
subscribe('html', (data) => { document.body.innerHTML = data });

The primary difference between this example and the previous one is that the content is placed into document.body.innerHTML instead of textarea.value.

Like the client, the server portion of this demo consists of two blocks of code:

app.get('/data.html', (request, response) => {
  fs.readFile('data.txt', 'utf8', (error, data) => {
    if (error) {
      response.status(404).end();
    } else {
      marked(data, (error, content) => {
        if (error) {
          console.log(error);
          response.status(500).send(error);
        } else {
          response.send(content);
        }
      })
    }
  })
});

// watch for file system changes.  when data.txt changes, send converted
// markdown output to all /html connections.
fs.watch(dirname, {}, (event, filename) => {
  if (filename == 'data.txt') {
    fs.readFile(filename, 'utf8', (err, data) => {
      if (data && !err) {
        marked(data, (err, content) => {
          if (!err) {
            for (connection of connections.html) {
              connection.sendUTF(content);
            }
          }
        }) 
      }
    })
  }
})

The salient difference between this example and the previous example is call to the marked function to perform the conversion.

If you visit http://localhost:8080/markdown, you will see the text converted to markdown. You can also visit http://localhost:8080/ to see both of these demos side by side, in separate frames. Updates make in the window on the left will be reflected on the right.

No changes were required to the first demo to make this happen as both demos watch for file system changes. In fact, you can edit data.txt on the server with your favorite text area and whenever you save your changes all clients will be updated.

Final example - JSON

In this final example, the server will be sending down a recursive directory listing, complete with file names, sizes, and last modified dates. On the client, Vue.js will be used to present the data. We start with a template:

<tbody>
  <tr v-for="file in filelist">
    <td></td>
    <td></td>
    <td></td>
  </tr>
</tbody>

And add a bit of code:

var app = new Vue({el: 'tbody', data: {filelist: []}});

fetch('filelist.json').then((response) => {
  response.json().then((json) => { app.filelist = json });
});

subscribe('json', (data) => { app.filelist = JSON.parse(data) });

The first line associates some data (initially an empty array) with an HTML element (in this case tbody). The remaining code should look very familiar by now. Because of the way Vue.js works, all that is required to update the display is to update the data.

The server side should also seem pretty familiar:

app.get('/dir.json', (request, response) => {
  response.json(stats(dirname));
});

fs.watch(dirname, {recursive: true}, (event, filename) => {
  var data = JSON.stringify(stats(dirname));
  for (connection of connections.json) {
    connection.sendUTF(data)
  }
})

Not shown is the code that extracts the information from the filesystem, the rest is the same basic pattern that has been used for each of these demos.

If you visit http://localhost:8080/filelist, you will see a table showing each of the files on the server. This list will be updated whenever you create, delete, or update any file. The server will push a new (and complete) set of data, and Vue.js will determine what needs to be changed in the browser window. All this generally takes place in a fraction of a second.

Vue.js is only one such framework that can be used in this way. Angular, Ember.js, and React are additional frameworks that are worth exploring.

Recap

By focusing on file system modified events, these demos have tried to demonstrate server initiated updates.

With comparatively little code, web sites can be prepared to receive and apply unsolicited updates from the server. The granularity of the updates can be as little as a single string, can be a HTML fragment, or can be arbitrary data encoded in JSON.

Reserving web sockets for server initiated broadcast operations can keep your code small and understandable. Traditional HTTP GET and POST requests can be used for all client initiated retrieval and update operations.

This makes the division of labor between the client and server straightforward: the server is responsible for providing state -- both on demand and as the state changes. The client is responsible for updating the view to match the state.