When creating an API that delivers data that is changing rapidly or where speed is very much of the essence a real-time component soon becomes a necessity. Traditional ‘request-response’ technologies where the client makes a request, hangs around for a response and then processes it adds a huge overhead both in terms of response time but also on resources spent on wasted requests polling for changes when there are none. Indeed Zapier, a service that connects together APIs, estimated that they could reduce load by 66x by implementing a real-time component and eliminating polling.

Real-Time Streaming Technologies

Fortunately there are a whole array of real-time streaming technologies and an equally large number of open source projects and ‘real-time as a service’ companies to handle the requirements of these APIs. I’ll outline some of the major delivery mechanisms below.

Webhooks are one of the simplest techniques to implement and involves the server making a POST request to the client’s server whenever there is new data, eliminating the need for client polling. The data is easily processed by the client as it is the same as responding to any traditional POST request so the barrier to entry for this method is very low.

Long polling is a variation of the traditional polling technique where the client polls the server for changes but the server, instead of immediately responding, waits until there is new data to send. On receiving the data the client immediately makes another request, repeating the cycle. Whilst less glamorous than other technologies long polling certainly has many benefits and due to its ease of implementation by the client will often suffice for a large number of scenarios.

Websockets, Server Sent Events and HTTP Streaming all create a persistent connection between a client and server that can be used to send data in real time.

Websockets are ideal for building browser-based applications as they are part of the HTML5 standard and as such are increasingly well supported. They allow full duplex communication between the client and server and use a ws:// or wss:// prefix. Server Sent Events, also a HTML5 standard, by contrast is a one-way communication mechanism from server to client albeit one that offers a number of other advantages over Websockets particularly when one-way communication is all that is required.

HTTP Streaming as is used by the popular Twitter Streaming API is where new data is simply pushed to the client over a persistent HTTP connection. Due to the complexities of handling the connection, incoming data and re-connection it’s a more complex system to implement for the client.

Available Real-Time Solutions

Real-time is big business and there are a plethora of hosted services available. The behemoths in the field are Firebase (recently acquired by Google), Pubnub and Pusher. All of these services support numerous environments and are often implemented into an application with a few lines of javascript.

If you fancy doing it yourself you won’t be disappointed with the likes of the open-source socket.io. For PHP developers the Ratchet library is an example websocket implementation.

My need for a real-time API was derived from an application that I was developing (cbix.ca). The Canadian Bitcoin Index (cbix.ca) is a blended bitcoin index derived from a volume-weighted moving average from all trades conducted by Canadian exchanges. As the Index was designed to be a publicly available resource the index, and trades which constituted it, were to be published on a public API. In the Bitcoin ecosystem websockets are becoming the norm for real-time information with the newly launched regulated exchange Coinbase adopting the technology alongside well-funded block explorer applications such as Chain, Blockchain.info and Toshi.io. The issue that I had was that there was no way of combining the convenience of a hosted solution such as Pusher with the flexibility of a public API provided by a self-hosted solution. This is where Fanout.io fits in.

Fanout is unique in a couple of senses. First it lets users have control over how they present the API and secondly it offers a wide variety of methods over how the real-time information is delivered. The first allows for custom domains such as wss://socket.cbix.ca to be used to present the public API and the second means minimal additional effort is required to support a wide variety of delivery mechanisms.They currently support Websockets, Webhooks, Server Sent Events, HTTP Long-polling/Streaming and XMPP as well as their own FPP protocol (which is implemented by the client in a similar fashion to Pusher et al.).

In their own words “Fanout acts as a reverse proxy, Fanout’s global cloud invisibly bridges realtime clients with the services you’ve already built”. Or to put another way, Fanout’s service manages the persistent client connections and all you have to do is notify Fanout when there is data to send out. Whilst the mechanism for responding to Fanout and delivering messages varies slightly depending on which delivery mechanism(s) you choose to use, it’s usually just a few additional lines of code to support any method.

Implementing Fanout

The rest of this post details how to configure a HTTP Streaming and Websocket API using Fanout and PHP. Example implementations of both may be found on the cbix.ca API docs.

cbix.ca is a Laravel application so the below code uses some Laravel specific shorthand methods for ease that will need to be substituted for other frameworks. We are going to be simulating the case of publishing a new trade to both a Websocket and HTTP Streaming API.

Initial Fanout Configuration

Upon signing up with Fanout up you will need to configure a ‘realm’. A realm is simply a container for your channels to which messages are published. Within each realm there is a realm id and realm key which are needed to publish to channels and authenticate any requests to Fanout.

We will configure two custom API domains on the Fanout control panel to handle our HTTP Streaming and Websocket APIs namely streaming.cbix.ca and socket.cbix.ca. These domains will need a CNAME created to point to the Fanout service and then Fanout will act as a reverse proxy getting the intial connection setup information from our origin server before opening a persistent connection to the client. Note, that you don’t need to use subdomains for custom API domains and we could for example have used cbixstreamingapi.com if it made sense.

When adding a new domain with Fanout via their control panel you are presented with a form as below that allows us to map a domain to a target origin server and also upload a SSL certificate such that https:// and wss:// are both supported. Just be aware if you are responding on a subdomain you will need a valid certificate for all domains used in the request including the origin server. To use Websockets simply check the ‘Origin Server Uses Websocket-Over-HTTP Protocol’ and leave it unchecked for a HTTP streaming API.

All of this can also be managed over Fanout’s API is you prefer that method. Once we have added the streaming.cbix.ca and socket.cbix.ca domains and our CNAMEs have been setup we move on to how to respond to an incoming connection from Fanout.

Creating a Connection with Fanout

Now when someone requests the streaming.cbix.ca/trades endpoint Fanout expects a response from our origin server that tells it what connection type we are opening and to which channels to subscribe the client to. How we respond depends on the transport mechanism we wish to support. In our case a request to streaming.cbix.ca/trades will create a subscription to the trades channel via a HTTP Streaming API. Similarly socket.cbix.ca/trades would open a Websocket connection and subscribe the client to the trades channel. In Laravel you can use domain routing to respond to the same route on different subdomains such that your origin server can be a single machine and you can design the routes as you wish.

To create a response for a HTTP Streaming API at streaming.cbix.ca/trades create a domain route that has a response as follows taking particular note of the headers and the channel(s) that we wish to subscribe the client to. We can also pass an optional message that will be sent to the client when the connection is opened.

Route::group(array('domain' => 'streaming.cbix.ca'), function()
{
    Route::get('/trades', function() {
    	$content = [
            "hold" => [
                "mode" => "stream",
                "channels" => [
                    [
                        "name" => "trades"
                    ]
                ]
            ],
            "response" => [
                "headers" => [
                    "Content-Type" => "text/plain"
                ],
                "body" => "An optional connection message here"
            ]
        ];
	    
	    $statusCode = 200;
	    $response = Response::make($content, $statusCode);
	    $response->header('Content-Type', 'application/grip-instruct');
	    return $response;
    });
});

If successful there is a now an open connection between the client and Fanout and any data pushed to Fanout on the trades channel using the http-stream message type will be delivered to all connected clients. Similarly when opening a Websocket connection the origin server at socket.cbix.ca/trades will respond as follows:

Route::group(array('domain' => 'socket.cbix.ca'), function()
{
	Route::get('/trades', function() {
    	$contents = <<<EOD
    	OPEN\r\nTEXT 29\r\nc:{"type":"subscribe","channel":"trades"}\r\n
    	EOD;

    	$statusCode = 200;
    	$response = Response::make($contents, $statusCode);
    	$response->header('Content-Type', 'application/websocket-events');
    	$response->header('Sec-WebSocket-Extensions','grip');
    	return $response;
    });
});

Fanout is very particular about this response and you should take care not to introduce additional carriage returns or whitespace. The 29 following the TEXT is a hex representation of the amount of data to follow in the c:{"type":"subscribe","channel":"trades"} object. As this is 41 bytes it is equivalent to a hex value of 29. Yup, it’s all a bit involved but once the route is set up there is nothing more to worry about. Now going to wss://socket.cbix.ca/trades will open up a Websocket connection subscribed to the trades channel waiting for you to push data to it.

Sending Data

In the cbix.ca application when a new trade is recorded an event is fired that publishes the data to Fanout. All messages to be delivered need to be authenticated via JSON web tokens. To do this we’ll use the handy JWT library which you can install via composer "firebase/php-jwt": "dev-master". To create a new token simply substitute your realm key into the following snippet. The following example creates a token that is valid for an hour such that it could be used for future requests without the need for regenerating it with every message.

$key = base64_decode("YOUR_KEY");

$settings = array(
    "iss" => "8ebb747b",
    "exp" => time() + 3600
);

$token = JWT::encode($settings, $key, 'HS256');

Once we have a valid token we will publish a message to both the Streaming and Websocket endpoints simultaneously. For simplicity we’ll use the Unirest library to deliver our message. Once again it is installable via Composer "mashape/unirest-php": "dev-master". For the Streaming API we’ll delimit our messages with a /r/n as exemplified on the Twitter Streaming API to make the identification of message boundaries easier. The URI to publish to includes the realm id and channel which will need to be substituted as appropriate below.

//Example trade data to publish
$trade_data= [
        "exchange" => "Cavirtex",
        "volume" => "1.89050000",
        "price" => "293.47",
        "value" => "554.80",
        "date" => "2015-01-27T16:17:57+00:00",
        "transaction_id" => 55235
];

$message = json_encode([
        "items" => [
            [
                "ws-message" => [
                    "content" => "{$trade_data}"
                ],
                "http-stream" => [
                    "content" => "{$trade_data}\r\n"
                ],
            ]
    ]);
Unirest::defaultHeader("Authorization", "Bearer {$token}");
Unirest::defaultHeader("Content-Type", "application/json");
$response = Unirest::post("https://api.fanout.io/realm/YOUR_REALM_ID/publish/trades/", null, $message);

And that is it! We have a fully functioning Websocket and HTTP Streaming API that will scale as needed. We simply repeat the logic for sending any new data and multiple items (trades) can be sent with each request.

Conclusion

If you want to offer a public real-time API and don’t want to be concerned with the complexities of managing the infrastructure then Fanout is certainly worth a look. Of course no solution is perfect and currently it lacks any metrics to track usage and the docs can feel a little daunting. All being said given their wide array of supported delivery mechanisms there is likely something to fit all needs!