Ajax based streaming and progress monitor

Streaming (Comet)

Streaming means to send content part by part at some intervals in the same request. Or in other words, the client is able to process the response part by part, instead of waiting for the whole transfer to complete. In the context of http and web applications it is more specifically called Comet. The wikipedia page defines it as

Comet is a web application model in which a long-held HTTP request allows a web server to push data to a browser, without the browser explicitly requesting it. Comet is an umbrella term, encompassing multiple techniques for achieving this interaction. All these methods rely on features included by default in browsers, such as JavaScript, rather than on non-default plugins. The Comet approach differs from the original model of the web, in which a browser requests a complete web page at a time.

The best tool for streaming over the http protocol is SSE or Server Sent Events. But SSE is not available IE 10 so far. While looking for a clean solution (that is one without much hacks) for IE, I came across this hidden feature of XmlHttpRequest object that many browsers seemed to support.

First lets look at a conventional ajax request to get a greater depth.

var oReq = new XMLHttpRequest();
oReq.onreadystatechange = function()
{
	if (this.readyState == 4 && this.status == 200) 
	{
		var response = this.responseText;
		alert(response);
	}
}
oReq.open("get", "http://localhost/", true);
oReq.send();

The response is supposed to be available only when the ajax request completes fully. Now lets say that the server is sending out contents at some delayed interval, such that each part of data received needs to be reported to the user. Consider a long running task on the server that echoes its progress and the client side needs to show the progress report in realtime.

echo '10% done';

... processing for 1 minute... 

echo '20% done';

... processing for 1 minute...

echo '30% done';

... more processing

It makes sense only if the user is able to see the progressbar grow from 0 to 100% in realtime and not just 100% after a long wait. Some common techniques to achieve this goal are iframe streaming and ajax polling. But both of them require lot of setup.

So now lets talk about the strange feature that I mentioned a while ago. It is possible to make the XMLHttpRequest object report on each chunk of data received. And the trick is very very simple.







oReq.onreadystatechange = function()
{
	if (this.readyState > 2) 
	{
		var partial_response = this.responseText;
	}
}

Thats it! Simple, neat and clean. Now everytime some data is received by the xhr object, the onreadystatechange event is triggered and this.responseText will have the data received so far. The partial response can be processed to report the user of the progress of the server side task.

Lets take a look at a complete demo of what we can achieve with this. Click the start button to start receiving server side data.

Demo

On chrome, ff and ie10 you should notice that progress bar moving from start to end in an incremental manner. However in older version of ie and opera the progressbar would come to 10% and then after a wait for 10 seconds it would hit 100%.

Code

Server side

ajax_stream.php

<?php
/**
	Ajax Streaming without polling
*/

//type octet-stream. make sure apache does not gzip this type, else it would get buffered
header('Content-Type: text/octet-stream');
header('Cache-Control: no-cache'); // recommended to prevent caching of event data.

/**
	Send a partial message
*/
function send_message($id, $message, $progress) 
{
	$d = array('message' => $message , 'progress' => $progress);
	
	echo json_encode($d) . PHP_EOL;
	
	//PUSH THE data out by all FORCE POSSIBLE
	ob_flush();
	flush();
}

$serverTime = time();

//LONG RUNNING TASK
for($i = 0; $i < 10; $i++)
{
	//Hard work!!
	sleep(1);
	
	//send status message
	$p = ($i+1)*10;	//Progress
	
	send_message($serverTime, $p . '% complete. server time: ' . date("h:i:s", time()) , $p); 
}
sleep(1);
send_message($serverTime, 'COMPLETE'); 

The above script runs for about 10 seconds and echoes chunks of output at one second interval. Every chunk is processed the moment it is received on the client side. The content type has been kept octet-stream to make it different from normal html content. Apache for example might be configured to compress html output using gzip/deflate. So mentioning a different content type helps keep the output free from such effects.

ob_flush and flush are called immediately after echoing the data to ensure that the output is pushed out and send to the client without any buffering.

Client Side

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html dir="ltr" xmlns="http://www.w3.org/1999/xhtml">
<head>
	<meta content="text/html; charset=windows-1252" http-equiv="Content-Type" />
	<title>Test XHR / Ajax Streaming without polling
</title>
	<script>
	
	function doClear()
	{
		document.getElementById("divProgress").innerHTML = "";
	}
	
	function log_message(message)
	{
		document.getElementById("divProgress").innerHTML += message + '<br />';
	}
	
	function ajax_stream()
	{
		if (!window.XMLHttpRequest)
		{
			log_message("Your browser does not support the native XMLHttpRequest object.");
			return;
		}
		
		try
		{
			var xhr = new XMLHttpRequest();  
			xhr.previous_text = '';
			
			//xhr.onload = function() { log_message("[XHR] Done. responseText: <i>" + xhr.responseText + "</i>"); };
			xhr.onerror = function() { log_message("[XHR] Fatal Error."); };
			xhr.onreadystatechange = function() 
			{
				try
				{
					if (xhr.readyState > 2)
					{
						var new_response = xhr.responseText.substring(xhr.previous_text.length);
						var result = JSON.parse( new_response );
						log_message(result.message);
						//update the progressbar
						document.getElementById('progressor').style.width = result.progress + "%";
						xhr.previous_text = xhr.responseText;
					}	
				}
				catch (e)
				{
					//log_message("<b>[XHR] Exception: " + e + "</b>");
				}
				
				
			};
	
			xhr.open("GET", "ajax_stream.php", true);
			xhr.send("Making request...");		
		}
		catch (e)
		{
			log_message("<b>[XHR] Exception: " + e + "</b>");
		}
	}

	</script>
</head>

<body>
	Ajax based streaming without polling
	<br /><br />
	<button onclick="ajax_stream();">Start Ajax Streaming</button>
	<button onclick="doClear();">Clear Log</button>
	<br />
	Results
	<br />
	<div style="border:1px solid #000; padding:10px; width:300px; height:200px; overflow:auto; background:#eee;" id="divProgress"></div>
	<br />
	<div style="border:1px solid #ccc; width:300px; height:20px; overflow:auto; background:#eee;">
		<div id="progressor" style="background:#07c; width:0%; height:100%;"></div>
	</div>
</body>
</html>
 

Browser Support

The latest version of Chrome, Firefox and IE10+ have this fancy feature. Opera does not. Opera supports SSE so there is a quick escape. For older IE versions however you would have to resort to techniques like iframe script buffering.

A very old spec draft over here mentions ...

onreadystatechange - An attribute that represents a function that must be invoked when readyState changes value. The function may be invoked multiple times when readyState is 3 (Receiving). Its initial value must be null.

Note the "may be" thing. This is something browsers are not obliged to do. But chrome, firefox and ie are very courteous. However the latest specification here has no mention of it. So it still remains a question if this is really a feature or just something by chance. And whether would it be there always.

Last Updated On : 26th June 2013

Subscribe to get updates delivered to your inbox

12 Comments + Add Comment

  • Hi silver, nice script! Have you observed how it behaves on slow internet like mobile networks? It fails on JSON.parse line and it looks like the new_response doesn’t do the job. I’m wondering if there’s better way of consuming incremental json feed over sloooooow internet like 3g mobile networks. Though, it works fine when you switch to WIFI.

  • Silver, your approach is nice, however you may find useful to see my blog post on how to display task progress using the HTML progress element and ProgressEvent W3C recommendation: http://zinoui.com/blog/ajax-request-progress-bar

  • It does not work if you just use it on a standard apache/php installation. apache seems to buffer everything until the php code is done, ignoring any flush or no-cache.

    • Make sure your Apache installation is properly configured to support the `text/octet-stream` content type.

  • I’ve been working on this now and updated the script, so it works with and without heavy load. I’m trying to implement a fallback for IE, will post an update here ^^

  • >> So it still remains a question if this is really a feature or just something by chance. And whether would it be there always.
    Did you read a section about “progress” event in the spec?

  • For the record, this does NOT work in IE 8. None of the progress messages are shown, and it doesn’t even indicate that the request is complete when all of the messages have been received.

    • yes, it works on ie10+

      i updated the demo. on browsers not supporting the feature, you should see the progressbar at 100% mark once the request completes.

  • Hmmm. Microsoft claims this is not possible in IE :-/

  • Any idea what versions of Ie support this?

Leave a comment