All about long polling (Extra Reading)
Push technology, is a style of internet based communication where the request for a given transaction is initiated by the server, rather than the client. This is especially useful for applications that require real time updates.
Long polling is not a true push, but given the restricted web environment where incoming HTTP requests are restricted, it emulates the push function sufficiently well.
With long poll, the client requests information in exactly the same way. If the server does not have the information required, instead of sending an empty response, the server holds the request until the information becomes available. Once the response is sent to the client, another request is fired again until the server returns with the information requested.
Live Messageboard, or (what we will be doing)
The application will be done with web.py in the back end, serving the web pages for index / and the message submission page /add, alongside responses to http request /get for new updates.
The messages submitted will be stored on a sqlite database, chosen for its simplicity and size. The request send to /get will include a timestamp, whereby only messages sent after that timestamp will be return.
The JSON response will then be received by the client, after the page has been updated, another XHR will be fired to the server, awaiting new updates for the message board
Frontend request firing script
So, in this tutorial, we will be using a standard XMLHttpRequest to send request to the server to obtain any new updates on the message board. creating a generic function startConn function which passes the JSON information sent from the server into a callback function, then sends another request to the server using the link returned from the callback function.
Save all these into static/conn.js:
function startConn(link, callback){
var conn = new XMLHttpRequest();
conn.open("GET", link);
conn.onreadystatechange = function(){
if(conn.readyState==4){
var link_new = callback(eval("("+conn.responseText+")"));
if(!link_new){
link_new = link;
}
startConn(link_new, callback);
}
};
conn.send();
return conn;
}
Setup Database
Using sqlite3, create a table called messages with a few fields, mainly the message content msg_content and the timestamp, for retrieval purposes, msg_time.
CREATE TABLE messages(
msg_id INTEGER PRIMARY KEY,
msg_content TEXT NOT NULL,
msg_time INTEGER NOT NULL
);
I have saved the database in data.db (you could choose your own name). So, we can start on the main server file main.py:
import web, time, json
render = web.template.render("")
urls = [
"/", "Index"
]
class Index:
def GET(self):
return "<h1>Hello World!</h1>"
app = web.application(urls, globals())
db = web.database(dbn="sqlite", db="data.db")
if __name__ == "__main__":
app.run()
This creates a web.py app, which is run on port 8080 (by default). In any browser, go to localhost:8080 and you will see a header welcoming you into the world.
Adding Messages
Next we will work on the interface for submitting messages to the server. This will just be a simple form where there is a textarea for typing messages and a simple submit button. Create this file as form.html under the same directory as the main server file.
<html>
<body>
<form action="/add" method="POST">
<textarea type="text" name="s" style="width:500px; height:400px;"></textarea>
<input type="submit" value="Send It In" />
</form>
</body>
</html>
For the main server script, we will add a few lines to serve this form.html when users visit the page /add, then we will also add a POST function so as to retrieve the message content and then put it into our database.
import web, time, json
render = web.template.render("")
urls = [
"/", "Index",
"/add", "MsgAdd"
]
class Index:
def GET(self):
return "<h1>Hello World!</h1>"
class MsgAdd:
def POST(self):
## Check if the content is empty
s = web.input().get("s")
if not s:
web.seeother("/add")
## Insert the message into the database
db.insert("messages",
msg_time = str(int(time.time()*1000)),
msg_content = s
)
web.seeother("/add")
def GET(self):
## Show contents of form.html
return render.form()
app = web.application(urls, globals())
db = web.database(dbn="sqlite", db="data.db")
if __name__ == "__main__":
app.run()
The Messageboard itself
The main page, when first visited will show all the messages, then connects to the server to see if there is any new messages to load. To do this, we make use of web.py’s templating system to create a index.html, and dynamically load all the message content onto the website when it loads.
$def with (msgs)
<html>
<head>
<title>Message Board</title>
<script src="static/conn.js"></script>
</head>
<style>
body, div{
margin:0;
padding:0;
}
.msg{
padding:5px 8px;
border-bottom:1px solid #000000;
cursor:pointer;
}
.msg:hover{
background:#DDD;
}
</style>
<body>
<div id="main">
$for msg in msgs:
<div class="msg">
$msg["content"]<br />
<div style="text-align:right;">$msg["time"]</div>
</div>
</div>
</body>
</html>
To retrieve the message content, we write a function inside main.py to retrieve the content and then parse it into an object:
def loadMsgs(msgs):
payload = []
for msg in msgs:
payload.append({
"content":msg["msg_content"],
"time":time.strftime("%Y-%m-%d %H:%m:%S %p", time.gmtime(msg["msg_time"]/1000))
})
return payload
Then to serve this page, we now make a few edits to the Index class of the server script:
class Index:
def GET(self):
msgs = db.select("messages")
return render.index(loadMsgs(msgs)[::-1])
Getting new messages
To retrieve new messages, a request is fired to the server and it searches through the database for new messages. And here is the important part, when the server sees that there is no new message, it simplys wait and search again later. To relieve load, a staggering effect is used, where with each failure, the staggering time is increased, until a valid response is received This is how the long polling is achieved.
urls = [
"/", "Index",
"/get", "MsgGet",
"/add", "MsgAdd"
]
class MsgGet:
def GET(self):
t = web.input().get("t")
if not t:
raise web.notfound()
msgs = []
t_slp = 1
t_add = 1
t_max = 20
while not len(msgs):
if type(msgs) != "list":
time.sleep(t_slp)
t_slp = min(t_slp+t_add, t_max)
msgs = db.select("messages", where="msg_time>"+t)
msgs = [dict(msg) for msg in msgs]
return json.dumps({
"msgs":loadMsgs(msgs)
})
Now we create a callback function to handle the request send back by the server and put it inside static/main.js, remembering to add it to index.html:
function loadMsgs(obj){
// Format the information into html and add onto the page
var html = "";
for(var i=0; i<obj["msgs"].length; i++){
var msg = obj["msgs"][i];
var div = '<div class="msg">' +
msg["content"] + '<br />' +
'<div style="text-align:right;">' +
msg["time"] + '</div></div>';
html += div;
}
var old = document.getElementById("main").innerHTML;
document.getElementById("main").innerHTML = html + old;
// return an updated link with the current time
return "/get?t=" + d.getTime().toString();
}
When the page starts, we will trigger the connection using the startConn() function that we have written in conn.js by editing the onload function:
<body onload="var d = new Date(); startConn('/get?t='+d.getTime().toString(), loadMsgs);">
Improvements to make, places to go
Well, with everything done, save all the files, then run the server file. Add a message on localhost:8080/add and then watch the index page refreshes and updates all the messages automatically.
However, the limited webserver that web.py uses means that it is unlikely that it is capable of supporting multiple long poll request at a time. So next time, I will be teaching you how to setup web.py with lighttpd to handle these request smoothly.
Development 