Running webpy Projects with Lighttpd (FastCGI)

Running webpy Projects with Lighttpd (FastCGI)

In this tutorial, we will be running web.py from a lighttpd server. Existing online tutorials mainly shows how to write the config file, not fully mentioning how to solve all the weird errors that will pop up all over the place.

Setting up everything

Well, remember to add #! /usr/bin/evn python to the front of your python script, then run chmod +x /path/to/script.py.

After that, run pip install flup to install flup onto our system. This will be required to setup the FastCGI server later on.

Also, do run this command chmod 755 -R /path/to/web/app to ensure that the default server.username (www-data) is able to to access the files later on.

The Configuration File

Now here is the easy part. The .conf file tells the FastCGI server that we will be setting up where to file the file and where to get everything. The lighttpd.conf file can be created anywhere in the system and run with lighttpd -D -f /path/to/lighttpd.conf.

server.modules = (
    "mod_access",
    "mod_alias",
    "mod_accesslog",
    "mod_compress"
)

var.document-root = "/path/to/web/app"
server.document-root = document-root

static-file.exclude-extensions = (
    ".py", ".pyc"
)

compress.filetype = (
    "application/javascript",
    "text/css",
    "text/html",
    "text/plain"
)

## Settings for FastCGI and everything else
server.modules += (
    "mod_fastcgi",
    "mod_rewrite"
)

fastcgi.server = (
    "/main.py" => (
        "webpy" => (
            "socket" => "/tmp/fastcgi.socket",
            "min-procs" => 1,
            "bin-path" => "main.py",
            "bin-environment" => (
                "REAL_SCRIPT_NAME" => ""
            )
        )
    )
)

## So that you can run the website normally without the "main.py"
url.rewrite-once = (
    "^/static/(.*)$" => "/static/$1",
    "^/(.*)$" => "/main.py/$1"
)

Now cross your fingers and run lighttpd -D -f /path/to/lighttpd.conf.

Possible solution to your problems

Now, here are a list of problems that I encountered when running the server.

Error: Unable to connect to /tmp/fastcgi.socket

For this, the fix would just be to remove /tmp/fastcgi.socket, because sometimes, when lighttpd runs into a problem, it does not fully clean up after itself, hence, you will have to do it manually. Don’t worry, this don’t happen often.

Error: File/Directory Not Found (Error 127)

Even after you have checked the permission to make it executable, somehow something is just wrong. The good news is, there is a work around for this. Compile the main.py by running python -m compileall /path/to/web/app. Now edit your lighttpd.conf to use the compiled file(.pyc) instead.

fastcgi.server = (
    "/main.py" => (
        "webpy" => (
            "socket" => "/tmp/nfastcgi.socket",
            "min-procs" => 1,
            "bin-path" => "main.pyc",
            "bin-environment" => (
                "REAL_SCRIPT_NAME" => ""
            )
        )
    )
)

Any other errors

Check that you have done all the things in the setup section, because I spend half the time trying to figure out all the correct file permissions. If there is some other error, do remember, google is your good friend.

Tutorial – Uploading Web.py Projects To Heroku

Setup The Environment

So first, we have to setup the environment for heroku server. First, we nid to tell the server what python libraries to download and we can easily do that with pip, with the freeze function to get all the dependencies downloaded. Mainly, this will include web.py and whatever dependencies you have downloaded for your project.

pip freeze > requirements.txt

This saves all the dependencies into requirements.txt, a file which heroku will use to tell what to download.

Procfile

The Profile consist of all the dynos that will be requried to run your application. The most basic of all application uses one dyno only, which is the web. This dyno allows your app to run and liten to HTTP request.

web: python main.py $PORT

The variable $PORT, is a envrionment variable defined by Heroku and will result in the application attaching to the pre-defined port number.

Pushing to Heroku

Heroku uses git to manage its project. And you can copy the git link by going to your project dashboard and put the link into your project as heroku.

Copy GIT Link

git remote add heroku <link>

After that you just git push heroku master to push your project online. You will see Heroku setting up your application, downloading all the neccessary dependencies and running the dynos.

In case of error

If you receive any error about not being able to authenticate with Heroku, visit my tutorial here to learn how to setup your Git with heroku.

Tutorial – Long Polling with web.py

All about long polling (Extra Reading)

Push technology, is a style of internet based communication where the request for a given transaction is initiated by the server, rather than the client. This is especially useful for applications that require real time updates.

Long polling is not a true push, but given the restricted web environment where incoming HTTP requests are restricted, it emulates the push function sufficiently well.

With long poll, the client requests information in exactly the same way. If the server does not have the information required, instead of sending an empty response, the server holds the request until the information becomes available. Once the response is sent to the client, another request is fired again until the server returns with the information requested.

Live Messageboard, or (what we will be doing)

The application will be done with web.py in the back end, serving the web pages for index / and the message submission page /add, alongside responses to http request /get for new updates.

The messages submitted will be stored on a sqlite database, chosen for its simplicity and size. The request send to /get will include a timestamp, whereby only messages sent after that timestamp will be return.

The JSON response will then be received by the client, after the page has been updated, another XHR will be fired to the server, awaiting new updates for the message board

Frontend request firing script

So, in this tutorial, we will be using a standard XMLHttpRequest to send request to the server to obtain any new updates on the message board. creating a generic function startConn function which passes the JSON information sent from the server into a callback function, then sends another request to the server using the link returned from the callback function.

Save all these into static/conn.js:

function startConn(link, callback){
    var conn = new XMLHttpRequest();
    conn.open("GET", link);
    conn.onreadystatechange = function(){
        if(conn.readyState==4){
            var link_new = callback(eval("("+conn.responseText+")"));
            if(!link_new){ 
                link_new = link;
            }

            startConn(link_new, callback);
        }
    };
    conn.send();
    return conn;
}

Setup Database

Using sqlite3, create a table called messages with a few fields, mainly the message content msg_content and the timestamp, for retrieval purposes, msg_time.

CREATE TABLE messages(
    msg_id INTEGER PRIMARY KEY,
    msg_content TEXT NOT NULL,
    msg_time INTEGER NOT NULL
);

I have saved the database in data.db (you could choose your own name). So, we can start on the main server file main.py:

import web, time, json
render = web.template.render("")

urls = [
    "/", "Index"
]

class Index:
    def GET(self):
        return "<h1>Hello World!</h1>"

app = web.application(urls, globals())
db = web.database(dbn="sqlite", db="data.db")

if __name__ == "__main__":
    app.run()

This creates a web.py app, which is run on port 8080 (by default). In any browser, go to localhost:8080 and you will see a header welcoming you into the world.

Adding Messages

Next we will work on the interface for submitting messages to the server. This will just be a simple form where there is a textarea for typing messages and a simple submit button. Create this file as form.html under the same directory as the main server file.

<html>
    <body>
        <form action="/add" method="POST">
            <textarea type="text" name="s" style="width:500px; height:400px;"></textarea>
            <input type="submit" value="Send It In" />
        </form>
    </body>
</html>

For the main server script, we will add a few lines to serve this form.html when users visit the page /add, then we will also add a POST function so as to retrieve the message content and then put it into our database.

import web, time, json
render = web.template.render("")

urls = [
    "/", "Index",
    "/add", "MsgAdd"
]

class Index:
    def GET(self):
        return "<h1>Hello World!</h1>"

class MsgAdd:
    def POST(self):
        ## Check if the content is empty
        s = web.input().get("s")
        if not s:
            web.seeother("/add")

        ## Insert the message into the database
        db.insert("messages", 
            msg_time = str(int(time.time()*1000)),
            msg_content = s
        )

        web.seeother("/add")

    def GET(self):
        ## Show contents of form.html
        return render.form()

app = web.application(urls, globals())
db = web.database(dbn="sqlite", db="data.db")

if __name__ == "__main__":
    app.run()

The Messageboard itself

The main page, when first visited will show all the messages, then connects to the server to see if there is any new messages to load. To do this, we make use of web.py’s templating system to create a index.html, and dynamically load all the message content onto the website when it loads.

$def with (msgs)

<html>
    <head>
        <title>Message Board</title>
        <script src="static/conn.js"></script>
    </head>
    <style>
body, div{
    margin:0;
    padding:0;
}

.msg{
    padding:5px 8px;
    border-bottom:1px solid #000000;
    cursor:pointer;
}

.msg:hover{
    background:#DDD;
}
    </style>
    <body>
        <div id="main">
            $for msg in msgs:
                <div class="msg">
                    $msg["content"]<br />
                    <div style="text-align:right;">$msg["time"]</div>
                </div>
        </div>
    </body>
</html>

To retrieve the message content, we write a function inside main.py to retrieve the content and then parse it into an object:

def loadMsgs(msgs):
    payload = []
    for msg in msgs:
        payload.append({
            "content":msg["msg_content"],
            "time":time.strftime("%Y-%m-%d %H:%m:%S %p", time.gmtime(msg["msg_time"]/1000))
        })

    return payload

Then to serve this page, we now make a few edits to the Index class of the server script:

class Index:
    def GET(self):
        msgs = db.select("messages")
        return render.index(loadMsgs(msgs)[::-1])

Getting new messages

To retrieve new messages, a request is fired to the server and it searches through the database for new messages. And here is the important part, when the server sees that there is no new message, it simplys wait and search again later. To relieve load, a staggering effect is used, where with each failure, the staggering time is increased, until a valid response is received This is how the long polling is achieved.

urls = [
    "/", "Index",
    "/get", "MsgGet",
    "/add", "MsgAdd"
]

class MsgGet:
    def GET(self):
        t = web.input().get("t")
        if not t:
            raise web.notfound()

        msgs = []
        t_slp = 1
        t_add = 1
        t_max = 20
        while not len(msgs):
            if type(msgs) != "list":
                time.sleep(t_slp)
                t_slp = min(t_slp+t_add, t_max)

            msgs = db.select("messages", where="msg_time>"+t)
            msgs = [dict(msg) for msg in msgs]

        return json.dumps({
            "msgs":loadMsgs(msgs)
        })

Now we create a callback function to handle the request send back by the server and put it inside static/main.js, remembering to add it to index.html:

function loadMsgs(obj){
    // Format the information into html and add onto the page
    var html = "";
    for(var i=0; i<obj["msgs"].length; i++){
        var msg = obj["msgs"][i];
        var div = '<div class="msg">' +
            msg["content"] + '<br />' +
            '<div style="text-align:right;">' +
            msg["time"] + '</div></div>';
        html += div;
    }
    var old = document.getElementById("main").innerHTML;
    document.getElementById("main").innerHTML = html + old;

    // return an updated link with the current time
    return "/get?t=" + d.getTime().toString();
}

When the page starts, we will trigger the connection using the startConn() function that we have written in conn.js by editing the onload function:

<body onload="var d = new Date(); startConn('/get?t='+d.getTime().toString(), loadMsgs);">

Improvements to make, places to go

Well, with everything done, save all the files, then run the server file. Add a message on localhost:8080/add and then watch the index page refreshes and updates all the messages automatically.

However, the limited webserver that web.py uses means that it is unlikely that it is capable of supporting multiple long poll request at a time. So next time, I will be teaching you how to setup web.py with lighttpd to handle these request smoothly.