On Smalltalk

Seaside is a stateful application server and a Squeak VM is only capable of taking advantage of a single processor. When hosting a website on a multi processor server, or several servers, you need to load balance your requests across several instances of Squeak to fully utilize the hardware and handle more load than a single Squeak VM is capable of dealing with.

All serious Seaside applications that I'm aware of are front ended by Apache to offload the serving of static content like images, CSS files, JavaScript files, and static HTML while proxying only dynamic content to Seaside. Apache is an awesome platform and absolutely should be one of the tools is your toolbox; however, it can be quite challenging to setup in such a way that it makes load balancing and deploying new code seamless to your users.

I want to thank Avi Bryant for some tips about how DabbleDB handles this issue last year which gave me the basic approach and Lukas Renggli for some recent discussions that spurred me to finally spend some time on this issue and settle it for myself.

Previously I've used HAProxy, and when it became available moved to Apache's modproxybalancer module to load balance multiple Squeak VM's. I found both approaches lacking when it came to rolling out new code without being forced to blow away all existing user sessions when restarting Squeak with newly updated application code. Because of this I'd been rolling out code only during non peak hours when few users would be affected; however, I need to be able to roll out new code anytime and dynamically shift new sessions to the new servers while allowing the old VM's to remain up and running until their sessions expire and are no longer needed.

I've now abandoned modproxybalancer in favor of mod_rewrite and a few scripts which allowed me to tailor a custom solution that gives me much more control allowing seamless deployment of new code while allowing Apache to launch Squeak VM's as necessary and on the fly to handle load. The basic approach is as follows...

When a request comes in I use a rewrite condition to check for a cookie set by the Seaside telling me what server this request should be handled by.

If the cookie exists, I look up the server name in a rewrite map to find out what port that server is running on. That port feeds directly into another rewrite map that runs a bash script which uses netcat to see if the port is open or closed and returns the result.

verifyPort

#!/bin/bash
while read ARG; do
  nc -z localhost $ARG
  if [ $? = 0 ]; then
    echo open$ARG
  else
    echo closed
  fi
done

If the port is open, I proxy the request directly to the server with a rewrite rule when it sees the word "open" in the result.

If the port is closed or if no cookie is found, I proxy the request on a random port chosen from the rewrite map that contains the server mappings. This script also checks to see if the port is open and return the port number immediately if it is; however, if the port isn't open, possibly because an image died, it kills any old process on that port and launches a new Squeak VM on the specified port and waits until it finds the new port open before returning the port number and allowing Apache to proxy the request.

imageLauncher

#!/bin/bash
while read ARG; do
  nc -z localhost $ARG
  if (( $? != 0 )); then
    #kill any hung images
    if [ -f workers/squeak.$ARG ]; then
      cat workers/squeak.$ARG | xargs -r kill
    fi
    #launch squeak dropping permissions to a non root user via sudo
    sudo -u www-data squeak -mmap 150m -headless \
        -vm-sound-null -vm-display-null \
        /var/squeak/app.image /var/squeak/startScript port $ARG &
    #log worker process id
    echo $! > workers/squeak.$ARG
    sleep 1
    nc -z localhost $ARG
    while (( $? != 0 )); do
      sleep 1
      nc -z localhost $ARG
    done
  fi
  echo $ARG
done

When a VM starts up, it's fed a script...

startScript

[[[ 60 seconds asDelay wait.
    WARegistry allSubInstances do: [ :e | e unregisterExpiredHandlers ].
    (WASession allSubInstances allSatisfy: [ :e | e expired ])
        ifTrue:
          [ SmalltalkImage current snapshot: false andQuit: true ] ]
        on: Error
        do: [ :error | error asDebugEmail ] ] repeat ]
        forkAt: Processor systemBackgroundPriority.

Project uiProcess suspend.

The script kicks off a background process that runs every 60 seconds and expires any sessions that are ready for expiration and shuts the image down when it finds all sessions have expired. It also kills the UI process which isn't needed on a headless server and just wastes CPU cycles if left running.

Not that this process starts off with an immediate 60 second delay, this allows ample time for the image to startup and start receiving its first request and establish a session. If you don't start with the delay, you'd see the image startup and immediately shut back down when it found it contained no active sessions (yes, I did this a few times before figuring out what the hell was happening).

Also, this being a low priority background process, I'm not concerned with using #allSubInstances as speed isn't relevant.

When a request hits Seaside for the first time, the root component of the Seaside app checks to see if the cookie is set and has the correct value. If the cookie is missing (new request) or the cookie has the wrong value (expired request for a VM that's no longer running and timed itself out) Seaside resets the cookie to the correct value so all future requests are served by that VM.

initialRequest: aRequest 
    self setServerCookie

setServerCookie
    | newVal cookieVal |
    cookieVal := self session currentRequest cookies at: #server ifAbsent: [ nil ].
    newVal := 'app' , (HttpService allInstances detect: [ :each | each isRunning ]) portNumber asString.
    cookieVal ~= newVal ifTrue: 
        [ self session redirectWithCookie: (WACookie key: #server value: newVal) ]

I keep three versions of the server mappings, the active version and an A version and a B version which I can simply copy over the active version to shift traffic to a new set of VM's which Apache will launch dynamically. A simple "cp balance.confA balance.conf" instantly shifts new traffic to the ports specified in the "ALL" mapping.

balance.conf

app3001 3001
app3002 3002
app3003 3003
app3004 3004
app3005 3005
app3006 3006
app3007 3007
app3008 3008
app3009 3009
app3010 3010
ALL 3001|3002|3003|3004|3005

balance.confA

app3001 3001
app3002 3002
app3003 3003
app3004 3004
app3005 3005
app3006 3006
app3007 3007
app3008 3008
app3009 3009
app3010 3010
ALL 3001|3002|3003|3004|3005

balance.confB

app3001 3001
app3002 3002
app3003 3003
app3004 3004
app3005 3005
app3006 3006
app3007 3007
app3008 3008
app3009 3009
app3010 3010
ALL 3006|3007|3008|3009|3010

Though the port number is part of the cookie value, I don't use that directly, it's simply a convenient way to name the app images dynamically. A user could fake a cookie value and looking it up in the mapping to find the port ensures I maintain full control over what ports images are launched on.

And finally here's a full production Apache configuration I use to invoke this setup with all the bells and whistles I currently use in my production sites...

Apache Config

<VirtualHost *:80>
    ServerName www.yoursite.com
    DocumentRoot /var/www/www.yoursite.com
    RewriteEngine On
    ProxyRequests Off
    ProxyPreserveHost On
    UseCanonicalName Off

    # tag cachable items with an expiry date for browsers
    ExpiresActive on
    ExpiresByType text/css A864000
    ExpiresByType text/javascript A864000
    ExpiresByType application/x-javascript A864000
    ExpiresByType image/gif A864000
    FileETag none

    # rewrite maps for managing Seaside application pool
    RewriteMap SQUEAK prg:/var/squeak/imageLauncher
    RewriteMap PORTS prg:/var/squeak/verifyPort
    RewriteMap SERVERS rnd:/var/squeak/balance.conf

    # http compression
    DeflateCompressionLevel 9
    SetOutputFilter DEFLATE
    AddOutputFilterByType DEFLATE text/html text/plain text/xml application/xml 
application/xhtml+xml text/javascript text/css
    BrowserMatch ^Mozilla/4 gzip-only-text/html
    BrowserMatch ^Mozilla/4.0[678] no-gzip
    BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

    # Logfiles
    CustomLog /var/log/apache2/www.yoursite.com.access.log combined
    ErrorLog  /var/log/apache2/www.yoursite.com.error.log

    # Let apache serve any static files NOW
    RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} -f
    RewriteRule (.*) $1 [L]

    #if the cookie has the server in it and the port is open, proxy to it
    RewriteCond %{HTTP_COOKIE} "server=(app[0-9]*)"
    RewriteCond ${PORTS:${SERVERS:%1}} open(.*)
    RewriteRule ^/appPath(.*)$ http://localhost:%1/seaside/appPath$1 [P,L]

    #otherwise proxy to a random Seaside server launching one if necessary
    RewriteRule ^/appPath(.*)$ http://localhost:${SQUEAK:${SERVERS:ALL}}/seaside/appPath$1 [P,L]
</VirtualHost>

And in my httpd.conf I specify a rewrite lock to ensure Apache doesn't have concurrency issues with any of the rewrite map scripts.

RewriteLock /var/squeak/squeak.lock

And that's it. If anyone sees any problems with this approach (Avi, Lukas, or random Apache guru) I'd appreciate a heads up. I ran it by some folks in the #apache channel in IRC and they seemed to think it was fine. I've had this in production for a little bit now and so far it's kicking ass and I'm now free to increase or decrease my pool size dynamically or shift to a whole new set of images when publishing code.