Scaling Seaside: More Advanced Load Balancing And Publishing
By Ramon Leon - 29 July 2008 under Seaside, Smalltalk, Squeak, Apache
Seaside is a stateful application server and a Squeak VM is only capable of taking advantage of a single processor. When hosting a website on a multi processor server, or several servers, you need to load balance your requests across several instances of Squeak to fully utilize the hardware and handle more load than a single Squeak VM is capable of dealing with.
All serious Seaside applications that I'm aware of are front ended by Apache to offload the serving of static content like images, CSS files, JavaScript files, and static HTML while proxying only dynamic content to Seaside. Apache is an awesome platform and absolutely should be one of the tools is your toolbox; however, it can be quite challenging to setup in such a way that it makes load balancing and deploying new code seamless to your users.
I want to thank Avi Bryant for some tips about how DabbleDB handles this issue last year which gave me the basic approach and Lukas Renggli for some recent discussions that spurred me to finally spend some time on this issue and settle it for myself.
Previously I've used HAProxy, and when it became available moved to Apache's modproxybalancer module to load balance multiple Squeak VM's. I found both approaches lacking when it came to rolling out new code without being forced to blow away all existing user sessions when restarting Squeak with newly updated application code. Because of this I'd been rolling out code only during non peak hours when few users would be affected; however, I need to be able to roll out new code anytime and dynamically shift new sessions to the new servers while allowing the old VM's to remain up and running until their sessions expire and are no longer needed.
I've now abandoned modproxybalancer in favor of mod_rewrite and a few scripts which allowed me to tailor a custom solution that gives me much more control allowing seamless deployment of new code while allowing Apache to launch Squeak VM's as necessary and on the fly to handle load. The basic approach is as follows...
When a request comes in I use a rewrite condition to check for a cookie set by the Seaside telling me what server this request should be handled by.
If the cookie exists, I look up the server name in a rewrite map to find out what port that server is running on. That port feeds directly into another rewrite map that runs a bash script which uses netcat to see if the port is open or closed and returns the result.
verifyPort
#!/bin/bash while read ARG; do nc -z localhost $ARG if [ $? = 0 ]; then echo open$ARG else echo closed fi done
If the port is open, I proxy the request directly to the server with a rewrite rule when it sees the word "open" in the result.
If the port is closed or if no cookie is found, I proxy the request on a random port chosen from the rewrite map that contains the server mappings. This script also checks to see if the port is open and return the port number immediately if it is; however, if the port isn't open, possibly because an image died, it kills any old process on that port and launches a new Squeak VM on the specified port and waits until it finds the new port open before returning the port number and allowing Apache to proxy the request.
imageLauncher
#!/bin/bash while read ARG; do nc -z localhost $ARG if (( $? != 0 )); then #kill any hung images if [ -f workers/squeak.$ARG ]; then cat workers/squeak.$ARG | xargs -r kill fi #launch squeak dropping permissions to a non root user via sudo sudo -u www-data squeak -mmap 150m -headless \ -vm-sound-null -vm-display-null \ /var/squeak/app.image /var/squeak/startScript port $ARG & #log worker process id echo $! > workers/squeak.$ARG sleep 1 nc -z localhost $ARG while (( $? != 0 )); do sleep 1 nc -z localhost $ARG done fi echo $ARG done
When a VM starts up, it's fed a script...
startScript
[[[ 60 seconds asDelay wait. WARegistry allSubInstances do: [ :e | e unregisterExpiredHandlers ]. (WASession allSubInstances allSatisfy: [ :e | e expired ]) ifTrue: [ SmalltalkImage current snapshot: false andQuit: true ] ] on: Error do: [ :error | error asDebugEmail ] ] repeat ] forkAt: Processor systemBackgroundPriority. Project uiProcess suspend.
The script kicks off a background process that runs every 60 seconds and expires any sessions that are ready for expiration and shuts the image down when it finds all sessions have expired. It also kills the UI process which isn't needed on a headless server and just wastes CPU cycles if left running.
Not that this process starts off with an immediate 60 second delay, this allows ample time for the image to startup and start receiving its first request and establish a session. If you don't start with the delay, you'd see the image startup and immediately shut back down when it found it contained no active sessions (yes, I did this a few times before figuring out what the hell was happening).
Also, this being a low priority background process, I'm not concerned with using #allSubInstances as speed isn't relevant.
When a request hits Seaside for the first time, the root component of the Seaside app checks to see if the cookie is set and has the correct value. If the cookie is missing (new request) or the cookie has the wrong value (expired request for a VM that's no longer running and timed itself out) Seaside resets the cookie to the correct value so all future requests are served by that VM.
initialRequest: aRequest self setServerCookie setServerCookie | newVal cookieVal | cookieVal := self session currentRequest cookies at: #server ifAbsent: [ nil ]. newVal := 'app' , (HttpService allInstances detect: [ :each | each isRunning ]) portNumber asString. cookieVal ~= newVal ifTrue: [ self session redirectWithCookie: (WACookie key: #server value: newVal) ]
I keep three versions of the server mappings, the active version and an A version and a B version which I can simply copy over the active version to shift traffic to a new set of VM's which Apache will launch dynamically. A simple "cp balance.confA balance.conf" instantly shifts new traffic to the ports specified in the "ALL" mapping.
balance.conf
app3001 3001 app3002 3002 app3003 3003 app3004 3004 app3005 3005 app3006 3006 app3007 3007 app3008 3008 app3009 3009 app3010 3010 ALL 3001|3002|3003|3004|3005
balance.confA
app3001 3001 app3002 3002 app3003 3003 app3004 3004 app3005 3005 app3006 3006 app3007 3007 app3008 3008 app3009 3009 app3010 3010 ALL 3001|3002|3003|3004|3005
balance.confB
app3001 3001 app3002 3002 app3003 3003 app3004 3004 app3005 3005 app3006 3006 app3007 3007 app3008 3008 app3009 3009 app3010 3010 ALL 3006|3007|3008|3009|3010
Though the port number is part of the cookie value, I don't use that directly, it's simply a convenient way to name the app images dynamically. A user could fake a cookie value and looking it up in the mapping to find the port ensures I maintain full control over what ports images are launched on.
And finally here's a full production Apache configuration I use to invoke this setup with all the bells and whistles I currently use in my production sites...
Apache Config
<VirtualHost *:80> ServerName www.yoursite.com DocumentRoot /var/www/www.yoursite.com RewriteEngine On ProxyRequests Off ProxyPreserveHost On UseCanonicalName Off # tag cachable items with an expiry date for browsers ExpiresActive on ExpiresByType text/css A864000 ExpiresByType text/javascript A864000 ExpiresByType application/x-javascript A864000 ExpiresByType image/gif A864000 FileETag none # rewrite maps for managing Seaside application pool RewriteMap SQUEAK prg:/var/squeak/imageLauncher RewriteMap PORTS prg:/var/squeak/verifyPort RewriteMap SERVERS rnd:/var/squeak/balance.conf # http compression DeflateCompressionLevel 9 SetOutputFilter DEFLATE AddOutputFilterByType DEFLATE text/html text/plain text/xml application/xml application/xhtml+xml text/javascript text/css BrowserMatch ^Mozilla/4 gzip-only-text/html BrowserMatch ^Mozilla/4.0[678] no-gzip BrowserMatch \bMSIE !no-gzip !gzip-only-text/html # Logfiles CustomLog /var/log/apache2/www.yoursite.com.access.log combined ErrorLog /var/log/apache2/www.yoursite.com.error.log # Let apache serve any static files NOW RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} -f RewriteRule (.*) $1 [L] #if the cookie has the server in it and the port is open, proxy to it RewriteCond %{HTTP_COOKIE} "server=(app[0-9]*)" RewriteCond ${PORTS:${SERVERS:%1}} open(.*) RewriteRule ^/appPath(.*)$ http://localhost:%1/seaside/appPath$1 [P,L] #otherwise proxy to a random Seaside server launching one if necessary RewriteRule ^/appPath(.*)$ http://localhost:${SQUEAK:${SERVERS:ALL}}/seaside/appPath$1 [P,L] </VirtualHost>
And in my httpd.conf I specify a rewrite lock to ensure Apache doesn't have concurrency issues with any of the rewrite map scripts.
RewriteLock /var/squeak/squeak.lock
And that's it. If anyone sees any problems with this approach (Avi, Lukas, or random Apache guru) I'd appreciate a heads up. I ran it by some folks in the #apache channel in IRC and they seemed to think it was fine. I've had this in production for a little bit now and so far it's kicking ass and I'm now free to increase or decrease my pool size dynamically or shift to a whole new set of images when publishing code.
As always, something cool to learn from you. I will try this setup next weekend, but as I see it, it is very well done.
Thank you very much.