On Smalltalk

I thought this was too simple to bother posting about, but someone requested it, so here it is. Having made my journey to Linux about a year ago, I’ve found a new appreciation for the simplicity of plain old text files and the file system in general.

Some of the tools I’ve come to rely on like Daemontools from Daniel Bernstein also show this appreciation of the Unix way, small simple reliable tools that just work and are configured easily with plain text files. The most recent versions of Apache have also broken up the monolithic httpd.config into much smaller simpler individual files spread out over various directories that greatly ease automation by keeping individual site settings in separate files and using simple symbolic links to enable or disable individual sites.

Inspired by the simplicity and ease of managing these tools, I decided my Smalltalk programs, while ungodly easy to write, were much too hard to configure and deploy because configuration was based either on code in the image or in the case of Seaside, its custom configuration system, neither of which are simple or easy to move around between images and servers. I also like how Ruby on Rails configuration system allows multiple configurations that you can easily switch between to run your code in your various environments.

Rather than trying to design some ultimate configuration system, I just looked at how I write and deploy *my* applications, which is always one image per application in its own directory (possibly launched multiple times on various ports) and decided the simplest thing that could possibly work for me was just a directory named config, in my image directory, with three subdirectories named prod, dev, and test, and a single file named config which tells me which config subdirectory is active.

So I’ll declare a configuration class…

Object subclass: #SSConfig
    instanceVariableNames: ''
    classVariableNames: ''
    poolDictionaries: ''
    category: 'SentorsaSeaside-Configuration'

and a class side #create method to setup this structure on the filesystem…

create
    | configDir |
    configDir := (FileDirectory default directoryNamed: #config) assureExistence.
    FileStream
        forceNewFileNamed: (configDir fullNameFor: #config)
        do: [ : f | f nextPutAll: 'dev' ].
    (configDir directoryNamed: #prod) assureExistence.
    (configDir directoryNamed: #dev) assureExistence.
    (configDir directoryNamed: #test) assureExistence

Next, I want a simple dictionary style API, “SSConfig at: #someSetting”, so I’ll need the core class side method #at: which grabs the current configuration and uses the key you hand it to looks for the file name that matches, opening it in read only mode if it exists or simply creating it if it doesn’t.

at: aKey
	| currentConfig configDir value |
	currentConfig := FileStream
		readOnlyFileNamed: ((FileDirectory default directoryNamed: #config) fullNameFor: #config)
		do: [ : f | f contentsOfEntireFile ].
	configDir := (FileDirectory default directoryNamed: #config) directoryNamed: currentConfig.
	value := (configDir fileExists: aKey)
		ifTrue:
			[ FileStream
				readOnlyFileNamed: (configDir fullNameFor: aKey)
				do: [ : f | f contentsOfEntireFile ] ]
		ifFalse:
			[ FileStream
				forceNewFileNamed: (configDir fullNameFor: aKey)
				do: [ : f | f contentsOfEntireFile ] ].
	^ (value endsWith: Character lf asString)
		ifTrue: [ value allButLast ]
		ifFalse: [ value ]

Next I want to be able to ask for a value and provide a default in case there is no configured value, built simply upon the previous method…

at: aKey default: aValue
    ^(self at: aKey)
        ifEmpty: [ aValue ]
        ifNotEmptyDo: [ : it | it ]

I also want to be able to ask for a config value typed to a boolean (SSConfig can: #useBlaBla) for the common case of true/false settings, another simple method built upon #at:…

can: aKey
    ^(self at: aKey) = 'true'

And of course, having strings and booleans, I now want a couple of simple methods for asking for integers from config…

numAt: aKey
    ^(self at: aKey)
        ifEmpty: [ 0 ]
        ifNotEmptyDo: [ : it | it asInteger ]

numAt: aKey default: aValue
    ^(self at: aKey)
        ifEmpty: [ aValue ]
        ifNotEmptyDo: [ : it | it asInteger ]

These are all class side methods, and so far I’ve not found the need for anything more than strings, bools, and integers. If I do, I’ll extend it more, however, this seems to be all I need for all of my configuration needs thus far. It’s simple, every setting is its own file, and it allows me to easily switch between various configs and easily share configurations between various images with nothing more than a simple file copy. I can also easily edit the configuration from the shell, where I do most of my administration tasks.

NOTE: This method is not application aware, so it won’t work well if you run multiple applications within a single image, but I’m not writing code for use cases I don’t have and it works well for how I deploy my applications.

Tags: General, Linux, Programming, Ruby, Seaside, Smalltalk

One of the challenges you’ll face when deploying an application in Seaside, especially if you’re running headless on a Linux box somewhere, is how to upgrade your production application to the latest version of your code. For the longest time, I’ve done this the easy way, upgrade the production image off-line, upload it, and restart the services. It’s simple and straight forward, but it blows away all active sessions and kicks off any users currently online. There are other more interesting ways.

Some people use VNC to connect to their production image, fire up Monticello, and hot upgrade the code. Personally, I’ve never been able to get a stable version of VNC going, there are always issues, the latest being that the client is messed up by the nice Squeak UI Enhancements hiding the menus. Too much hassle, I gave up, I don’t trust VNC at all.

There’s a simple web based method within Seaside under /tools/versionuploader, you can use the web version of Monticello to load packages into the running image and then save the image from there. It’s simple and effective, if not a bit clunky, but it works great if you’re willing to expose that to the internet wrapped with just basic http authentication. I’ve been using this, but I thought I could have a little fun hacking up something a little more interesting.

So what are we talking about here, just loading a few Monticello packages and saving the image, how hard could that be to automate? Turns out, not so hard if I use the Installer that comes in Damien’s dev image (that my image is based upon). I decide to make my images upgrade themselves from a staging directory where I can just secure copy or ftp up the new versions plus an empty trigger file named “load” to kick the process off. I’ve been playing with it a few days now, seems to work pretty well. Here’s what it takes to make a self upgrading image (assuming you use Monticello to package all your code, as I do).

It’s just two methods really, one to load all the packages deleting them after each one is installed, assuming the load file exists…

loadChanges
    | dir |
    dir := (FileDirectory default directoryNamed: 'load-package') assureExistence.
    dir fileNames
        select: [ : ea | ea endsWith: '.mcz' ]
        thenDo:
            [ : fileName |
            ((dir fileExists: fileName) and: [ dir fileExists: #load ]) ifTrue:
                [Transcript show: (fileName , ': ').
                Installer installFile: (dir fullNameFor: fileName).
                Transcript cr.
                dir deleteFileNamed: fileName ] ].
    (dir fileExists: #load) ifTrue:
        [ dir deleteFileNamed: #load.
        SmalltalkImage current saveSession]

And a second to kick off a background process that runs continually, say every few seconds, that kicks off #loadChanges.

watchForChanges
    [[ (Delay forSeconds: 5) wait.
    [ self loadChanges ]
        on: Error
        do: [ : error | Transcript show: error ] ] repeat ]
        forkAt: Processor userBackgroundPriority + 1
        named: ‘code loader’

Then I call watchForChanges in a class side #initialize somewhere on a utility class so the background process is kicked off automatically in any image I load my code into. Seems pretty slick so far, just upload the new packages, upload an empty file named load, the image upgrades itself, deletes all the files in the staging directory, saves itself, and chugs along like nothing happened at all. No session are lost, no users booted, if anything they might notice a slight pause for a bigger package loading.

This all of course assumes your production image is basically read only, i.e. has no code changes in it, as mine are. Enjoy!

Tags: Linux, Seaside, Smalltalk

One of the nicest things about prototyping in Smalltalk is that you can delay the need to hook up a database during much of your development, and if you’re lucky, possibly even forever.

It’s a mistake to assume every application needs a relational database, or even a proper database at all. It’s all too common for developers to wield a relational database as a golden hammer that solves all problems, but for many applications they introduce a level of complexity that can making development feel like wading through a pond full of molasses where you spend much of your time trying to keep the database schema and the object schema in sync. It kills both productivity and fun, and god dammit, programming should be fun!

This is sometimes justified, but many times it’s not. Many business applications and prototypes are built to replace manual processes using Email, Word, and Excel. Word and Excel by the way, aren’t ACID compliant, don’t support transactions, and manage to successfully run most small businesses. MySql became wildly popular long before it supported transactions, so it’s pretty clear a wide range of apps just don’t need that, no matter how much relational weenies say it’s required.

It shouldn’t come as a surprise that one can take a single step up the complexity ladder, and build simple applications that aren’t ACID compliant, don’t support transactions, and manage to successfully run most small businesses BETTER than Word and Excel while purposely not taking a further step and moving up to a real database which would introduce a level of complexity that might blow the budget and make the app infeasible.

No object relational mapping layer (not even Rails and ActiveRecord) can match the simplicity, performance, and speed of development one can get just using plain old objects that are kept in memory all the time. Most small office apps with no more than a handful of users can easily fit everything into memory, this is the idea behind Prevayler.

The basic idea is to use a command pattern to apply changes to your model, you can then log the commands, snapshot the model, and replay the log in case of a crash to bring the last snapshot up to date. Nice idea, if you’re OK creating commands for every state changing action in you’re application and being careful with how you use timestamps so replaying the logs works properly. I’m not OK with that, it introduces a level of complexity that is overkill for many apps and is likely the reason more people don’t use a Prevayler like approach.

One might attempt to use the Smalltalk image itself as a database (and many try), but this is ripe with problems. My average image is well over 30 megs, saving it takes a bit of time, and saving it while processing http requests risks all kinds of things going wrong as the image prepares for what is essentially a shutdown/restart cycle.

Using a ReferenceStream to serialize objects to disk Prevayler style, but ignoring the command pattern part and just treating it more like crash proof image persistence is a viable option if your app won’t ever have that much data. Rather than trying to minimize writes with commands, you just snapshot the entire model on every change. This isn’t as crazy as it might sound, most apps just don’t have that much data. This blog for example, a year and a half old, around 100 posts, 1500 comments, has a 2.1 megabyte MySql database, which would be much smaller as serialized objects.

If you’re going to have a lot of data, clearly this is a bad approach, but if you’re already thinking about how to use the image for simple persistence because you know your data will fit in ram, here’s how I do it.

It only takes a few lines of code in a single abstract class that you can subclass for each project to make a Squeak image fairly robust and crash proof and more than capable enough to allow you just use the image, no database necessary. We’ll start with a class…

Object subclass: #SMFileDatabase
	instanceVariableNames: ''
	classVariableNames: ''
	poolDictionaries: ''
	category: 'SimpleFileDb'

SMFileDatabase class
	instanceVariableNames: 'lock'

All the methods that follow are class side methods. First, we’ll need a method to fetch the directory where rolling snapshots are kept.

backupDirectory
	^ (FileDirectory default directoryNamed: self name) assureExistence.

The approach I’m going to take is simple, a subclass will implement #repositories to return the root object that needs serialized, I just return an array containing the root collection of each domain class.

repositories
	self subclassResponsibility

The subclass will also implement #restoreRepositories: which will restore those repositories back to wherever they belong in the image for the application to use them.

restoreRepositories: someRepositories
	self subclassResponsibility

Should the image crash for any reason, I want the last backup will be fetched from disk and restored. So I need a method to detect the latest version of the backup file, which I’ll stick a version number in when saving.

lastBackupFile
	^ self backupDirectory fileNames
		detectMax: [:each | each name asInteger]

Once I have the file name, I’ll deserialize it with a read only reference stream (don’t want to lock the file if I don’t plan on editing it)

lastBackup
	| lastBackup |
	lastBackup := self lastBackupFile.
	lastBackup ifNil: [ ^ nil ].
	^ ReferenceStream
		readOnlyFileNamed: (self backupDirectory fullNameFor: lastBackup)
		do: [ : f | f next ]

This requires you extend ReferenceStream with #readOnlyFileNamed:do:, just steal the code from FileStream so nicely provided by Avi Bryant that encapsulates the #close of the streams behind #do:. Much nicer than having to remember to close your streams.

Now I can provide a method to actually restore the latest backup. Later, I’ll make sure this happens automatically.

restoreLastBackup
	self lastBackup ifNotNilDo: [ : backup | self restoreRepositories: backup ]

I like to keep around the last x number of snapshots to give me a warm fuzzy feeling that I can get old versions should something crazy happen. I’ll provide a hook for an overridable default value in case I want to adjust this for different projects.

defaultHistoryCount
	^ 15

Now, a quick method to trim the older versions so I’m not filling up the disk with data I don’t need.

trimBackups
	| entries versionsToKeep |
	versionsToKeep := self defaultHistoryCount.
	entries := self backupDirectory entries.
	entries size < versionsToKeep ifTrue: [ ^ self ].
	((entries sortBy: [ : a : b | a first asInteger < b first asInteger ])
		allButLast: versionsToKeep)
			do: [ : entry | self backupDirectory deleteFileNamed: entry first ]

OK, I’m ready to actually serialize the data. I don’t want multiple processes all trying to do this at the same time, so I’ll wrap the save in a critical section, #trimBackups, figure out the next version number, and serialize the data (#newFileNamed:do: another stolen FileStream method), ensuring to #flush it to disk before continuing (don’t want the OS doing any write caching).

saveRepository
	| version |
	lock critical:
		[ self trimBackups.
		version := self lastBackupFile
			ifNil: [ 1 ]
			ifNotNil: [ self lastBackupFile name asInteger + 1 ].
		ReferenceStream
			newFileNamed: (self backupDirectory fullPathFor: self name) , ‘.’ , version asString
			do: [ : f | f nextPut: self repositories ; flush ] ]

So far so good, let’s automate it. I’ll add a method to schedule the subclass to be added to the start up and shutdown sequence. You must call this for each subclass, not for this class itself.

UPDATE: This method also initializes the lock and must be called prior to using #saveRepository, this seems cleaner.

enablePersistence
	lock := Semaphore forMutualExclusion.
	Smalltalk addToStartUpList: self.
	Smalltalk addToShutDownList: self

So on shutdown, if the image is actually going down, just save the current data to disk.

shutDown: isGoingDown
	isGoingDown ifTrue: [ self saveRepository ]

And on startup we can #restoreLastBackup.

startUp: isComingUp
	isComingUp ifTrue: [ self restoreLastBackup ]

Now, if you want a little extra snappiness and you’re not worried about making the user wait for the flush to disk, I’ll add little convience method for saving the repository on a background thread.

takeSnapshot
	[self saveRepository] forkAt: Processor systemBackgroundPriority
		named: ’snapshot: ‘ , self class name

And that’s it, half a Prevayler and a more robust easy to use method that’s a bit better than trying to shoehorn the image into being your database for those small projects where you really really don’t want to bother with a real database (blogs, wikis, small apps, etc). Just sprinkle a few MyFileDbSubclass saveRepository or MyFileDbSubclass takeSnapshot’s around your application whenever you feel it important, and you’re done.

Here’s a file out if you just want the code fast, SMFileDatabase.st

Tags: Databases, Programming, Smalltalk, Sql

Just a quick notification that I updated my squeak image (several of you have asked).

It’s based on Damien Cassou’s latest Squeak Dev Image (Squeak 3.9), an awesome base image with all the necessary goodies a developer needs. Of course I’ve loaded up my window customizations and preferences, nicer looking fonts, and have all the preferences set the way I like.

Nothing major in this update other than keeping up with the latest versions of everything I use. I have removed the SentorsaBrowser and TrickRefractoringBrowser because the new OmniBrowser is just getting too good to use anything else. Enjoy.

Tags: Smalltalk, Updates