It’s been a while since IPS ceased to offer free unlimited trials of IPB (Invision Power Board) and I’ve been trying out various forum software since, but most don’t seem to have the same level of usability as well as featureset as compared to IPB.
The venerable phpBB has been around for years and wasn’t updated much till only recently when they released v3.0 beta1. I have not tested it out much, but v2 had a very weak admin control panel which I found annoying to use. It being a target of heaps of exploits don’t help much either. See v2 here and v3 here (via KhimHoe.Net).
SMF (Simple Machines Forum) v1.1 RC looked promising and decent aesthetically, but the admin CP was poor as well. I didn’t quite like how the code was modularised when I tried to modify the underlying code. See it here.
One forum software which I came across recently and sort of liked is PunBB. It’s very lightweight and quick and only contains the minimum functionality (even the PM function is left out! – a little absurd IMHO). Mods to add functionality to it are quite plentiful, so you should be able to search for what you want and not code it up yourself. It’s probably a hassle for the initial set up, but this would definitely keep bloat to a bare minimum and allow the user to choose what he/she requires (similar to Firefox’s extension based system). See it running here and here.
I’ll probably stick to PunBB for now. Wikipedia has a long list of forum software here in a comparison table.