Recently in a awesome thread in HN I've learnt about this awesome programs: file2ban, molly-guard and logwatch.
Well, I really knew that program before, but I thought that was kind of difficult to configure. It seems that the default options are really good. I make intensive use of the iptables firewall in my servers and it is nice that you can have an utility that adds/removes rules when something is trying to do bad things to you. Haven't really explored all the options of file2ban, but it look like the defaults are pretty awesome. After doing the vanilla install, the number of SSH attempts to connect to my server has decreased by an order of magnitude.
Is one of this programs that can save you from doing nasty things on your servers. Lets face it, some time you have executed the halt or reboot command.... well, I don't know you, but I have done it. Luky me it was a testing server, so only a few developers when to blame to the sysadmin why the hell that machine was off. This package patches the reboot/halt/etc... scripts and when you execute them you see something like that:
jan@lara> sudo reboot W: molly-guard: SSH session detected! Please type in hostname of the machine to reboot: ^C Good thing I asked; I won't reboot lara ...
Justs scans all the logs in /var/log/* and make a beautiful report with the relevant information. HTTP 404/500 error codes in Apache logs, kernel warning errors, sshd faliure login attemps, new users created and recently installed packages, commands executed by "sudo", report of the disk usage.... If you get use to execute it every time you log into a remote machine, you will get used to see what is the "normal" status of your machine and you will be able to detect the oddities when something bad has happended.