I've split this article into compromise hacks (i.e. rootkit attacks) which are relatively easy to detect and application compromises which can be more subtle.
UPDATE: After a discussion about the article on WebHostingTalk I've incorporated a few great suggestions.
This means someone has full access to the system, here are the tell tale signs in order of most likely to give you a quick feel for what's going on.
This is the first thing I would do.
find /etc /var -mtime -2
The "-2" means 2 days, i.e. show me all files modified in the last 2 days.
Now if you haven't installed any new software on your server for a while then this command will run and produce very little output. For a server I investigated there were references to postfix. clearly someone had installed a mail server probably for sending spam.
who user1 pts/2 2012-03-28 13:38 (188.8.131.52)
This should give you a list of users on the system, what you're looking for is users other than yourself especially root.
Login as root and run history, this will give you a list of recently run commands. If you're the only person who logs in as root you should have an idea if anything looks suspicious or not.
It's a good idea at this point to see what processes are running on which ports.
netstat --listen -A inet tcp 0 0 *:64010 *:* LISTEN tcp 0 0 *:http-alt *:* LISTEN tcp 0 0 *:ssh *:* LISTEN tcp 0 0 *:https *:* LISTEN
An example of an application breach might be a shopping cart that allows file uploads these uploads can be executed as say PHP scripts (This happened with oscommerce). The attacker doesn't have root access but they can certainly use up resources.
Sudden spikes in memory and load usage are an indication of possible root breaches but sometimes the only indication an application has been breached.
If you're a ServerMonitoringHQ.com user then you'll receive notifications of CPU and Memory spikes. (Along with bandwidth and more).
You can use the following command to track load (CPU usage). Generally values over 1 indicate the CPU is getting used quite hard.
cat /proc/loadavg 0.04 0.35 0.26 1/83 27412
The first column
is the number to look at
command is a quickly way to see what processes are consuming resources.
As with process and memory if you don't know what you normally consume it's difficult to tell if you get a spike or not.
netstat -i Kernel Interface table Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR eth0 1500 0 2937582 0 0 0 3127979 0 0 0 BMRU lo 16436 0 531312 0 0 0 531312 0 0 0 LRU
The larger numbers for TX and RX are bytes transferred (roughly), run this twice, a minute apart will give you an indication of bandwidth transfer per minute.
We've all been there, best of luck getting your server up and running again.
Spot server issues in lightning time.
"It's like PingDom for servers"