Compromised? Where are the logs?
Posted by Tate Hansen Fri, 10 Nov 2006 05:55:00 GMT
Full compromises are crazy – it happens, but nearly every one I touch still sends a little shiver down my spine mixed with a “holy shit” moment.
Case in point: we got a call last week to help out with an incident and to do an investigation of some Linux based security devices. Financial industry, lots of customers.
Their engineers were working feverishly to scope the incident and to learn all the details then slam – they hit a wall. Their firewall logs everything. Their only-way-out-is-via-a-proxy-server logs everything.
And their security point solution in the DMZ? Oops, it only logs locally (it also happens to be a party to the fully compromised club inside this trusted net). And what about their desktops which appear to have offered free remote admin VTYs? Well, this all has a kicker for an answer.
It happened a year ago. Ouch. Talk about sensing a cold sweat and suddenly not trusting anything. By the way, the desktops exhibiting the worst behavior traveled the evolutionary path and have been wiped clean and rebuilt, upgraded, or replaced since the break-in. No substantive logging or backups of mischievous desktops; no way to reconstruct the perpetrator’s methods. Security point solution in DMZ had standard local logging (i.e. full rotation in weeks), therefore it is unlikely things we're looking for are there (still investigation). No network device logging. There are lots of unanswered questions.
The details of this should strengthen your neuron paths connecting logging to your “holy shit” moments. That is to say…
- If you got important data to protect, log everything you reasonably can. All the “security” in this scenario failed and failed to help reconstruct events.
- Do you have one of those semi hands-off security appliances that you presume is working fine because you can connect to the web admin portal? Make it forward logs to somewhere.
- Do you have workstations which touch sensitive data anytime? Yes. Then boost the priority to configure central logging, stop procrastinating, then take comfort you’ll be in better shape than the poor souls at this company fighting to salvage their pride, and maybe their jobs.
