Came across an interesting blog entry by Raffy at Splunk. As a marketing guy I am jealous as they are generating a lot of buzz about “IT Search”. Splunk has led a lot of people that are knowledgeable to wonder how this is something different than what all the log management vendors have been providing.
Still, while Raffy touched on what is one of the real differences between IT Search and Log Management, he left a few of the salient points out in the discussion of a “connector” and how a connector puts you at the mercy of the vendor to produce the connector, and what happens when the log data format changes?
Let’s step back — at the most basic level in log management (or IT Search for that matter) you have to do 2 fundamental things, you have to help people 1) collect logs from a mess of different sources, and 2) help them do interesting things with them. The “do interesting things” includes the usual stuff like correlation, reporting, analytics, secure storage etc.
You can debate fiercely the relative robustness of collection architectures – and there are a number of differences if you are evaluating vendors you should look at. For the sake of this discussion however most any log management system worthy of its salt will have a collection mechanism for all the basic methods – if you handle (in no particular order) ODBC, Syslog, read the Windows event format, maybe SNMP, throw in a file reader for custom applications, well you have the collection pretty much covered..
The reality is, as Raffy points out, there are a few totally proprietary access methods to get logs like Checkpoint. It is far easier for a system or application vendor to write one of the standard methods. So getting access to the raw logs in some way, shape or form is straightforward.
So here is where the real difference between IT search and Log Management begins.
Raffy mentions a small change in the syslog format causing the connector to break. Well syslog is a standard so if it would not break any standard syslog receiver, what it actually meant is that the syslog message has not changed but the content had.
Log Management vendors provide “knowledge” about the logs beyond simple collection.
Let’s make an analogy – IT Search is like the NSA collecting all of the radio transmissions in all of the languages in the entire world. Pretty useful. However, if you want to make sense of the Russian ones you hire your Russian expert, Swahili, your Swahili expert and so on. You get the picture.
Logs are like languages — the fact of the matter is the only thing that is the same about logs is that the content is all different. If you happen to be an uber-log weenie and you understand the format of 20 different logs, simple IT Search is really powerful. If you are only concerned about a single log format like Windows (although Windows by itself is pretty darn arcane), IT Search can be a powerful tool. If you are like the rest of us whose entire lives are not spent understanding multiple log formats, or get really rusty because many of us often don’t get exposed to certain formats all the time, well, it gets a little harder. What Log Management vendors do is to help you ( as the user) out with the knowledge – rules that categorize important event logs from unimportant ones, alerts, reports that are configured to look for key words in the different log streams. How this is done is different from vendor to vendor – some normalize, i.e. translate logs into a standard canonical format, others don’t. And this knowledge is what can conceivably get out of date.
In IT Search, there is no possibility for anything to get out of date mainly because there is no knowledge, only the ability to search the log in its native format. Finally, if a Log Management vendor is storing the original log and you can search on it, your Log Management application gives you all the capability of IT Search.
Seems to me IT Search is much ado about nothing…
-Posted by Steve Lafferty
Tags: IT search, Log Management
[...] posted a commentary to my blog post about IT Search vs. SIEM – Data Collection. I want to address some of his comments [...]
I posted a follow-up entry on my blog about your comments here. You can find the full text on my blog. In short, IT search is more than just searching your logs. It has to do with imposing (dynamic) schemas at search time. There are significant limitations in requiring data parsing at collection time. But I already talked about them in my original blog entry.
To Raffy’s ppoint about Syslog formats changing, I believe he may have meant that if information is being written to a syslog, a connector may be expecting a certain number of fields to parse out so the user can do remedial searching or reporting. If that field format changes by the application writing to syslog, the connector may have to be reconfigured or modified in some way. Tools like Splunk (wait, is there anything out there like Splunk?), just index everything and provide true search, and some cool technology that is way more malleable than brittle connector and fixed format based approaches.
I think the most important point that you are missing is most IT people supporting applications DON”T look in syslog for the real meat of whats going on. Standard places to look, such as Event Logs and a Syslog only contain a small amount of the data to help resolve application outages and other issues The golden nuggets are in the complex multi-line formats that .NET, J2EE, Weblogic, Websphere, Apache, Oracle and other application developers write out to the file system as a record of everything that happens in an application. The poor first and second level support guys in a company that has a decent size infrastructure are left with a lot of log data, in wide and varying formats all over their production systems.
Finally, the incredible value of IT search in general is not only in its ability to index, but search. Search allows us to link events together because not only has it already indexed everything regardless of format, but search languages allow the expression of the search query in a more flexible manner to the user.
In my experience as a sysadmin, log management is more about lassoing and storage (hence the word “management”). IT search is about letting IT people get away from being “tool operators”, giving them technology to get work done, and show their peers and organizations just how damned good they are.
Thanks for your comments, Lachlan. A few thoughts:
I have said repeatedly that one of the core value positions that SIEM solutions offer is to provide the user with knowledge. And I commended Raffy on Splunk’s drive to add knowledge.
The reality is that not many people understand the intricacies of dozens of different log formats. IT infrastructures are a bit of a grab-bag. The issue with simply indexing, or even in presenting the most flexible language in the world is that the person searching is going to have to know the difference languages (log formats). Big problem.
A simple example. 2 systems. You have to know that on ‘Very Important System A’ the event “I am giving up the ghost and crashing now” is “System Abend XYZ” and on ‘Very Important System B’ it is “Throat Wharbler Mangrove” (the developer was having an off day on error messages when he wrote that). Ok, that is manageable, but darn, it is never 2 systems, as you observed. It is many, many more and they are all different.
I like your statement “The golden nuggets are in the complex multi-line formats that .NET, J2EE, Weblogic, Websphere, Apache, Oracle and other application developers write out to the file system as a record of everything that happens in an application.” I would argue that none of your first level guys and most of your second level guys will able to find those nuggets that you write about, without help, regardless of the robustness of their ability to sift thought the haystack.
You summed it up well. Splunk allows you to show off how damned good you are – in other words you supply the knowledge, and it allows you to show off how smart you are, faster. But it does little to make you smarter.