We just finished a beta cycle on a new application that provides the ability to search for any logs collected by AppFirst, aka Log Search. We’re pretty excited about this capability. We know there are a lot of log search tools available, but we think what we are able to offer is pretty cool. The ability to correlate and now search log data along with all other data collected and streamed by AppFirst represents a unique solution. (Yes, that means our patent-pending Miss Nothing Data, Nagios plugins, Windows Performance Counters, and StatsD.)
We’ve been collecting log data for a while now – available for alerting and correlation with the rest of our data. With our new Log Search application, you can search any and all logs for keywords and severity level in any time range you want. You can even throw two keywords in there if you’d like (let’s get crazy!).
We spend a lot of time looking at logs when we are working with our production systems. We’ve found that having all logs from all sources consolidated in one place has been really useful for us. No more having to go into each box and search for the log sources individually. They’re all now in one place, making search a hell of a lot easier.
There are a few scenarios where we’ve found that we want to watch a particular log file. We want to see what’s happening right now. In this scenario, it’s less about looking for specific entries in log data and more about what’s transpiring. Since we use this capability a lot, we added support for it in our new application. We call it Log Watch. It’s like using the command “tail -f /var/log/messages” in Linux/Unix.
In a lot of scenarios there is nothing that replaces the ability to search logs for specific information. We will expand our keyword search capability to include support for a search language as we proceed (this would include Boolean expressions). We are also looking at supporting SQL statements at some level. We’re not sure if this is needed or if it’s as useful. Something we’re looking into.
Search results are pretty fast. This is something we were worried about. We took a very different approach to search technology and it wasn’t obvious how performance would work out when we started. We use HBase for persistent storage. We really wanted to avoid the need for any sort of ETL – Extract, Translate, Load – into a separate search technology. An ETL would require additional storage and increased complexity. We wanted to find out if we could perform searches directly on information resident in HBase.
Turns out, you really can perform searches directly from HBase. In our design, it greatly simplifies search capabilities. Now that we are able to search log data, we can extend the service to search any of the data that we aggregate. We hope to provide this capability in the near future.
We made use of an HBase feature to perform column searches as the basis for locating specific log entries. We had to modify the way we store log data in HBase in order to make this effective. It did give us the performance and accuracy we were looking for. We were pretty excited when we figured out how to make this work. The details of how HBase works and the specifics of column organization is beyond the scope of this blog. Maybe we should do a blog on these details at some point. Let us know if you’re interested in the comments.
Of course, making something like this work and making it work at scale are very different tasks. We provide REST APIs to perform search on log data and our Log Search application uses these public APIs. You can extend the log application or create your own using the same APIs.
We found that we needed to pay a lot of attention to the size of data being returned. There is a practical limit to the size of a JSON object that you want to return to the browser. Part of the solution was to create paging with the APIs. One of the biggest issues to deal with is the fact that it’s difficult to determine the size of the data resulting from a given search. Indexing of the data helps with this. Of course, indexing is fundamental to search, but we are looking at ways to avoid the need to index massive amounts of data. We aren’t sure where the indexing question is going to end up.
There is a lot of detail about how search can be performed using HBase. We didn’t intend to get into that detail just yet. For this post, we just wanted to describe an overview of the approach as we release our Log Search application. If there’s interest we can describe details. Let us know. Would be great to hear from people who are interested. We’d like to see if other people are looking at similar approaches. We’d like to cooperate with, maybe even commiserate with, people working on similar approaches. Thanks.
Donn Rochette, CTO
P.S. If you’d like to try our beta version of Log Search and Log Watch, you can sign up for our Beta Program.