Why we like StatsD
- StatsD is simple. Detailed metrics from your code? Include a class and call a method – that’s it. That’s all you do. It doesn’t get any easier than that.
- Statistic counters reveal important things about the internal workings of your code. The only limitation is your imagination. Whatever you need to see you can expose with a simple method call.
- StatsD is unlimited. You can call it from any place in your code and create whatever, and as many, metrics as you want.
AppFirst’s spin on StatsD: Extensions
At AppFirst we created extensions to StatsD that allow you to “annotate” metrics on your AppFirst Dashboard with the information that you would normally search for in log files. Our version preserves your existing investments in StatsD, so you have make the most of your hard work.
1. Status messages: This gives you the ability to add an optional information string to each StatsD msg, which is really helpful for adding context to your statistics counter. For example, if you have a bucket that describes failed logins, with the existing StatsD capability you would be able to increment the count of failed logins, with AppFirst StatsD you can do this and also describe the user names that were unable to login using status strings.
2. Hierarchical namespace: This give you the ability to manage how buckets are organized. For example, using the table below, if you were using the application component Redis and you wanted to see how much network traffic Redis was using, you would use the Namespace pattern: sys.app.redis.byte_sent. This can also be used to incorporate external sources of statistics metrics, which we’ll talk about more later in this post.
3. Timestamps with gauges: Standard StatsD, as defined by Etsy, uses gauges which provide you with a metric at a point in time. AppFirst StatsD extends this capability by allowing you to provide a timestamp defining when this measurement was taken.
4. Reliable and encrypted data: StatsD normally sends data in the clear (unencrypted) via UDP, which does not guarantee delivery. AppFirst sends the data via HTTPS which gives us not only encryption, but also the underlying reliability of TCP. Ensuring that your data is not only sent safely, but that you don’t lose any of your metrics either.
5. Correlation: In order to see the big picture, what’s really happening, requires looking at multiple sources of information in a given time frame. AppFirst allows you to correlate all of your statsd metrics with detail process data, server hardware information, log data, and application data. In a single visualization you can get a very complete picture of what’s happening inside your system.
7. Alerting: Given the importance of metrics that you really care about – that you have extracted from inside your code – it’s really important to get notification when somethings not right. You can set up alerts based on thresholds of your counters, gauges, and timers. Alerts can be sent via email and SMS text.
External Statistics Metrics.
Additionally you can combine external sources of statistical counters, which we collect using polled data, with your StatsD metrics by use of the AppFirst hierarchical namespace management (told you we’d talk about it again).
But first, lets clarify with what polled data is….
At AppFirst, we describe the ability to gather information by executing certain plugins on intervals as “Polled Data”. Conceptually, this is what most traditional monitoring technologies do: Execute small scripts (commonly known as plugins) at intervals in order to collect information. One of the most widespread examples of this is Nagios and the use of Nagios plugins. In a Nagios monitoring system, plugins are executed in (for example) 5 minute intervals. AppFirst has integrated the Nagios plugin standard, which defines how results are output to standout out. The result is that any existing Nagios plugins or any plugin that emits output defined by the Nagios plugin standard will work with AppFirst.
Now back to external statistic metrics.
There are a number of stats counters: namely Microsoft’s Windows Performance Counters and Java Managed Beans. Since we know how important these external stats counters metric might be to you, we knew it was equally important to make it possible to combine them with the stats counters you created in your own code. Therefore, we created the ability to take the external source (Performance Counters, Managed Beans, etc) and combine them with your StatsD metrics, using a hierarchical namespace. To do that, simply name your polled data metrics with metric.* (you can find examples here). This allows you to see your metrics in multiple places on the AppFirst web application: The Polled Data widget in Workbench, in Correlate as StatsD and in the dashboard.
We’re really happy with StatsD, both how Etsy made it and with how we’ve enhanced it. Being able to visualize it, both solo and in combination with other external metric counters, helps give a more complete picture of what is going on in your system.
Take a look at how to best utilize StatsD using AppFirst in this video (we recommend changing the settings to 1080p):