Concanon Presents:
Observability Series
Part 3: Logging
By Tony Kustwan
In Part 2 of the Observability Series, Application Performance Monitoring, we looked at application performance monitoring and the different application performance monitoring solutions. We also looked at why application performance is not enough, and the pros and cons of these specific solutions. In the third post of the Observability Series, we focus on logging.
What is Logging?
IT organizations focus their measurements on the underlying technology that is currently found in the data center. These include: the servers, network devices, and applications that support the delivery of services. When a machine goes down, an application fails, or a service degrades performance, IT worries about getting that particular service back up and running so they can move on to the next technical issue. Thus, log data plays an essential role in the observability landscape. Log data is used for various purposes such as IT operations, system, network, and application monitoring. In addition, we can also use log data for business analytics, security, and compliance.
What are some types of devices that potentially produce valuable log data? The typical devices that potentially will produce valuable log data are:
- Servers and Applications
- Network Appliances
- Containers
- Mobile Devices
- Sensors, IoT, and Industrial IoT
- Security Device
- Compliance Tools
How is logging different from monitoring?
Monitoring is about an objective measure of Key Performance Indicators collected through logs, traces, and metrics. It takes forethought, discussion, and planning to determine the correct measurements for an organization to make the best decisions.
What is Log Management?
Log management is the ideology of collecting event data about your system into log files. Systems administrators in the past would access individual systems and use grep or some other text editor to find a needle in a haystack. Over time we moved from siloed points of collection of event data into a centralized collection point.
While it is essential to move your event data into a central location to observe the behavior holistically of your services, it is equally important to ensure that event data in your logs is standardized. To explore this further, take a look at my white paper on SMART Monitoring. SMART monitoring method is focused on providing IT organizations with concentrated guidance on building service-centric monitoring and logging mechanisms for both applications and systems.
What are the typical collection options?
Tool specific collection agents
Syslog
The two major types of Syslog systems are Rsyslog and Syslog-ng for NIX-based systems and Kiwi for Windows-based systems. Syslog event data is typically used for hardware errors, application failures, lost contact, and misconfiguration. Syslog has been around for a very long time and is simple. It does have some downsides being that Syslog communicates for UDP protocol. Are you looking for a better approach to manage event data than Syslog? Check out Josh Nudell’s Ten Reasons to Replace Syslog With Cribl LogStream.
Windows Event Log/WMI
3rd Party log collection
There are 3rd party event data collections tools available in the IT ecosphere. Silectis/MagPie, Cribl, Kafka, or whatever other 3rd party log collection gives you the ability to grab log data and provides you the ability to send that data to whatever destination for event analytics of your choice. In the case of Kafka, event analytics tools can subscribe to specific data that is needed.
To Conclude
We looked at what logging is, what type of devices can potentially provide valuable event data. We also looked at how logging is different from monitoring, and we explored what log management is. Finally, we investigated the different types of tools and options to collect event data.
So, what is next in our observability series? Now that we have completed the Introduction Into Observability, Application Performance Monitoring, and Logging, we are going to take those three and show how they all fit together in our final blog post in this series, Observability – END GAME.
Need help gaining Insight in your Apps?
Concanon provides digital transformation and observability consulting to help you gain confidence and trust in your analytics. This is through applying the best of breed tooling to gather the right data for the right problems. Reach out today to find out more.
Concanon's Author
Tony Kustwan is the Director of IT Operations at Concanon LLC, a big data solutions company. He has over 20 years of experience managing and leading global IT initiatives in some of the largest environments. He thrives on solving a customer’s greatest challenges with simple and repeatable frameworks to collect, process, and enrich data for decision makers.