As an author and practitioner in the DevOps space, I spend a good deal of my quality time with brilliant engineers and visionary product managers who have amazing capabilities. An area where I enhance their solutions is isolating, simplifying, and eliminating cyber security threats across the development lifecycle into production.
This a challenge as we move to greater abstraction their is a distinct need for a hacker mindset at each stage. I am pulling together research on this topic and look forward to plenty of hysterics and feedback 😉
For now … I wanted to share a nice article on Logging and Monitoring. A core area of trouble that by applying these principles should be less opaque. Looking forward to collaborating this year.
Should services log? To what detail? Where should those logs go? To a syslog daemon, to files on disk, to a message queue? When is it appropriate to start instrumenting your code? Once you’ve started, what sort of information should you be recording? And what metrics system makes sense? Here are a few recommendations :
- Understand that logging is expensive. I’ve seen entire teams of absolutely brilliant engineers spend years building, managing, and evolving logging infrastructure.
- Services should only log actionable information.
- Serious, panic-level logs may have to be consumed by end users.
- Structured data needs to be consumed by machine, to take necessary actions.
- Logging has limited applicability.
- Avoid multiple production levels (warn, info, error) and runtime level configuration. In production you should be logging information that needs to be seen. An exception is debugging which is useful during development and problem diagnosis. Therefore, make the application log level a dynamic configuration variable.
- It’s the responsibility of the OS or the agent to route the stdout/stderr to the desired destination.
- Make sure that your logging agent is lightweight and does not consume too many system resources. Monitoring In God we trust, the rest we monitor.