Written By: Daniel Kulvicki, Solutions Director at Calavista
Our previous blogs have defined DevOps as a collaborative culture with its own defined practices, ideas, tools, technology, processes, and metrics. Integrating some of these elements into your workflow can help streamline and improve your development process. Today, I want to focus on metrics that we associate with DevOps. Read on to learn more about the different types of DevOps metrics, what they are, and how they can add to your development pipeline.
What Are DevOps Metrics?
If you drive a car, you will see several gauges (such as speed and fuel level) and possibly warning lights (such as Check Engine and Low Tire Pressure) on the dashboard that provide key information about how the car is performing. DevOps metrics provide the same visibility for your development and operational processes. Adopting DevOps processes means adopting a culture of collaboration and drive for improvement in your software development. In previous blogs, we split DevOps into six main pillars: Collaboration, Automation, Continuous Integration, Continuous Testing, Continuous Delivery, and Continuous Monitoring. DevOps metrics are a key element of Continuous Monitoring which provide key insights for many of the other pillars.
DevOps metrics are measurements and indicators that can be used as part of Continuous Monitoring to not just assess application performance upon completion, but to assess the efficiency of the development process as well.
Using DevOps metrics as part of continuous monitoring enables proactive tracking, analysis of data, and insight to potential automation steps. Metrics and key performance indicators (KPIs) are introduced for continuous monitoring to enable more insight into how the production code is both developed and running.
One thing that makes DevOps metrics stand out is that they are about the development process, not just the outcome. Many companies may rely largely – or even solely – on performance metrics, measurements and indicators of how the software functions once it is complete. But those who are familiar with DevOps know that there is much more to software development than the final product. Like the gauges on the car dashboard, these metrics are about how things are running – not where you will be when you get there.
Metrics like the ones below are more process-oriented, rather than outcome oriented, which can help assess the efficiency and effectiveness of software development pipelines or processes, providing insights to your DevOps operation. Not only that, but they also help ensure that your product is on the right track from the start, and that mistakes or inefficiencies are caught early.
Types of DevOps Metrics
Code Coverage Metrics
You can think of code coverage metrics as those that look at the amount of code covered by test automation tools like SonarQube or Clover. They’re kind of like metrics for your metrics tools – whereas test tools will tell you what percent of the tests you have are passing or failing, code coverage tools will tell you what percent of your code is actually being tested by those tests. This gives you an idea of how in-depth DevOps metrics can be, providing data on the way that you track and manage data.
For example, code coverage metrics may reveal that a chunk of your code is not being automatically covered, telling you that any tests done by those automation tools are not relevant to this portion of the code. Having a 100% pass rate may sound great, until you realize that you’re only testing 4% of your code base. Code coverage is one of the most common KPIs and is a great place to start if you want to start putting numbers to your development process.
DevOps & Quality Metrics
Development and quality metrics will give you an idea of how effective your development process is. This includes things like velocity, deployment frequency, change volume, failed deployment rate, and more. For example, velocity, one of the most common and essential development metrics, shows the average quantity of completed story points over previous sprints. It is both easy to measure and to understand and can be used to identify inefficiencies in your process, as well as make projections about forthcoming sprints. If you notice velocity slows down, you know that you need to take a closer look at your processes.
An example of a quality-related metric is Defect Injection Rate, which represents the number of defects that were discovered and reported during a particular phase of development. This can tell you about what iteration of product development your team has the most trouble with over projects, but it does not tell you the rate at which the errors are fixed. Mean Time To Recovery (MTTR), on the other hand, reflects your ability to respond appropriately to issues by tracking how much time elapses between issue identifications and resolutions. This is an example of a metric that takes continuous monitoring one step further; it is one thing to effectively detect errors, it is another thing entirely to address them rapidly.
Using development and quality metrics ensures that quality is baked into the product from the beginning through a sound development process. They allow you to keep tabs on your development team and give numbers and data to the entire process.
Application Performance Metrics
Application performance metrics are probably some of the most common metrics used in the industry, even by groups that don’t use DevOps practices. These types of metrics are more outcome oriented, measuring the way that software functions once it is “up and running.” Some examples include average response time, error rates, and application availability, among others.
Error rate is a well-known example, providing a measurement of the velocity at which errors are occurring in the system. This can be used to measure general performance once the application is launched, or you can use it to proactively diagnose larger failures, by running it in specific functional areas of an application as they are being built out.
Database Metrics
If you need to get an idea of how your databases are functioning, turn to database metrics like memory utilization, throughput, and more. These provide insight into any messiness within your databases which can be the cause of servers slowing down or crashing. Firstly, database throughput will tell you how much work is done by your database server per second, showing you how quickly it is able to process incoming queries. You can understand how long it takes for queries to be processed, or get a response, by measuring how effective your queries are. It should help point you to needed indexes or specific problem areas with your query.
Keeping track of database metrics such as these allows you to stay on top of database performance. A choked-up database makes it very difficult to get anything done or for anything to run effectively. Database metrics help you keep on track of your development process and identify potential bottlenecks.
Security & Vulnerability Metrics
Security and vulnerability metrics can tell you if there are any weaknesses in your current security systems and policies. They can be used to determine general susceptibility or specifically where the vulnerabilities are. When operating under a DevOps framework, there is a lot of collaboration, so keeping a close eye on security is of the utmost importance.
One example is static application security testing (SAST), which can automatically scan and identify vulnerabilities and flaws within the source code. On the other hand, dynamic application security testing (DAST) is a type of test that will examine applications for vulnerabilities that might not be visible in the source code once the application has been deployed.
These types of metrics can be measured by programs like Splunk, Intruder.io, or other security information and event management (SIEM) tools.
Setting Which Metrics to Use
Using metrics throughout the entire development process falls under the concept of Continuous Monitoring. This really does mean continuous, from development to app completion. This automated process provides real-time metrics like the ones discussed above to help you understand how your development process is working, how secure it is, and how effective the software it creates is.
There are even more metrics that could be used than the ones discussed in this blog. To utilize all of them may be too overwhelming or simply unnecessary depending on your goals. When deciding which metrics to incorporate into your DevOps pipeline, it is important to consider what information will be helpful to you. A metric means nothing if the information will not be utilized.
If you know you would like some help building out your DevOps pipeline and incorporating new metrics and monitoring strategies, email us at info@calavista.com. You can also subscribe to our mailing list at the bottom of the page to learn more about DevOps and software.
