How To Do Infrastructure Performance Monitoring: Key Metrics And Best Practices Follow

This is just one example of how infrastructure performance monitoring can help you resolve system malfunctions. Here, we explain how to do it, why you can’t skip this process, key data engineering metrics to track, and best practices to follow when establishing a monitoring process. 

Table of contents

• How does infrastructure monitoring work?

• Tools for infrastructure monitoring

• Why you need to monitor your infrastructure performance

• Infrastructure performance metrics to track

• Best practices for infrastructure performance monitoring 

• How NaNLABS can help you build a more performant and scalable infrastructure

Refine your infrastructure monitoring strategy with NaNLABS. Hop on a call with our team to discuss your infrastructure challenges and reevaluate which metrics to track.

Register for free

How does infrastructure monitoring work?

Infrastructure monitoring works by continuously gathering data from your cloud components and hardware and analyzing it to identify any performance, health, or availability issues. You can also use metrics to track security, cost optimization, and other pillars of great system designs established in the AWS Well-Architected Framework. 

 Pro tip: Present infrastructure monitoring data in at-a-glance dashboards.

This diagram explains how you can view system data depending on the area of the business it affects. For example, the data on a customer experience dashboard usually comes from your client’s perspective and the metrics that measure the functionality of your API microservices.

See how different infrastructure monitoring dashboards offer distinct views of the services and system’s performance. Source: AWS.

You can monitor the performance of your infrastructure by tracking two different types of data points:  

• Service metrics. AWS services track default metrics depending on the service you’re using. For instance, AWS Lambda monitors the number of function invocations, throttles, and errors, as well as performance and concurrency metrics. 

• Custom metrics. You can also publish and monitor your own service or business metrics. These should complement the service metrics and allow you to gain more context for your particular use case. You can publish high-resolution metrics or single data points.

Tools for infrastructure monitoring

When choosing infrastructure monitoring tools, you can go with a solution that lets you track multiple areas of your system, or an out-of-the-box, all-in-one SaaS solution. 

An Amazon CloudWatch dashboard containing key infrastructure metrics and set alarms. Source: AWS.

For instance, Amazon CloudWatch lets you monitor multiple parts of your cloud and on-premise infrastructure. You can use:

• Amazon CloudWatch Logs for logging collection, aggregation, and discovery

• Amazon CloudWatch Metrics for collecting metrics out of the box from vendor services and creating custom data points for technical and business tracking

• AWS X-Ray for tracing your application and AWS services

• Amazon CloudWatch Alarms for defining KPIs and alarms on those

• Amazon CloudWatch Dashboard for viewing the state of metrics and alarms in a centralized dashboard

You can then add other AWS systems to cover the areas you’re missing. For example, use AWS CodePipeline for building, testing, and deploying automated CI/CD pipelines. 

Another alternative is to choose an all-in-one SaaS tool like New Relic. This product allows you to track all of your infrastructure and team metrics and visualize them together in one dashboard. This way, all relevant stakeholders can access the same information in one tool. New Relic also lets you set alarms and remediation steps as tasks on Jira. 

The choice depends on your specific monitoring requirements.

Why you need to monitor your infrastructure’s performance

You need to continuously monitor your infrastructure’s performance to stay on top of your service’s health, performance, and availability. Also, this allows you to anticipate problems instead of reacting to them when they happen and to avoid risking unplanned outages. 

Additionally, you need to monitor your infrastructure to install a first line of defense in case of outages that lets you identify its root cause—and meet data compliance regulations.

Key benefits of infrastructure monitoring

Observing and tracking system metrics brings many benefits to your business, including: 

• Better performance and reliability 

• Higher customer and employee satisfaction

• Increased scalability that adjusts to the fluctuating demand

• Enhanced cost savings and avoiding having to pay for underused services

• Better data ingestion, processing, storage, and quality

• More efficient development and deployment pipelines

However, to gain these benefits from infrastructure monitoring, you need to observe the right metrics. Below are the ones we recommend you track. 

Want to optimize your cloud’s performance and costs? Sign up for the webinar “From Confusion to Control: Proven Strategies to Decode Cloud Complexity” on September 26th.

Register for free

11 infrastructure performance metrics you can’t skip

There are hundreds of different metrics you can track to measure your system’s health, performance, and availability. Here are 11 we recommend you don’t skip:

1. Task rebound and bug count

Task rebound shows many times a task was returned from testing due to bugs. Bug count refers to the number of defects found in the software, either during development or post-release. 

A high task rebound and bug count is a red flag as these KPIs indicate that you may need to improve your processes, requirements gathering, or testing protocols. This may also indicate that you need to upskill your team. 

2. Change in lead time

This refers to the variation in the amount of time it takes to move a task or feature from initiation to completion. The result can imply an increase or decrease in lead time. 

An increase in lead time may indicate issues with your requirements clarity, bottlenecks in the pipeline, or inefficient development practices. 

3. Mean time to recovery (MTTR) 

Track the average time it takes to recover from a failure. This is a critical metric as it indicates your team’s ability to respond to and resolve issues efficiently. 

A high MTTR could imply that your infrastructure problems are outside of your team’s capabilities or coordination. It’s also a direct indicator of your system’s resilience. 

Plus, this metric has a direct impact on your business as a fast recovery usually means less downtime and higher customer satisfaction and trust. 

4. Recovery time objective (RTO) 

RTO refers to the maximum allowed duration between a failure event and the recovery of the system. This metric answers the question: How quickly can we get back up and running after an incident? 

4.1. Recovery point objective (RPO)

RPO, on the other hand, accounts for the maximum acceptable amount of data loss, indicating how far back in time data can be restored. This KPI answers the question: How much data can we afford to lose?

Achieving lower RTOs and RPOs often requires significant investment in infrastructure, technology, and processes. So, it’s crucial that you balance the cost of these measures against the potential impact of downtime or data loss.

It’s also important that you regularly test your disaster recovery plans to ensure that your RTOs and RPOs are realistic and achievable.

5. Deployment failure rate

This metric calculates the percentage of deployments that fail, causing service disruptions or requiring rollback. Ideally, with an efficient CI/CD pipeline, this rate should be quite low. 

So, a high deployment failure rate is often a direct indicator of issues with your CI/CD pipeline, maybe due to inadequate testing, poor code quality, or insufficient rollback procedures.

A high deployment failure rate also impacts innovation and customer satisfaction as delaying new feature launches or updates can affect your credibility.

6. System uptime and availability

These metrics track the percentage of time the service was up and running over a specific period. The goal is for this metric to be close to 100%. The main difference between these two KPIs is that system uptime takes into account all the time a system is running, while availability takes into account maintenance and downtime. 

It goes without saying that high uptime and availability are a direct measure of your system reliability and have a direct impact on your customer satisfaction and trust. 

7. System response times

The time it takes for the system to respond to a request. This is measured using percentile (p).

Depending on the service level agreements (SLAs) you provide, this number might be p95, p99, p99.9, or even p99.999…

Note: We use percentile, instead of max, to prevent one single, or a couple of, delayed requests to skew the whole data point.

You should track this metric because a system with slow response times can lead to a poor user experience and may drive customers to churn. Also, slow response times could indicate that your infrastructure can’t handle the increased load—crucial for growing businesses. 

8. Error rates

This refers to the frequency of errors occurring in the system, including application errors, server errors, or network errors. 

High error rates could indicate issues with code quality, technical debt, or inefficient architecture. This metric also has a direct impact on your customer service team and end user experience as frequent platform errors could frustrate them, causing an increase in complaints.

9. Infrastructure utilization KPIs

Track metrics related to the usage of compute, memory, disk, and bandwidth resources in the infrastructure. These will vary depending on the services you have, deployment configurations, business needs, and stage of the business. 

Examples of specific infrastructure KPIs to track: 

• CPU load average

• CPU idle time

• Total memory

• Used/free memory

• Disk read/write rates

• Disk I/O

• Disk utilization/capacity

• Hardware errors

• Service/process status

• Total cloud spend

• Resource utilization

• Cost per user or team

• Cloud ROI

Infrastructure usage KPIs allow you to monitor and optimize resource usage and spend. These metrics also let you improve your capacity planning to ensure that the infrastructure can meet current and future demands.

10. Code security static analysis 

This indicates the number of security failures identified during static analysis of the code. Here, you examine the code for vulnerabilities without executing it. 

Since a high result may indicate potential security risks that could lead to breaches, tracking this metric ensures the code follows security standards and regulation requirements. 

Pro tip: If you integrate static analysis into the CI/CD pipeline, you’ll be able to identify and address security issues early in the development process.

11. Customer complaints and severity count

This stands for the number of complaints received by the end user, along with the severity of those issues. Ideally, you want to prevent this number from growing. If a customer is the one notifying you of product issues, there is a big gap in your monitoring processes. 

Also, a high number of complaints can cause your customer-facing team to burnout, which could decrease the quality of service they provide or lead them to quit. 

Consider this as a lagging metric. Don’t rely on it but use it as your cue to improve your systems in case you see this number increasing.

Best practices for infrastructure monitoring

You’ve seen how each metric is an indicator of underlying issues with your infrastructure. Here are the best practices we follow at NaNLABS for monitoring our systems that you can follow too: 

• Have monitoring alarms and centralized dashboards. This allows you to get notified in case a metric reaches a certain threshold through your preferred tools, e.g., a Slack message. Plus, viewing your metrics in a centralized dashboard lets you spot patterns, trends, and insights into your KPIs, e.g., see if the number of complaints is going up but still in the safe zone. 

• Establish a process around reliability. Define a standard operating procedure (SOP) to respond to and deal with incidents. Include who will be accountable for and when it happens, for instance, you may install an on-call rotation system with your team. Explain how this process works in the SOP. 

• Rely on automation to give responses or trigger a remediation plan when possible. For example, if your servers are running out of capacity, you may need more aggressive scaling. However, you don’t need to wake up one of your team members in the middle of the night to add more boxes. Instead, use automation to trigger a message on your tool to alert end users of the system running slow and conduct a proper analysis during working hours.

• Be mindful of over-aggressive alarms. If your alarms are constantly going off, it’ll become a cry-wolf tale—your engineers will start ignoring them. Make sure your alarms are the exception, not the norm. So, choose the right thresholds when automating notifications to avoid this from happening.

How NaNLABS can help you build a more performant and scalable infrastructure

By tracking and optimizing these metrics, you can ensure your infrastructure is reliable, efficient, and secure, directly contributing to overall business success. Regularly reviewing these metrics and acting on the insights they provide is essential for maintaining high performance and customer satisfaction.

If your team doesn’t have the capacity to establish monitoring processes or improve the state of your infrastructure, we can help you out. At NaNLABS we have over 11 years of experience in data engineering consulting and team augmentation services. We’ve helped hundreds of clients build more performant, reliable, and scalable infrastructures, as well as giving them the tools to improve their internal processes. 

Want to try it out for yourself? Hop on a call with our team and get a glimpse of how we can improve your system infrastructure.

Book a 30-minute free consultation