A Guide to Application Monitoring Best Practices

Mukul Khanna  on May 25, 2021

The software industry has been leading the automation trend by setting up tools, frameworks, and platforms to simplify and automate application development and deployment workflows. This enables less human involvement, less error-prone situations, and more consistency and reliability in rolling out software. But just like automation in building, testing, shipping, and deployment pipelines, it is only reasonable to have a setup that, after deployment, ensures the smooth functioning of the very many components that make up your application. A robust performance monitoring setup for applications in production is the need of the hour!

Because it is not just about putting your application out there, you also need to monitor its performance – its resource usage, user traffic, request rates, response times, bottlenecks, memory issues, etc. – to be able to overcome these limitations and ensure a good end-user experience. This is why Application Performance Monitoring (APM) tools have proven valuable to the software industry.

However, it is not just about being aware of the requisite tools and simply plugging them into your stack. You also need to understand established best practices that have worked well for the industry – effective methods and tips to get the most out of these tools. Best practices are best for a reason.

This post will dive into Application Performance Monitoring and give a real sense of its utility and value in today’s day and age of booming web applications. We will do so by discussing a bunch of use cases where these tools shine. We will then take a look at some of the best practices to be followed in the realm of monitoring to learn to extract the most value out of these APM tools. 

Here’s a preview of what we will be covering so you can easily navigate or skip ahead in the guide:

What is Application Performance Monitoring (APM)?

Application Performance Monitoring (APM), as the name suggests, is the process of monitoring the performance of the many aspects of your application.

When an end-user logs into your application, for even just one web page to load on their device, there are very many backstage components that need to come together and operate in synchrony to ensure a smooth and fast experience. These include network components (that carry the bytes of data), software components (e.g., server-side frameworks, front-end code, and other dependencies), and hardware components (i.e., CPU processors, memory, and storage of machines that host your web servers, APIs, databases, file systems, etc.) It can become overwhelming to manually keep track of your application performance on all these different levels and across all components. This is even truer when you ideally want monitoring and checks to happen all the time, in real-time!

Well, this is precisely the problem that APM solutions target. APM tools, like Scout APM, allow organizations to get a detailed analysis of the performance of their applications, in real-time. This includes critical information about server requests, response times, time-consuming methods and end-points, errors and their root cause analysis, and lots more – presented in a way that is easy to understand and troubleshoot.

These performance insights provide a lot of valuable information about more optimum resource allocations and effective cost reductions while surfacing other issues that could potentially fail your application – all of this and more before the user gets a hint of anything being amiss.

Why you Need an APM Tool for your Application

Apart from presenting a bird’s eye view of what is happening within your application as a whole, APM tools provide you with your application’s score on particular metrics that quantify its performance along different grounds. 

They provide metrics like request rates, response times, server load, CPU and memory usage, application throughput, server health status, and lots more, enabling organizations to understand what drives their application’s performance or failures.

They bring to light and help you identify performance bottlenecks, memory leaks, bloat, slow database queries, wasted execution cycles, and much more in your application. Additionally, tools like ScoutAPM enable teams to trace the cause of these issues to the specific line of the code causing them so that developers need to spend less time debugging and more time building.

Different platforms, frameworks, and APIs allow you to monitor the performance of a few of your applications’ components – for example, your cloud service provider could provide information about resource usage, logging frameworks could help you capture backend errors and processing times, etc. But wouldn’t it be much more useful to have everything you need under one roof – as a one-stop platform to provide all the information about everything you might need to know about your application’s performance.

Different organizations might want to optimize their application’s performance on different metrics. Some teams might want to prioritize more reliability and uptime, over other applications that might want to focus on higher speeds and lower response times. In this regard, equally important is the amount of flexibility that many of these tools offer in creating customizable dashboards – allowing you to focus on aspects of performance that matter the most to your application.

undefined
[Source]

APM tools, therefore, can go a long way in resolving issues faster, preventing interruptions, boosting performance, increasing business and revenue, and understanding customer interactions. To dive even deeper into the realms of Application Monitoring – the tools, their importance, and their many benefits, you can also check out the “Application Performance Monitoring - What is APM?” on our blog.

Let us now look at some common use cases of APM solutions to get a pragmatic understanding of how helpful they can be for developers and organizations to ensure that everything about their application is on track.

Application Performance Monitoring (APM) Use Cases

Use case 1: Application Development

Application development involves a lot of playing around with the code, tweaking, solving bugs, adding features, experimenting with different libraries and frameworks, refactoring, etc. This can lead to minor fluctuations in performance that developers might want to track and monitor throughout the development lifecycle and in the staging and production environments. 

Therefore, application development can benefit a great deal from the insights provided by APM tools. These could be insights about the application’s performance or an in-depth analysis of issues down to the code level. By highlighting the source of the problem and isolating issues to specific lines (or methods) in the code causing them, these tools narrow down the areas of the project that they should be focusing more on. 

Below is an example of code traceability in ScoutAPM, with Github integration enabled. You can read more about it here. 

undefined
Source: ScoutAPM Docs

Use case 2: Identifying Performance Bottlenecks

A bottleneck in software engineering refers to the negative effect on performance caused by the limited ability or capacity of one component of the system – similar to impeding water flow caused near a bottle’s constricted neck. A bottleneck is like the slower car on a single-track road that keeps everyone else waiting.

Even with the best software and hardware infrastructure in place, all it takes is one sub-optimal component to make your application crawl when it could be flying. APM tools help you identify performance bottlenecks with accuracy. These range from bottlenecks in disk usage, CPU utilization, memory to software and network components. APM platforms like Scout provide a complete analysis of several metrics like the memory allocation, response times, throughput, and error rates corresponding to each end-point in your application. Metrics like these provide insights into the long-term performance of these applications and help highlight where such bottlenecks lie.

undefined
Scout’s Endpoint Dashboard

If you are interested in learning more about performance bottlenecks, we have explored the topic in great detail in the How to Steer Clear of Application Performance Bottlenecks post on our blog.

Use case 3: Real-time Performance Alerts and Insights

APM tools like Scout provide live alerts and insights about your application’s performance. Many applications can benefit from the real-time nature of these alerts and updates. For example, you might not discover several memory bloats and leak issues until there is a decent amount of traffic on your website. And it’s not always possible to predict surges in user traffic. Therefore, in such a case, alert notifications from these APM tools can serve as a handy alarm signal – from a system that can 24 x 7 be on the lookout for such short-term anomalies and immediate failures. If something goes wrong, they can send out alerts through all your integrated platforms (e.g., Slack). This ensures issues are given attention before the end-user experiences any inconvenience. With these tools, there’s much more flexibility and customization offered; for example, options to configure the events you want to be alerted about, their duration, priority levels, messaging platforms, etc. Below is a snapshot of what this dashboard looks like in ScoutAPM.

undefined
Alerts Configuration Dashboard in ScoutAPM

Use case 4: Monitor and Track End User Experience

When evaluating your application’s performance, you might want to go beyond monitoring server response times, memory consumption, throughput, etc. On most occasions, what matters equally (if not more) is the end user’s experience. Several APM tools, like Scout, measure this using an ApDex score. The Application Performance Index, or Apdex, is essentially a quantifiable measurement of a user’s general level of satisfaction when using an application. Broadly, it is calculated based on the ratio of requests completed within a threshold amount of time. Therefore, the higher the ApDex score, the higher the supposed customer satisfaction levels concerning the speed and performance of your application.

undefined
ApDex score in ScoutAPM

You can read more about the ApDex score and how ScoutAPM measures customer satisfaction in the Monitoring ApDex with Scout APM post on our blog.

Here are some more use cases of APM tools worth mentioning:

Application Performance Monitoring (APM) Best Practices

Now we have a good understanding of what APMs are, their many use cases, and why you need one for your application.  Assuming you already use an APM tool or will get one for your application, let’s look at how you can make the most out of these tools. 

Here are the 10 best practices that you can follow for your APM setups:

Best Practice 1: Don’t Build Your Own APM Solution(s) 

Every organization at some point or the other comes across this build vs. buy dilemma – do you develop your own set of tools for your project(s), or do you go ahead and buy an existing, working solution? However, when it comes to APM tools, you are much better off not trying to handcraft one for your organization. 

The primary purpose of APM tools is to take care of unexpected issues in your application. If you build a custom APM solution for your application, you expose yourself to more issues across two platforms – your project and the APM tool itself. Therefore, it is much better to rely on more foolproof, trustworthy, specialized APM tools in the market. Besides, many of these tools are quite affordable and are therefore a much more viable option. 

There are already many challenges and difficulties associated with running and maintaining applications at scale. By using an existing reliable APM solution, you now have one less thing to worry about. Do what you do best, and let outsourced expertise take care of the rest.

Best Practice 2: Ensure You Have the Right Tool(s) for the Job 

After discussing the advantages of going for a third-party APM suite instead of building one for your own, it’s time to focus on things you need to keep in mind when opting for a tool that works best for your application. Before choosing an APM tool, it is important to do the research on your business, application requirements, service-level agreements (SLAs), and customers; then, see what feature set suits your setup best. There are a bunch of factors that you need to consider for this. These include the APM’s feature set, pricing model, flexibility, programming language support, data granularity, user interface, integration with other tools and services, technical support, ease of use, and many more. 

Some APM tools might focus on monitoring some minimal but essential operations, while others may go above and beyond in providing a comprehensive list of features.

Therefore, having a good understanding of what you need out of these tools and at what cost, will help you make an informed decision.

Are you confused about the plethora of APM tools in the market? We have got you covered with an elaborate analysis of the top 9 APM tools in the “A Comparison of the Top 9 Application Performance Monitoring Tools” post that you can check out on our blog.

Best Practice 3: Set Up a Customized Dashboard with the Most Useful Information  

When using APM tools for monitoring large-scale applications, you are likely to have many metrics, graphs, and other data visualizations thrown at you, which can sometimes make it difficult to focus on the aspects of performance that really matter. To this end, several APM tools provide an option to customize the appearance of your dashboard page. 

undefined
ScoutAPM dashboard

Your dashboard is the first page that opens up when you log in to your APM service. This page is supposed to present you with a broad idea of how your application is faring overall. Therefore, when setting up your APM tool, you should spend some time understanding what metrics provide the most relevant and vital information about the functioning of your application. Once that is done, the next obvious step is to ensure that these receive considerable attention in the dashboard you set up.

Best Practice 4: Prioritize Critical Transactions 

Along the same lines, in most cases, you’ll find that some transactions in your application are more critical compared to others. For example, you would be more concerned about the user’s home page’s response times than those of a rarely used static Terms and Conditions page.

undefined
ScoutAPM Endpoints Dashboard

These last two practices should declutter your setup quite a bit – letting the more important metrics and functions shine through and convey more actionable information about what is working well and what needs to be optimized.

Best Practice 5: Configure Custom Alert Policies and Notifications 

When issues arise and performance drops, individuals in your team need to be updated about the impact before the end-user catches wind of anything. After all, what is the point of a real-time alerting system if these alerts aren’t set up properly, or if they don’t reach you where you’d like, or you are unable to act on them?

As we have discussed before, each application and organization differs in different aspects. Therefore, based on your requirements, you might want to set up your own set of alerting conditions – because what may be worth receiving an alert for in your application will differ from what it would be for in another application. All APM tools allow you to create alerting policies by specifying thresholds on different metrics like response times, error rates, ApDex score, etc.

undefined
Creating New Alert Conditions in ScoutAPM
undefined
Alert Conditions List

Apart from email alerts, tools like ScoutAPM also allow you to integrate these updates with a messaging platform like Slack, making it easier for teams to stay updated and collaborate easily.

undefined
ScoutAPM alert example in Slack using Zapier

Also, you might not want to just stop at setting up these alert policies. There should be internal systems and processes in place that define the delegation of these issues and other practices that can ensure the corresponding issues are resolved efficiently.

Best Practice 6: Factor in the End User Experience 

As we previously discussed, it is the end user’s experience that matters most. And end-users expect faster applications and smoother experiences. Therefore, it is important to be on the lookout for patterns related to the ApDex score of your application. For example, you might observe sudden drops in the score after particular deployments or repeated peaks in response times that might rapidly lower the ApDex score, indicating a likely infrastructure limitation.

Your application’s ApDex scores are visible throughout the ScoutAPM platform and can be easily toggled on or off at any time. 

undefined

Therefore, when other performance metrics are alone unable to present a clear narrative of your application’s performance, it can be a good practice to start with the ApDex score and take things from there.

Best Practice 7: Keep Up with the Manual Checks (every once in a while) 

The software industry is an extremely fast-paced one. Things quickly change – usage patterns evolve, requirements change, expectations change. Alerts and policies once set up might not be relevant throughout. Given how precariously poised many of our applications are and the various number of things that can go wrong, it’s quite important to have the occasional manual checks to ensure that things are in order. This can include periodic checkups for inconsistencies and inaccuracies and ensuring the metrics and policies initially set up are scaling with the growth of your application. After all, no news is not good news.

Best Practice 8: Be Intelligent in Interpreting Metrics – Don’t Oversimplify 

Most APM tools in summarizing performance metrics present an average (or mean) of all metric values.

Even though an average is perhaps the most plausible (and most easily understood) indicator of performance on a broader scale, organizations might want to occasionally dig deeper into these metrics. Let’s see why this is through an example.

Consider an end-point that has a mean response time of 1 ms. This gives an impression of most (if not all) users experiencing a 1 ms response time. However, the average response time can also find a way to be 1 ms even when 20 percent (or any arbitrary percentage less than 50) of your audience experiences a 2x (or 3x or 4x) response time. This is what we like to refer to as the average fallacy. In this case, it is quite possible for individuals to overlook the response times’ distribution and be content with the lower overall average. However, if there is a careful analysis of the metrics, as shown below, the organization can focus on ways to optimize the response times for the rest.

undefined
ScoutAPM Endpoint Analysis

Best Practice 9: Train Personnel for Working with APM Tools 

As you can see, APM tools do need some playing around with to get acquainted with the many features they offer. Usually, getting the most value out of them requires some hands-on experience and understanding of the best practices to follow. Therefore, it would be a good practice to train individuals in your team about the operations of these tools.

Organizations can decide for themselves whether they need a dedicated group that overlooks APM operations or wants everybody to pitch in and take APM insights for their own work. For example, developers can benefit a lot from some minimal training with the APM platforms. This can help them understand the business importance of application performance, and code and build software accordingly. On the other hand, if a dedicated group is assigned to take care of everything APM, it makes things much more systematic as there is a clear understanding of responsibilities. This minimizes the chances of things falling through the cracks.

Most APM tools offer great documentation that makes getting started with these tools super easy!

Best Practice 10: Don’t Hesitate to Seek Help 

All top APM tools out there provide excellent technical support for their customers and provide support quickly. However, in working with the most highly intuitive platforms, teams might not feel the need to consult or seek external help. As a result, operators might miss out on several useful features, tips, and tricks. 

Therefore, it can be pretty valuable to get some guidance and insight from these support teams. They have much more experience dealing with everyday issues that organizations might face with their APM tool and therefore provide constructive feedback. This can therefore be quite helpful in improving the way you utilize the APM tool, helping you make the most out of it.

Summary and Important Takeaways 

It is important to note that if you are just starting with web development and working on smaller, personal projects, understanding the importance of APM tools might not come easily or seem super relevant to you. However, these tools become exponentially more valuable as your application(s) scale up and cater to hundreds or thousands of users.

If you are interested in:

then check out ScoutAPM by signing up for a free trial - you can thank us later.

Cheers! Happy coding!