Which transactions are responsible, and what caused it?
In this video, we are going to take a look at what memory bloat is, and how you can use Scout to eliminate it from your applications.
Memory related performance issues have the potential to bring your entire application down, and yet, most APMs completely ignore this fact and fail to provide any useful way of monitoring memory usage at all. At Scout, we understand the importance of memory monitoring, and have added many useful features throughout our APM tool to help give you the visibility and peace of mind that you need.
What is Memory Bloat?
When we talk about memory bloat, we are referring to the way in which a web request or background job triggers a sharp increase in memory. If there happens to be enough memory allocated to service the request at this time, then usually this event will go unnoticed. But in the scenario when this memory is not currently allocated, a sudden spike will occur as the language and framework allocates this new memory.
Now the real issue is that after this allocation has occurred, the memory isn’t necessarily unallocated for quite some time. So these events can cripple your application and bring it down completely if you are not careful. The exact behaviour will vary between different languages and frameworks, but the principle remains the same.
Memory bloat is a different issue from a memory leak, which is often a slow and gradual issue which lasts until the application is restarted. Memory leaks can be notoriously difficult to track down, but they are relatively easy to workaround. Cases of memory bloat, on the other hand, are often quite easy to fix, despite their dangerous implications.
Where to start
Try to focus on the endpoints that trigger the most memory allocations in a single request, rather than those which have a higher throughput or higher average number of allocations over a period of time. These are typically the types of endpoints which, when accessed by a big user, will cause a dangerous memory allocation spike. In the case of this screenshot below, your instinct might be to start investigating how to improve Endpoint A, but it is actually Endpoint B which poses the greater risk.
Another thing to bear in mind is that memory bloat is often triggered by a single user, often your biggest customers. This is another example of why APMs are so crucial in ensuring the performance and reliability of your application, because you will only see these events occurring in a live, production environment.
How can you use Scout to identify Memory Bloat?
If you are installing Scout for the first time, then I always recommend that you checkout our Insights tabs first. You can find these on the main page of Scout, and they are a great place to start with finding your biggest performance issues. We have a special Memory Bloat Insights tab which we have created to make finding the causes of memory bloat as simple as possible.
Here we list the worst offending, memory hungry endpoints in your application in an easy to process list. We also show key information such as the amount of memory increase that occurred as a result of this endpoint, and, whether or not this was caused by a single user or by multiple users.
Unlike most APMs, in Scout, full memory allocation breakdowns are provided on all our traces (framework permitent). This gives you a powerful layer of visibility into how much memory was allocated at each step of a given web request or background job.
The light green in the bars represent how much memory is typically allocated for this request, and the darker green shows you how much memory was used in this request. This is a great way of visualising the amount of memory bloat that occurred.
And of course, with GitHub integration enabled, we have full backtraces available to show you the offending line of code which caused the memory bloat to occur.
Another approach that you could take if you want to get a general understanding of your most memory intensive endpoints is to look at the Web Endpoints or Background job pages, and sort by the Max Allocations field. You can then take a look at the details of a given endpoint and add the Allocations Max metric to the chart to examine how this changed over time. You’ve also got the ability to filter the Traces tabs at the bottom of the page by Max Allocations.
Here we can also see the power of using our Custom Context feature. If we take a few moments to specify that we would like to pass our current_user value to Scout with each request, then we can reap the benefits instantly by being able to see who caused this case of memory bloat.
Conclusion
As you can see, the root cause of memory bloat is not as obvious as you might have first thought, and that’s why an APM solution like Scout can be vital to ensuring you have that peace of mind.
Scout enables you to quickly identify sources of memory bloat, and it gives you full visibility on the memory of all the requests in your application. If you are not already using Scout, then why not sign up today for a free 14-day trial, and start finding and fixing your memory bloat issues today!
matthew@scoutapp.com