The Linux kernel is an incredible circus performer, carefully juggling many processes and their resource needs to keep your server humming along. The kernel is also all about equity: when there is competition for resources, the kernel tries to distribute those resources fairly.
However, what if you’ve got an important process that needs priority? What about a low-priority process? Or what about limiting resources for a group of
The kernel can’t determine what CPU processes are important without your help.
Most processes are started at the same priority level and the Linux kernel schedules time for each task evenly on the processor. Have a CPU intensive process that can be run at a lower priority? Then you need to tell the scheduler about it!
There are at least three ways in which you can control how much CPU time a process gets:
- Use
the nice
command to manually lower the task’s priority. - Use
the cpulimit
command to repeatedly pause the process so that it doesn’t exceed a certain limit. - Use Linux’s built-in control groups, a mechanism which tells the scheduler to limit the
amount of resources available to the process.
Let’s look at how these work and the pros and cons of each.
Simulating high CPU usage
Before looking at these three techniques, we need to find a tool that will simulate high CPU usage on a system. We will be using CentOS as our base system, and to artificially load the processor we can use the prime number generator from the Mathomatic toolkit.
There isn’t a prebuilt package for CentOS so you will need to build it yourself. Download the source code from http://mathomatic.orgserve.de/mathomatic-16.0.5.tar.bz2 and then unpack the archive file. Change directory mathomatic-16.0.5/primes
make
sudo make install
matho-primes
/usr/local/bin
Run the command like this:
/usr/local/bin/matho-primes 0 9999999999 > /dev/null &
This will generate a list of prime numbers from zero to nine billion nine hundred ninety-nine million nine hundred ninety-nine thousand nine hundred ninety-nine. Since we don’t really want to keep the list, the output is redirected /dev/null
Now run top and you will see that the
Exit top (press the q key) and kill the
nice
nice
Start two matho-primes
tasks, one with nice and one without:
nice matho-primes 0 9999999999 > /dev/null & matho-primes 0 9999999999 > /dev/null &
Now top
Observe that the process started without nice
(at niceness level 0) gets more processor time, whereas the process with a niceness level of 10 gets less.
What this means in real terms is that if you want to run a CPU intensive task you can start it using nice and the scheduler will always ensure that other tasks have priority over it. This means that the server (or desktop) will remain responsive even when under heavy load.
Nice has an associated command renice
renice
:
renice +10 1234
Where 1234 is the PID.
Don’t forget to kill matho-primes
nice
renice
cpulimit
cpulimit
nice
cpulimit
is useful when you want to ensure that a process doesn’t use more than a certain portion of the CPU. The disadvantage nice
To install it on CentOS type:
wget -O cpulimit.zip https://github.com/opsengine/cpulimit/archive/master.zip unzip cpulimit.zip cd cpulimit-master make sudo cp src/cpulimit /usr/bin
The commands above will download the source code from GitHub, unpack the archive file, build the binary, and copy it /usr/bin
cpulimit
is used in a similar way nice
cpulimit -l 50 matho-primes 0 9999999999 > /dev/null &
Note how matho-primes
You can also limit a currently running process by specifying its PID using the‘-
cpulimit -l 50 -p 1234
Where 1234 is the PID of the process.
cgroups
Control groups (
The advantage of control groups nice
cpulimit
nice
cpulimit
By judiciously using
To demonstrate
The groups are created with the cgcreate
command like this:
sudo cgcreate -g cpu:/cpulimited sudo cgcreate -g cpu:/lesscpulimited
The “-g cpu†part of the command cpuset
memory
blkio
The
To set the
sudo cgset -r cpu.shares=512 cpulimited
To start a task in a particular
sudo cgexec -g cpu:cpulimited /usr/local/bin/matho-primes 0 9999999999 > /dev/null &
If you top
This is because when a single process is running, it uses as much CPU as necessary, regardless of which
Now start a second
sudo cgexec -g cpu:lesscpulimited /usr/local/bin/matho-primes 0 9999999999 > /dev/null &
The top command shows us that the process in the
Now start another
sudo cgexec -g cpu:cpulimited /usr/local/bin/matho-primes 0 9999999999 > /dev/null &
usageblog/image04.jpg”/>
Observe how the CPU is still being proportioned in a 2:1 ratio. Now the matho-primes
TL;DR
The finite resources of any server or desktop are a valuable commodity. The tools described above help you manage those resources, especially the CPU resource:
- nice is a great tool for ‘
one off ‘ tweaks to a system. cpulimit is useful when you need to run a CPU intensive job and having free CPU time is essential for the responsiveness of a system.cgroups are the Swiss army knife of process limiting and offer the greatest flexibility.
More servers? Or faster code?
Adding servers can be a band-aid for slow code. Scout APM helps you find and fix your inefficient and costly code. We automatically identify N+1 SQL calls, memory bloat, and other code-related issues so you can spend less time debugging and more time programming. We have Ruby, Python and Elixir agents.
Ready to optimize your site? Sign up for a free trial.
Updated version of an article first published on November 4th, 2014.
derek@scoutapp.com