I have used GNU/Linux on systems from 4 MB RAM to 512 GB RAM. When
they start swapping, most of the time you can still log in and kill
off the offending process - you just have to be 100-1000 times more
patient.
On my new 32 GB system that has changed: It blocks when it starts
swapping. Sometimes with full disk activity but other times with no
disk activity.
To examine what might be the issue I have written this program. The
idea is:
1 grab 3% of the memory free right now
2 if that caused swap to increase: stop
3 keep the chunk used for 30 seconds by forking off
4 goto 1
-
#!/usr/bin/perl
sub freekb {
my $free = `free|grep buffers/cache`;
my @a=split / +/,$free;
return $a[3];
}
sub swapkb {
my $swap = `free|grep Swap:`;
my @a=split / +/,$swap;
return $a[2];
}
my $swap = swapkb();
my $lastswap = $swap;
my $free;
while($lastswap >= $swap) {
print "$swap $free";
$lastswap = $swap;
$swap = swapkb();
$free = freekb();
my $used_mem = "x"x(1024 * $free * 0.03);
if(not fork()) {
sleep 30;
exit();
}
}
print "Swap increased $swap $lastswap\n";
Running the program forever ought to keep the system at the limit of
swapping, but only grabbing a minimal amount of swap and do that very
slowly (i.e. a few MB at a time at most).
If I run:
forever free | stdbuf -o0 timestamp > freelog
I ought to see swap slowly rising every second. (forever and timestamp
from https://github.com/ole-tange/tangetools).
But that is not the behaviour I see: I see swap increasing in jumps
and that the system is completely blocked during these jumps. Here the
system is blocked for 30 seconds with the swap usage increases with 1
GB:
secs
169.527 Swap: 18440184 154184 18286000
170.531 Swap: 18440184 154184 18286000
200.630 Swap: 18440184 1134240 17305944
210.259 Swap: 18440184 1076228 17363956
Blocked: 21 secs. Swap increase 2400 MB:
307.773 Swap: 18440184 581324 17858860
308.799 Swap: 18440184 597676 17842508
330.103 Swap: 18440184 2503020 15937164
331.106 Swap: 18440184 2502936 15937248
Blocked: 20 secs. Swap increase 2200 MB:
751.283 Swap: 18440184 885288 17554896
752.286 Swap: 18440184 911676 17528508
772.331 Swap: 18440184 3193532 15246652
773.333 Swap: 18440184 1404540 17035644
Blocked: 37 secs. Swap increase 2400 MB:
904.068 Swap: 18440184 613108 17827076
905.072 Swap: 18440184 610368 17829816
942.424 Swap: 18440184 3014668 15425516
942.610 Swap: 18440184 2073580 16366604
This is bad enough, but what is even worse is that the system sometimes
stops responding at all - even if I wait for hours. I have the
feeling it is related to the swapping issue, but I cannot tell for
sure.
My first idea was to tweak /proc/sys/vm/swappiness from 60 to 0 or
100, just to see if that had any effect at all. 0 did not have an
effect, but 100 did cause the problem to arise less often.
How can I prevent the system from blocking for such a long time?
Why does it decide to swapout 1-3 GB when less than 10 MB would suffice?
