Linux ：clear for memory spaces when memory becomes full

Applies to:

Sun Network QDR InfiniBand Gateway Switch - Version All Versions and later
Sun Datacenter InfiniBand Switch 36 - Version All Versions and later
Exalogic Elastic Cloud X4-2 Hardware - Version X4 to X4 [Release X4]
Information in this document applies to any platform.

Goal

How to clear for memory spaces when memory of an IB switch (NM2-36P and NM2-GW) becomes full or almost full.

When memory of an IB switch becomes full or almost full, its functionality and performance can degrade.  Above all, running out of memory also prevents the IB switch from being patched.

Solution

The easiest and most direct method to free memory is to reboot the IB switch (provided this is feasible during a maintenance window).

However, if customer is unwilling to reboot the IB switch, then there is another approach.  An example is given below.

# date ; head /proc/meminfo ; free -m

# sync ; sleep 10 ; sync
# echo 1 > /proc/sys/vm/drop_caches ; free -m  ## frees page cache
  OR
# echo 2 > /proc/sys/vm/drop_caches ; free -m  ## frees dentry and inode_cache
  OR
# echo 3 > /proc/sys/vm/drop_caches ; free -m  ## frees page cache, dentry, and inode_cache

# date ; head /proc/meminfo ; free -m

Running sync writes out dirty/touched pages to disks.  Normally dirty pages are memory in use, so they are not available for reclaim.

Page cache is memory held after reading files.  Linux kernel prefers to keep unused page cache, assuming files being read once will most likely to be read again in the near future, hence avoiding performance impact on disk IOs.

dentry and inode_cache are memory held after reading directory/file attributes, such as open() and stat().  dentry is common across all file systems, but inode_cache is on a per-file-system basis.  Linux kernel prefers to keep this information assuming it will be needed again in the near future, hence reducing disk IOs.

The above approach applies to Linux kernel v2.6.16 or newer.  To determine Linux kernel version of the IB switch, customer can run:

# uname -a

If the above approach cannot free sufficient memory, then rebooting the IB switch becomes unavoidable.

Now, if customer is worried about a potential memory leak in the IB switch firmware, then he or she can check for memory growth over time via:

# date ; cat /proc/meminfo ; free -lt ; showfree -dm

[Wait sometime...]

# date ; cat /proc/meminfo ; free -lt ; showfree -dm
 

If free memory is shrinking, but, on the other hand, sizes held by slabs are also growing, then, specifically in the "cat /proc/meminfo" output, customer will need check the percentage of SReclaimable compared to Slab, and check whether SReclaimable is growing or SUnreclaim is growing.  Growth in SReclaimable is not an issue because they are freed slabs ready for next allocation, so they are not adding to memory pressure at all.

Slabs are memory used by Linux kernel to hold its own internal data structures.  So, if customer only focus on free memory, then a shrinking value can sometimes be misleading when SReclaimable do not count toward free memory.  Subsequently, the perception of the IB switch running out of memory can be inaccurate.

Additionally, customer can monitor the IB switch via top, a common command on Linux-based platforms.

# top

While top is running and updating its outputs on screen, customer can sort based on various fields.  This can reveal which process is actively taking up resources like CPU and memory.

- Press 'O' (shift + o)

- Press 'n' to sort based on %MEM (%).  Then, type any other key to return. 这个redhat没有

  OR

- Press 'q' to sort based on RES (kb).  Then, type any other key to return.

In closing, when Linux kernel approaches a limit specified in /proc/sys/vm/min_free_kbytes, it should be forced to reclaim all unused/inactive memory.  So, tuning vm.min_free_kbytes higher makes this reclaim process happening earlier.

# sysctl vm.min_free_kbytes

# vi /etc/sysctl.conf
...
vm.min_free_kbytes=<suitable_value_in_KB>
...

# sysctl -p
# sysctl vm.min_free_kbytes

---production

 sysctl vm.min_free_kbytes
vm.min_free_kbytes = 183863

--default

sysctl vm.min_free_kbytes
vm.min_free_kbytes = 10296