OOM kill occurred on an x86_64 NUMA system

Problem

When running memory intensive application(s), the kernel reports the following error messages in /var/log/messages: 

Jul 19 05:19:24 asl131 kernel: Out of Memory: Killed process 2368 (jserver).
Jul 19 05:19:24 asl131 kernel: Out of Memory: Killed process 2368 (jserver).
Jul 19 05:19:24 asl131 kernel: OOM kill occurred on an x86_64 NUMA system!
Jul 19 05:19:24 asl131 kernel: The numa=off boot option might help avoid this.

Solution

Systems with processors featuring AMD64 and Intel® EM64T are typically configured as NUMA platforms, which means that the kernel constructs multiple memory nodes at boot-time rather than constructing a single memory node. The multiple node construct can result in memory exhaustion on one or more of the nodes before other nodes become exhausted. When memory exhaustion happens, the following could result:

  1. The system will swap the exhausted nodes while there is available memory on other nodes, resulting in poor overall performance
  2. Processes are killed due to Out-Of-Memory (OOM) errors even though there is available memory
  3. Less than optimal performance due to excessive memory bandwidth when processes running on an exhausted node allocate memory on one or more different nodes

The kernel is capable of detecting when OOM process kills occur on AMD64 and Intel® EM64T NUMA systems due to memory exhaustion on one or more nodes while there is available memory on other nodes. When the kernel detects this condition, it displays the following warning at the console:

OOM kill occurred on an x86_64 numa system! The numa=off boot option might help avoid this.

Adding numa=off at the boot command line in /boot/grub/grub.conf and rebooting the system disables the creation of separate memory nodes at boot-time and elminates the possibility of one memory node becoming exhausted even though other nodes have available memory.