In my previous article on hugepages, I discussed what hugepages are and talked about the page table, the Translation Lookaside Buffer (TLB) and TLB Misses, Page Walks and Page Faults. I also discussed how using hugepages reduces the amount of memory used and the also reduces the number of CPU cycles needed to do the logical to physical memory mapping.

In this post, I’d like to talk about how to use Hugepages with the Oracle database and with JVMs. I’ll also talk about Transparent Hugepages (THP) and why you should turn off this new Linux “feature”.

Hugepages and Oracle Databases

Hugepages are absolutely critical for performance of Oracle instances on Linux. When you consider how Oracle works, it makes sense. With Oracle instances, you have a large shared memory area (the System Global Area or SGA) that is attached to by multiple processes. This SGA is used predominantly for the buffer cache and the shared pool. Each process attaching to the SGA has its own logical to physical mapping of the memory pages. If I have a large shared pool and a large number of processes attached, there is a multiplicative effect in terms of the amount of memory I have to use for the pagetables. In highly concurrent environments, that means that I have a lot of processes mapping large memory mappings at the same time.

We have seen significant gains in performance when using hugepages in Oracle and significant degradations in performance when we don’t.

How do I know whether an Oracle database is using hugepages? Oracle makes it very easy – at instance startup, it logs information on hugepage (large page) usage in the alert.log for the instance. Here is an example of an instance whose SGA is entirely allocated using hugepages:

************************ Large Pages Information *******************
Per process system memlock (soft) limit = UNLIMITED
Total Shared Global Region in Large Pages = 3010 MB (100%)
Large Pages used by this instance: 1505 (3010 MB)
Large Pages unused system wide = 6163 (12 GB)
Large Pages configured system wide = 11000 (21 GB)
Large Page size = 2048 KB
********************************************************************

And here is an example of an instance whose SGA is not. Notice that Oracle provides a recommendation of how many additional hugepages need to be allocated (just for this instance):

************************ Large Pages Information *******************
Per process system memlock (soft) limit = 189 GB
Total Shared Global Region in Large Pages = 2690 MB (5%)
Large Pages used by this instance: 1345 (2690 MB)
Large Pages unused system wide = 2 (4096 KB)
Large Pages configured system wide = 53300 (104 GB)
Large Page size = 2048 KB

RECOMMENDATION:
Total System Global Area size is 50 GB. For optimal performance,
prior to the next instance restart:
1. Increase the number of unused large pages by
at least 24254 (page size 2048 KB, total size 47 GB) system wide to
get 100% of the System Global Area allocated with large pages
********************************************************************

Bottom line: Use hugepages with Oracle on Linux – don’t hesitate – just do it!

Hugepages and Postgres

Hugepages can be used starting in version 9.4 of Postgres. As we migrate to this newest version of Postgres, we will be researching the impact of hugepages on its performance.

Hugepages and JVMs

Hugepage support has been in Java since JDK 5.0u5, but there is less evidence and documentation on the use of hugepages for JVMs. There are very few references to the benefits – the ones I’ve seen point to an approximate 5-10% performance increase using hugepages.

To use hugepages in your Hotspot JVM, use the following arguments at startup time:

-XX:+UseLargePages
-XX:LargePageSizeInBytes=2m

For IBM’s JVM, use the “–Xlp” option. For JRockit, it is “-XlargePages”.

Note: there are a couple of bugs related to HotSpot crashes with JDK7 using hugepages on Linux. These bugs are fixed in JDK version 7u60 and JDK 8. So make sure you test well when using hugepages in your Hotspot JVMs.

If your application uses large JVM heap sizes and is memory intensive, you should benchmark with your JVM using hugepages to see what kind of benefit you get out of it.

Using and setting hugepages

Using hugepages on Linux involves making changes to the system limits and also setting / reserving the number of hugepages available on the system. You’ll need to know how much memory you need for the hugepages.  This would be the total size of all of your database instance SGAs or JVM heap sizes. For instance, if you have 4 Oracle instances on a server, you’ll have to total up all of the SGA sizes. If you have two JVMs on a server and you want to use hugepages for both of them, you’ll have to total up the max heap sizes (including perm gen) for both.  Round up a bit – you want to make sure there are enough hugepages to accommodate your needs.

limits.conf

Add the following entries into the “/etc/security/limits.conf” script, where the setting is at least the size of the total HugePages allocation in KB (HugePages * Hugepagesize).  For example:

* soft memlock 3145728
* hard memlock 3145728

or if you specify by user:

oracle hard memlock 3145728
oracle soft memlock 3145728

Set the number of hugepages

# sysctl -w vm.nr_hugepages=<value>
or
# echo <value> > /proc/sys/vm/nr_hugepages

Be sure to have your Linux administrator ensure that HugePages is allocated after system restarts. They can do this by adding an entry to the /etc/sysctl.conf or grub.conf files.

After hugepages have been configured, you can run the following command to check the number of used and available hugepages:

$ grep Huge /proc/meminfo

HugePages_Total: 11000
HugePages_Free: 6229
HugePages_Rsvd: 66
HugePages_Surp: 0
Hugepagesize: 2048 kB

To see how much space is being used by the page tables:

# grep PageTables /proc/meminfo

PageTables: 1244880 kB

Transparent Hugepages (THP)

Transparent hugepages were introduced in RHEL 6. However, we have seen significant performance problems with THP.  I recommend that this feature be turned off and that you explicitly set up hugepages. Oracle has gone so far as to remove THP from its UEK2 Linux distributions (MOS Doc ID 1557478.1)

How to detect whether you are using Transparent Hugepages.

# grep AnonHugePages /proc/meminfo

AnonHugePages: 632832 kB

cat /sys/kernel/mm/redhat_transparent_hugepage/enabled

OR

cat /sys/kernel/mm/transparent_hugepage/enabled (on non-RHEL Linux distros)

If it is set to ‘always’ or ‘madvise’ then you will want to set it to ‘never’.

Also, if you see a process called ‘khugepaged’, then THP is in use.

To disable THP at boot time, append the following to the kernel command line in /etc/sysctl.conf or grub.conf:

transparent_hugepage=never

Using hugepages is a great, but often under-utilized way of improving performance on large, memory-intensive applications.  Definitely give it a go on your Linux-based Oracle DB servers and research its effectiveness on your JVMs and Postgres DBs as well.  Please let me know your experiences in the comments section.