Running out of RAM in Ubuntu? Enable ZRAM

Orange Pi Development Boards

Whenever I ran out of RAM on a Linux system, I used to enable swap memory using the storage device to provide an extra bit of memory.  The main advantage is that it’s does not require extra hardware, but come at the cost of much slower access, and potential issues or wear and tear, unless you only use it temporary.

This week-end, I compiled Arm Compute Library on ODROID-XU4Q board, and the first time it crashed because the system ran out of memory, so I enable swap on the eMMC flash module to restart and complete the build successfully. However, I was told it would have been better to enable ZRAM instead.

So what is ZRAM? Wikipedia explains:

zram, formerly called compcache, is a Linux kernel module for creating a compressed block device in RAM, i.e. a RAM disk, but with on-the-fly “disk” compression.

So it’s similar to swap, expect it operates in RAM and compresses memory. It’s also possible to combine it with swap, but if you have to go that route, it may be worth considering upgrading the memory, or changing to a more powerful hardware platform. ZRAM support has been considered stable since Linux 3.14 released in early 2014.

Before showing how to use ZRAM, let’s check the memory in my board.


That’s about 2GB RAM, and swap is disabled. In theory, enabling ZRAM in Ubuntu or Debian should be one simple step:


Installation went just fine, but it did not enable ZRAM. The first thing to check is whether ZRAM is enabled in the Linux kernel by checking out /proc/config or /proc/config.gz:


ZRAM is built as a module. I tried different things to enable it and check, but I was not sure where to go at this stage at it was almost time to go to bed. What’s usually the best course of action in this case? Stay awake and work overnight to fix the issue? Nope! Rookie mistake. Years of experience have taught me, you just turn off your equipment, and have a good night sleep.

Morning time, breakfast, walk to office, turn on computer and board, et voilà:


Success! So I would just have had to reboot the board to make it work the previous day… So we have 1GB ZRAM swap enabled across 8 block devices. I assume those are used as needed to avoid eating RAM necessary.

So let’s try to use that swap by making it a bit more challenging that just building the Arm Compute Library, by running a few programs like Chromium and Thunderbird, and monitoring RAM usage at the same time with htop.

At the final stage we can see 506 MB “swap” is used, with 1.11GB memory, probably since the rest of the RAM is used for ZRAM. I was however wrong in myinitial assumption that ZRAM block devices would be used one by one as needed, as all eight devices were show to hold about the same amount of data:


That’s the output of free for reference:


The good news is that the build could complete just fine with ZRAM, even with Chromium and Firefox running in the background.

ZRAM requires compressing and decompressing data constantly, and reduces the amount of uncompressed RAM your system can access, so it may actually decrease system performance. However, if you run out of RAM frequently or for a specific application it may be worth enabling it. I just needed ZRAM for a single build, so I could disable it now by removing it:


However, you may also consider tweaking it by using zramctl utility:


Running zramctl without parameters allows use to see which how much actual compressed / uncompressed data is used per block:


zramswap service is handled by systemd in Ubuntu 18.04  and you could check out /etc/systemd/system/multi-user.target.wants/zram-config.service to understand how it is setup and control it at boot time too. For example, we can see one block device is set per processor. Exynos 5422 is an eight core processor, and that’s why we have 8 block devices here. Any other tips are appreciated in comments.

Support CNX Software - Donate via PayPal or become a Patron on Patreon

35
Leave a Reply

avatar
7 Comment threads
28 Thread replies
2 Followers
 
Most reacted comment
Hottest comment thread
16 Comment authors
Eversorback2futureWilliam BarathSuren Zaduriantheguyuk Recent comment authors
  Subscribe  
newest oldest most voted
Notify of
tkaiser
Guest
tkaiser

Switching to lz4 from lzo when possible is always a good idea since slightly faster compression times but magnitudes faster decompression. Also limiting zram devices to count of big cores on a big.LITTLE system could be a good idea. In my tests an A15 at 2 GHz was almost three times faster than an A7 at 1.4 GHz.

On the other hand these zram devices are just used as normal swap devices and the kernel treats these devices as being slow as hell since historically swap ends up on HDDs that show a horribly low random IO performance. So the kernel will take care that pages that end up swapped out (be it to disk or to such a zram device) will only be accessed again if really needed. That’s also what can be seen on your system with 506 MB used for swap while only utilizing 1.11 GB out of 1.95 GB.

Su Zet
Guest
Su Zet

Is there a reason why zram is not enabled by default on Armbian based on Debian stretch? I think it will be very useful especially for my OrangePi One with 512 MB RAM running OpenHab/InfluxDB/Grafana at same time.

tkaiser
Guest
tkaiser

It’s only active on recent Ubuntu Xenial builds (since on Ubuntu it’s simply adding the zram-config package to default package list and you’re done). I added a generic zram activation handler but other devs objected for reasons I partially forgot in the meantime.

But all you need is to edit line 422 in /etc/init.d/armhwinfo, remove the ‘#’ sign there and reboot:

https://github.com/armbian/build/blob/23a81c22a28264abb627c590ce6e29776a22b091/packages/bsp/common/etc/init.d/armhwinfo#L422

Su Zet
Guest
Su Zet

Thanks a lot for explanation.

Su Zet
Guest
Su Zet

Just as feedback as I tried your tweak and it worked but unfortunately the command of changing the compression algorithm (echo lz4 >/sys/block/zram${i}/comp_algorithm 2>/dev/null) is not working:

zram: Can’t change algorithm for initialized device

Suren Zadurian
Guest
Suren Zadurian

Thanks for reply but this error pops up from dmesg during the creation of zram devices done by the script armhwinfo tkaiser has pointed out. And in the script the creation of the devices is definitely after the algorithm selection. Anyway I’m glad that currently I have almost half of the memory free but I’m not sure if the kernel does not limit the caching to RAM assuming that the swap is slow which can lead to slower response of the system (this is not so critical as my processes are not so fast changing)

tkaiser
Guest
tkaiser

Unfortunately most of this zram stuff is kernel dependent (with Armbian we try to support +30 different kernels for the various boards but with some of them some things simply don’t work or work differently when trying to adjust things from userspace).

tkaiser
Guest
tkaiser

Two important notes wrt activating zram on Armbian (when not using Armbian’s Ubuntu variant where zram is enabled by default now).

1) It’s important to remove/comment the last line in /etc/sysctl.conf that reads ‘vm.swappiness=0’. This will restore default vm.swappiness (60) which is important to get a performant system when swapping to zram occurs.

2) At least with mainline kernel ‘comp_algorithm’ has to be defined prior to ‘disksize’ otherwise lzo remains active.

CampGareth
Guest
CampGareth

Ha, I just found this out by accident. I was about to start compiling large packages and thought zram would help but wouldn’t you know it, it’s installed by default on whichever armbian version I flashed! Thanks tkaiser, that’s allowed me to run two Make threads comfortably rather than one saving unknown but large amounts of time.

Eversor
Guest
Eversor

Agreed on limiting the number of zram devices. It is usually enough to use half the cores and I would not go further than 2, as to not impact latency when ZRAM is doing it’s thing.
I’ve used 1/2 A15 and for Ivy Bridge and been very happy thus far.

tonny
Guest
tonny

Nice info! Thanks!

tkaiser
Guest
tkaiser

The most common misunderstanding wrt zram is SBC users thinking zram would automagically reduce the amount of RAM available. People see half of the RAM assigned as swap via zram (as above, that’s Ubuntu’s default) and think now they could only use half of the RAM any more.

It’s not how this works. In the above example still 1.95 GB RAM is available. At one point in time 1.11 GB is used while 506 MB are reported as being swapped out. By looking at the zramctl output we see that there’s an actual 1:3 compression ratio so zram uses ~170 MB of the available and used DRAM to store the 506 MB pages that are swapped out.

So yes, zram is eating up available RAM when situations with higher ‘memory pressure’ occured but the kernel can deal with that pretty fine and you end up being able to run more memory intensive applications at the same time than without zram and this without the usual swapping slowdowns caused by ultra slow storage access with traditional swap on HDD (or SD cards / eMMC on SBC). There’s a reason zram is default on billions of devices (Android thingies).

To make it really efficient a few more tunables exist of course. E.g. on systems that only use zram and no traditional swap on spinning rust changing the default swap behaviour is a good idea (by default 8 pages are handled at the same time since HDDs are bad at accessing small chunks of data due to their nature). The following would tell the kernel to switch from 8 to just 1 page which is pretty fine with zram (and also swap on SD card and eMMC that also do not suffer from HDD behaviour):

C D
Guest
C D

note:

cat file.gz | grep something

may be shortened to

zgrep something myfile.gz

Catnip
Guest
Catnip

Every time there is a “needless use of cat”, the brain of a kitten implodes.

Tesla
Guest
Tesla

ZRAM is great, I am using it for over a year on 1 GB VPS and several SBCs. I confirm than lz4 for swapping works better than lzo. I also found that there is no need for several zram devices, that advise is a remnant from the times when zram code was not multithreaded. I create ZRAM swap alongside disk swap but give it higher priority and set vm.swappiness=100 to encourage it’s use. The kernel switches to disk swap when no more zram can be filled. I set it’s size 3x of RAM as my measurements also show 3x compression ratios and the overhead is negligible. One can also keep logs and tmp on it.

Tesla
Guest
Tesla

I thought you might fancy some stats:
1 GB VPS hosting ~10 websites with nginx, mysql and php uses 732 MB of ZRAM swap that is compressed to 195 MB (3.75x ratio). Reading ~70 KB/sec and writing ~12 KB/sec to ZRAM does not impact CPU at all. The system still has ~150 MB of disk cache and does ~200 read op/sec from SSD for ~30 sql queries/sec

Rocket
Guest
Rocket

but you didnt show your syntax for changing lzo to lz4 or setting a higher priority. You also didnt say how you changed to or grouped (in my case 4 instances of zram [4 core CPU]) into only one instance.

Tesla
Guest
Tesla

Yes, that would be too much for a comment. Basically it is all in manuals and searchable on the web. Do not be lazy 😉

Rocket
Guest
Rocket

It’s not about laziness, its about completeness. I am book marking this article to share with others and would be nice to say “Everything is explained” instead of bookmarking all those other places and linking them to here for people to run all over the web. I’m a firm believer in KISS and not giving people a trail to follow that get lost or give up before reaching the end. 🙂

Tesla
Guest
Tesla

Here is a good start: https://gist.github.com/sultanqasim/79799883c6b81c710e36a38008dfa374
Tune it to your liking. I still believe that the only way to comprehension is independent research 🙂

Drone
Guest
Drone

@Tesla said: “Yes, that would be too much for a comment. Basically it is all in manuals and searchable on the web. Do not be lazy… I still believe that the only way to comprehension is independent research.”

Yeah, now everyone else has to waste his or her time to duplicate the work you already did while you hide the solution because you think you know what is best for us. Or maybe you were just lazy and didn’t properly document your work, and this is a way for you to avoid admitting it. I sure hope it’s the latter because truly believing you are helping people by hiding things from them has no place in an Open Source community of Linux users!

Anyway, thanks for the one 2-year old GitHub link you provided, it’s better than nothing.

CampGareth
Guest
CampGareth

this person’s posted a comment on a website, they’re not writing up a how-to, they’re sharing anecdotal information.

Rocket
Guest
Rocket

if that were true, the author should not have included the hashtags How-to and Tutorial on this post at all.

theguyuk
Guest
theguyuk

Rocket you have to remember the clique here have contempt for users.

CampGareth
Guest
CampGareth

Aaah, thread over everyone, this person’s a regular troll on how-tos if this comment didn’t make it easy enough to tell:
modernromantix.com/2016/04/02/help-my-boyfriend-is-an-internet-troll-part-1/#comment-207

Also by this user, “You forgot to mention sudo is required”:
tecmint.com/cheat-command-line-cheat-sheet-for-linux-users/comment-page-3/#comment-753513

And “This guide about virtual machines, you didn’t mention virtual machines!”:
brianlinkletter.com/installing-debian-linux-in-a-virtualbox-virtual-machine/#comment-74904

mehmet
Guest
mehmet

> I also found that there is no need for several zram devices, that advise is a remnant from the times when zram code was not multithreaded.

Problem is you can never know which ancient version of ZRAM code will be running in those pesky vendor kernels so it is always a good idea to be on the safe side and create several zram devices.

William Barath
Guest

Couple things to note about ZRAM:
1) as you use more, there’s less and less room to keep inode cache and demand-paged shared files (mostly libraries and shared copies of forked daemons) in memory, so those demand-paged files are being evicted more and more often, resulting in increased read bandwidth to the system disk.
2) if you enable both ZRAM swap and disk swap, generally people apply the ZRAM swap at a higher priority so it gets used first. The problem is that the disk swap will only get used AFTER the ZRAM is full, and then the disk swap will become progressively less stale than the ZRAM – the ZRAM will become full of static paged-out content – which would make much more sense to page out to disk and free that RAM.
3) some configurations come with a certain amount of disk swap at a higher priority than the ZRAM swap. See LineageOS for example. The idea is that the disk swap will then fill with static paged-out content and performance will improve as the ZRAM gets better and better locality. But of course this is bad for startup performance.

In my own experience it’s best to allocate about 8 ZRAM disks with total size equal to system RAM, and then allocate disk swap the same or larger than the ZRAM disks. Then set up a cron job that watches for the system to be idle, ie load level 1.0 is idle for a 4-way SMP system. When that happens, and if the ZRAM is at least 75% full, then swapoff half the ZRAM disks. That will evict some of the least-often-used blocks to the disk swap. Then swapon the ZRAM disks again. Next time swapoff the other half of the ZRAM disks. Eventually the blocks that are never loaded will end up on disk and the ZRAM’s locality will approach the ideal, which will give you performance similar to actually having twice the RAM.

As for LZ4 vs LZO… I’m more excited about seeing Zstd enabled soon, which will give both better throughput and better ratio than LZO.

back2future
Guest
back2future

While feeling scheduler priority settings (time dependency) for ram organisation is just some kind of work around for not knowing what files are used intensively, it seems more interesting having a top like statistics tool, that would show locally file TYPES accessed and their time share of usage?
This knowledge followed by automation should improve swap prefetch (advanced anticipatory paging) and (configurable) page routing to ram or swap?

tkaiser
Guest
tkaiser

Since I never saw any benchmark comparing swap on zram vs. physical storage… I did one myself on an SBC platform: https://forum.armbian.com/topic/5565-zram-vs-swap/?do=findComment&comment=54487

NanoPi Fire3, most probably the most powerful, inexpensive and small SBC available today (octa-core A53 at 1.6 GHz for the same price you are charged for a lousy Raspberry Pi).

8 CPU cores, just 1 GB DRAM and cnxsoft’s try to build the ‘Arm Compute Library’ with OpenCL Samples.

Zram finishes the job in 47 minutes while swap on HDD takes 9.5 hours. The ‘average SD card’ shows even crappier random IO performance so in such situations compiling with swap on SD card could take days…

Simple conclusion: Use or at least evaluate zram. If you know what you’re doing and want to improve things explore using zram combined with traditional storage (you need to know that sequential and random IO performance are two different things — otherwise you’re only fooling yourself). Then zcache can be an interesting alternative to zram and the insights William Barath shared already.

back2future
Guest
back2future

“compcache” by Wheeler && team could be a source for other benchmark done before zram grew into mainline kernel. Zram has been available since more than 10 yrs now. So users are to blame for their steadiness, to some extent, but not only, like single board computers did a huge performance step during last 5 yrs and cpu power is needed for compression and decompression. Thx for Your numbers.
Maybe this will support a more appropriate (configurable) ram organisation, so that least updated pages could be swapped to lower bandwidth storage if necessary.
https://en.wikipedia.org/wiki/Page_replacement_algorithm#Least_recently_used