DAMON: Data Access Monitor

!! NOTE !!

This post has migrated to https://damonitor.github.io/posts/damon. This out-dated post will be removed soon.

With increasingly data-intensive workloads and limited DRAM capacity, optimal memory management based on dynamic access patterns is becoming increasingly important. Such mechanisms are only possible if accurate and efficient dynamic access pattern monitoring is available.

DAMON is a Data Access MONitoring framework subsystem for the Linux kernel developed for such memory management. It is designed with some key mechanism (refer to Design for the detail) that make it

• accurate (the monitoring output is useful enough for DRAM level memory management; It might not be appropriate for CPU Cache levels, though),
• light-weight (the monitoring overhead is low enough to be applied online), and
• scalable (the upper-bound of the overhead is in constant range regardless of the size of target workloads).

Therefore, DAMON can be used to develop memory management based on any access pattern. To make it easier to develop such systems, DAMON provides a feature called DAMON-based Operation Schemes (DAMOS). This allows DAMON users to develop and execute access-aware memory management without code but with a simple setup.

Simple mechanisms based on DAMOS can achieve up to 12% performance improvement and 91% memory savings. Detailed evaluation of DAMON and DAMON-based system optimizations are available at another post.

DAMON is also currently being used in real-world products including AWS Aurora Serverless v2 and SK hynix HMSDK v2.

Table Of Contents

• Demo Video
• Demo Screenshot
• Recent News
• Getting Started
• Install
• Source Code
• User-space Tool
• Tests Package
• Official Document
• Showcase Website
• Evaluation Results
• DAMON-based System Optimization Guide
• Profile-Guided Optimization Example
• Community
• Contribution
• Publications and Presentations

Demo Video

Demo Screenshot

Recent News

Below are only a short list of recent news. For complete list of the news, please refer to a dedicated post.

2024-10-15: DAMON debugfs interface removal RFC patch has posted.

2024-10-10: Monthly PyPI downloads of DAMON user-space too, DAMO, doubled again after ten days. !damo_9000_monthly_downloads !damo_rolling_monthly_downloads_2024-10-10

2024-10-08: Videos for DAMON recipes at Open Source Summit EU'2024 and DAMON long-term plans at Kernel Memory Management Microconference'2024 are uploaded to YouTube.

2024-10-01: 2024-Q3 DAMON news letter including news for new features, more users, repos reorganizations, and conference talks is posted.

2024-09-30: DAMON User Space Tool, DAMO, surpasses 4,000 monthly PyPI downloads!

Getting Started

You can start using DAMON by

• installing DAMON-enabled kernel and its user-space tool,
• following the tutorial of the user-space tool, and
• run the automated tests suite.

By following those, you will be able to know if DAMON works on your machine and how you can use it.

You can also participate in the development if you’re interested.

Install

DAMON is implemented in the Linux kernel, so you should install DAMON-enabled Linux kernel to use it. To check if you’re already running DAMON-enabled kernel, you could:

$ if grep -q '^CONFIG_DAMON=y' /boot/config-$(uname -r);
then
    echo "installed"
else
    echo "not installed"
fi

As of 2024-09-23, kernels of Linux distros including Amazon Linux, Android, Arch, CentOS, Debian, Fedora, and Oracle Linux are known to have enabled DAMON.

You could further find a list of DAMON-enabled Linux kernels from Oracle’s kconfigs tool. The tool doesn’t support every distros at the moment, though.

If your package system doesn’t support DAMON-enabled kernel, you can fetch a DAMON-merged Linux kernel source tree, build, and install it. Note that you should enable kernel configuration options for DAMON, depending on your demands. For example:

$ cd $THE_FETCHED_DAMON_KERNEL_SOURCE_TREE
$ make olddefconfig
$ echo 'CONFIG_DAMON=y' >> ./.config
$ echo 'CONFIG_DAMON_VADDR=y' >> ./.config
$ echo 'CONFIG_DAMON_PADDR=y' >> ./.config
$ echo 'CONFIG_DAMON_SYSFS=y' >> ./.config
$ echo 'CONFIG_DAMON_DBGFS=y' >> ./.config
$ echo 'CONFIG_DAMON_RECLAIM=y' >> ./.config
$ echo 'CONFIG_DAMON_LRU_SORT=y' >> ./.config
$ make -j$(nproc)
$ sudo make modules_install install

Source Code

There are several Linux kernel source trees having DAMON for different users. You may pick one among those based on your needs.

For users who want a stable version of DAMON, Linus Torvalds’ mainline tree is recommended.

If you have interests in DAMON features under development, below trees will be appropriate. These trees contain latest version of DAMON which having features under development.

• mm-unstable contains the latest DAMON patches, which are under testing with other unstable memory management subsystem patches. So this tree is likely unstable and frequently updated, but would be a good baseline for your DAMON development.
• damon/next contains the latest changes, which might not tested at all. So this tree is likely more unstable and frequently updated than mm-unstable. This tree also contains some changes that exist only for DAMON hacks itself rather than eventually be merged in the mainline.

For people who have interest in DAMON features under development but use LTS kernels as their baseline, three were trees that based on three latest LTS kernels.

NOTE: Below trees were continuously getting DAMON backports on latest 5.4.y, 5.10.y, and 5.15.y. But those are deprecated as of 2022-09-03, and therefore there will be no update to the trees.

• damon/for-v5.15.y
• damon/for-v5.10.y
• damon/for-v5.4.y

The source code of DAPTRACE, which is a prototype of DAMON, is also available.

User-space Tool

A user-space tool for DAMON, which is called DAMO (Data Access Monitoring Operator), is available at Github and PyPi. You may start using DAMON by following the Getting Started of the tool for start.

Tests Package

There is a tests suite for correctness verification and performance evaluation of DAMON. Those are actively used for the development of DAMON. Using that, you can test DAMON on your machine with only a few simple commands or deeply understand the user interface of DAMON.

So, if you finished the tutorial but have no idea about what to do next, running the tests would be a good start. If any test fails, please report that to the maintainer via mail (sj@kernel.org or damon@lists.linux.dev) or github.

Official Document

The official document of DAMON in the latest mainline for users/sysadmins and kernel programmers are available. Those for next major release are also available(users/sysadmins doc, kernel programmers doc).

Showcase Website

There is a showcase website that you can get more information of DAMON. The site hosts

• the heatmap format dynamic access pattern of various realistic workloads for heap area, mmap()-ed area, and stack area,
• the dynamic working set size distribution and chronological working set size changes, and
• daily performance test results.

Evaluation Results

Evaluation of DAMON and DAMON-based system optimizations are available.

DAMON-based System Optimization Guide

A guide for DAMON-based system optimizations are also available.

Profile-Guided Optimization Example

An example of DAMON-based profile-guided optimization is also available.

Community

DAMON is maintained and developed by its own community, which is a sub-set of the linux kernel development community.

The community is mainly driven by the mailing list (damon@lists.linux.dev). All the patches are posted there and reviewed. Almost every discussions are also made there.

For easy communication, a mailing tool called HacKerMaiL is recommended. The tool is maintained by DAMON maintainer and committed to support DAMON community.

If you prefer web-based interface for reading the mails, you can use lore. If you prefer the traditional subscription based mailing workflow, you can subscribe to the mailing list via subspace.kernel.org following the instruction.

The community also have an open, regular, and informal virtual bi-weekly meeting series for DAMON community called DAMON Beer/Coffee/Tea chat series.

Contribution

DAMON and its related projects including damo and hackermail are open source projects. You can contribute to the development following the guidelines. Please refer to below contribution guidelines of each project to further look into the process.

• DAMON
• damo
• hkml

Publications and Presentations

Below are featured and/or upcoming publications and presentations covering DAMON. For more complete list of those, please refer to another dedicated post.

What to read/cite?

For people who more familiar to academic papers, DAMON papers for Middleware'19 Industry and HPDC'22 are recommended to read and/or cite. The paper for Middleware'19 covers DAMON’s monitoring mechanisms and access pattern profiling-guided optimizations. The paper for HPDC'22 extends the coverage to DAMOS (automated access-aware system operations) and user-space driven auto-tuning.

• SeongJae Park, DAMON: Long-term Plans for Kernel That {Just Works,Extensible}. In Linux Kernel Memory Management Microconferenct at Linux Plumbers, Sep 2024. Slides, Video, Link
• SeongJae Park, DAMON Recipes: Ways to Save Memory Using a Linux Kernel Subsystem in the Real World. In Open Source Summit Europe, Sep 2024. Slides 1, Slides 2, Video, Link
• Jonathan Corbet, An update and future plans for DAMON. In Linux Weekly News, May 2024. Article
• SeongJae Park, DAMON Updates and Plans: Automation of DAMON tuning, tiering, and VM guest scaling. In Linux Storage | Filesystem | MM & BPF Summit, May 2024. Slides, Video, Link
• Jonathan Corbet, An update and future plans for DAMON. In Linux Weekly News, May 2024. Article
• SeongJae Park, DAMO[N,S]?: Implementing Self-Driven Data Access-Aware Efficient Linux System. In Open Source Summit North America, Apr 2024. Slides, Video, Link
• SeongJae Park, DAMON: Current Status and Future Plans. In Kernel Summit, Nov 2023. Slides, Video, Link
• SeongJae Park, Data Access Monitoring Operator (DAMO): User-Space Tool/Python Library for Access-Aware Profiling and Optimization of Your Linux Systems. In Open Source Summit Europe, Sep 2023. Slides, Video, Link
• Jonathan Corbet, A 2023 DAMON update. In Linux Weekly News, May 2023. Article
• SeongJae Park, DAMON, DAMOS, and DAMO: Kernel Subsystems and User-Space Tools for Data Access-Aware System Analysis/Optimizations. In Open Source Summit North America, May 2023. Slides, Video, Link
• SeongJae Park, DAMON updates and future plans. In Linux Storage | Filesystem | MM & BPF Summit, May 2023. Slides, Video, Link
• SeongJae Park, Current Status and Future Plans of DAMON. In The Linux Kernel Summit, September 2022. Slides, Video, Link
• Jonathan Corbet, LRU-list manipulation with DAMON. In Linux Weekly News, August 2022. Article
• SeongJae Park, Madhuparna Bhowmik, Alexandru Uta, DAOS: Data Access-aware Operating System. In The 31st International ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC'22), June 2022. Paper, Slides, Poster
• Jonathan Corbet, Using DAMON for proactive reclaim. In Linux Weekly News, July 2021. Article
• Jonathan Corbet, Memory-management optimization with DAMON. In Linux Weekly News, February 2020. Article
• SeongJae Park, Yunjae Lee, Heon Y. Yeom, Profiling Dynamic Data Access Patterns with Controlled Overhead and Quality. In 20th ACM/IFIP International Middleware Conference Industry, December 2019. Paper