DAMON: Data Access Monitor

Last updated on Mon, Jul 22, 2024 9 min read damon

With increasingly data-intensive workloads and limited DRAM capacity, optimal memory management based on dynamic access patterns is becoming increasingly important. Such mechanisms are only possible if accurate and efficient dynamic access pattern monitoring is available.

DAMON is a Data Access MONitoring framework subsystem for the Linux kernel developed for such memory management. It is designed with some key mechanism (refer to Design for the detail) that make it

  • accurate (the monitoring output is useful enough for DRAM level memory management; It might not be appropriate for CPU Cache levels, though),
  • light-weight (the monitoring overhead is low enough to be applied online), and
  • scalable (the upper-bound of the overhead is in constant range regardless of the size of target workloads).

Therefore, DAMON can be used to develop memory management based on any access pattern. To make it easier to develop such systems, DAMON provides a feature called DAMON-based Operation Schemes (DAMOS). This allows DAMON users to develop and execute access-aware memory management without code but with a simple setup. Simple mechanisms based on DAMOS can achieve up to 12% performance improvement and 91% memory savings.

Detailed evaluation of DAMON and DAMON-based system optimizations are available at another post.

Table Of Contents

Demo Video

damo monitor for water_nsquared

Demo Screenshot

masim stairs snapshot masim stairs heatmap in ascii

Recent News

Below are only a short list of recent news. For complete list of the news, please refer to a dedecated post.

2024-07-21: Memory Management subsystem pull request for Linux v6.11-rc1 is posted with DAMON changes for CXL memory tiering, documentation of a mailing tool for newcomers, and minor fixups.

2024-07-18: DAMON topic for Linux Kernel Memory Management Microconference at LPC'24 has accepted.

2024-07-11: ATC'24 also published two DAMON-citing papers at the same time. The first one proposes a way to improve monitoring accuracy of DAMON, while the second one mentions DAMON can be useful for extensible memory management.

2024-07-11: A couple of OSDI'24 papers (1, 2) for memory tiering that references and exploring DAMON as a part of them are available now.

2024-07-01: DAMON Quaterly Newsletter for 2024-Q2 has posted.

Getting Started

You can start using DAMON by

By following those, you will be able to know if DAMON works on your machine and how you can use it.

You can also participate in the development if you’re interested.

Install

DAMON is implemented in the Linux kernel, so you should install DAMON-enabled Linux kernel to use it. To check if you’re already running DAMON-enabled kernel, you could:

$ if grep -q '^CONFIG_DAMON=y' /boot/config-$(uname -r);
then
    echo "installed"
else
    echo "not installed"
fi

You could also find a list of DAMON-enabled Linux kernels from Oracle’s kconfigs tool. The tool doesn’t support every distros at the moment, though.

DAMON is enabled in Amazon Linux kernels that based on v5.4 or newer Linux kernels. If you’re using Amazon Linux 2 and not using DAMON-enabled kernel on the distro, you can install DAMON-enabled kernels using its package manager, amazon-linux-extras. For example, if you want to use DAMON with Amazon Linux v5.10 kernel, below commands will work.

$ sudo yum install -y amazon-linux-extras
$ sudo amazon-linux-extras install kernel-5.10
$ sudo reboot

Note that AL2 v5.10.y contains all DAMON features that available on mainline while AL2 5.4.y contains only a part of DAMON features. AL2 v5.10.y kernels are recommended for the reason.

If your package system doesn’t support DAMON-enabled kernel, you can fetch a DAMON-merged Linux kernel source tree, build, and install it. Note that you should enable kernel configuration options for DAMON, depending on your demands. For example:

$ cd $THE_FETCHED_DAMON_KERNEL_SOURCE_TREE
$ make olddefconfig
$ echo 'CONFIG_DAMON=y' >> ./.config
$ echo 'CONFIG_DAMON_VADDR=y' >> ./.config
$ echo 'CONFIG_DAMON_PADDR=y' >> ./.config
$ echo 'CONFIG_DAMON_SYSFS=y' >> ./.config
$ echo 'CONFIG_DAMON_DBGFS=y' >> ./.config
$ echo 'CONFIG_DAMON_RECLAIM=y' >> ./.config
$ echo 'CONFIG_DAMON_LRU_SORT=y' >> ./.config
$ make -j$(nproc)
$ sudo make modules_install install

Source Code

There are several Linux kernel source trees having DAMON for different users. You may pick one among those based on your needs.

For users who want a stable version of DAMON, below trees are recommended.

  • Linus Torvalds’ mainline tree contains the latest stable version of DAMON since v5.15.
  • Amazon Linux kernel v5.10.y tree, v5.4.y tree and all trees that based on v5.15 or newer Linux kernel contain stable versions of DAMON.

Note that AL2 v5.10.y contains all DAMON features that available on the latest major releases of Linux while AL2 5.4.y contains only a part of DAMON features, so AL2 v5.10.y is more recommended.

If you have interests in DAMON features under development, below trees will be appropriate. These trees contain latest version of DAMON which having features under development.

  • mm-unstable contains the latest DAMON patches, which are under testing with other unstable memory management subsystem patches. So this tree is likely unstable and frequently updated, but would be a good baseline for your DAMON development.
  • damon/next contains the latest changes, which might not tested at all. So this tree is likely more unstable and frequently updated than mm-unstable. This tree also contains some changes that exist only for DAMON hacks itself rather than eventually be merged in the mainline.

For people who have interest in DAMON features under development but use LTS kernels as their baseline, three were trees that based on three latest LTS kernels.

NOTE: Below trees were continuously getting DAMON backports on latest 5.4.y, 5.10.y, and 5.15.y. But those are deprecated as of 2022-09-03, and therefore there will be no update to the trees.

The source code of DAPTRACE, which is a prototype of DAMON, is also available.

User-space Tool

A user-space tool for DAMON, which is called DAMO (Data Access Monitoring Operator), is available at Github and PyPi. You may start using DAMON by following the Getting Started of the tool for start.

Tests Package

There is a tests suite for correctness verification and performance evaluation of DAMON. Those are actively used for the development of DAMON. Using that, you can test DAMON on your machine with only a few simple commands or deeply understand the user interface of DAMON.

So, if you finished the tutorial but have no idea about what to do next, running the tests would be a good start. If any test fails, please report that to the maintainer via mail (sj@kernel.org or damon@lists.linux.dev) or github.

Official Document

The official document of DAMON in the latest mainline for users/sysadmins and kernel programmers are available. Those for next major release are also available(users/sysadmins doc, kernel programmers doc).

Showcase Website

There is a showcase website that you can get more information of DAMON. The site hosts

  • the heatmap format dynamic access pattern of various realistic workloads for heap area, mmap()-ed area, and stack area,
  • the dynamic working set size distribution and chronological working set size changes, and
  • daily performance test results.

Evaluation Results

Evaluation of DAMON and DAMON-based system optimizations are available.

DAMON-based System Optimization Guide

A guide for DAMON-based system optimizations are also available.

Profile-Guided Optimization Example

An example of DAMON-based profile-guided optimization is also available.

Community

DAMON is maintained and developed by its own community, which is a sub-set of the linux kernel development community.

The community is mainly driven by the mailing list (damon@lists.linux.dev). All the patches are posted there and reviewed. Almost every discussions are also made there.

For easy communication, a mailing tool called HacKerMaiL is recommended. The tool is maintained by DAMON maintainer and committed to support DAMON community.

If you prefer web-based interface for reading the mails, you can use lore. If you prefer the traditional subscription based mailing workflow, you can subscribe to the mailing list via subspace.kernel.org following the instruction.

The community also have an open, regular, and informal virtual bi-weekly meeting series for DAMON community called DAMON Beer/Coffee/Tea chat series.

Contribution

DAMON and its related projects including damo and hackermail are open source projects. You can contribute to the development following the guidelines. Please refer to contribution guidelines of each project to further look into the process.

For DAMON in Linux kernel, please also read the maintainer’s profile document.

Publications and Presentations

Below are featured and/or upcoming publications and presentations covering DAMON. For more complete list of those, please refer to another dedicated post.

  • SeongJae Park, DAMON: Long-term Plans for Kernel That {Just Works,Extensible}. In Linux Kernel Memory Management Microconferenct at Linux Plumbers, Sep 2024. Link
  • SeongJae Park, DAMON Recipes: Ways to Save Memory Using a Linux Kernel Subsystem in the Real World. In Open Source Summit Europe, Sep 2024. Link
  • Jonathan Corbet, An update and future plans for DAMON. In Linux Weekly News, May 2024. Article
  • SeongJae Park, DAMON Updates and Plans: Automation of DAMON tuning, tiering, and VM guest scaling. In Linux Storage | Filesystem | MM & BPF Summit, May 2024. Slides, Video, Link
  • Jonathan Corbet, An update and future plans for DAMON. In Linux Weekly News, May 2024. Article
  • SeongJae Park, DAMO[N,S]?: Implementing Self-Driven Data Access-Aware Efficient Linux System. In Open Source Summit North America, Apr 2024. Slides, Video, Link
  • SeongJae Park, DAMON: Current Status and Future Plans. In Kernel Summit, Nov 2023. Slides, Video, Link
  • SeongJae Park, Data Access Monitoring Operator (DAMO): User-Space Tool/Python Library for Access-Aware Profiling and Optimization of Your Linux Systems. In Open Source Summit Europe, Sep 2023. Slides, Video, Link
  • Jonathan Corbet, A 2023 DAMON update. In Linux Weekly News, May 2023. Article
  • SeongJae Park, DAMON, DAMOS, and DAMO: Kernel Subsystems and User-Space Tools for Data Access-Aware System Analysis/Optimizations. In Open Source Summit North America, May 2023. Slides, Video, Link
  • SeongJae Park, DAMON updates and future plans. In Linux Storage | Filesystem | MM & BPF Summit, May 2023. Slides, Video, Link
  • SeongJae Park, Current Status and Future Plans of DAMON. In The Linux Kernel Summit, September 2022. Slides, Video, Link
  • Jonathan Corbet, LRU-list manipulation with DAMON. In Linux Weekly News, August 2022. Article
  • SeongJae Park, Madhuparna Bhowmik, Alexandru Uta, DAOS: Data Access-aware Operating System. In The 31st International ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC'22), June 2022. Paper, Slides, Poster
  • Jonathan Corbet, Using DAMON for proactive reclaim. In Linux Weekly News, July 2021. Article
  • Jonathan Corbet, Memory-management optimization with DAMON. In Linux Weekly News, February 2020. Article
  • SeongJae Park, Yunjae Lee, Heon Y. Yeom, Profiling Dynamic Data Access Patterns with Controlled Overhead and Quality. In 20th ACM/IFIP International Middleware Conference Industry, December 2019. Paper
damon linux kernel mm projects
Avatar
SeongJae Park
Kernel Development Engineer

SeongJae Park is a programmer who loves to analyze and develop systems.

Related