DAMON: Data Access Monitor

The trend of increasing data intensive workloads and limited DRAM capacity makes dynamic access pattern based optimal memory management important. Such mechnisms can be made only when not only accurate but also efficient dynamic access pattern monitoring is available.

DAMON is a Data Access MONitoring framework subsystem for the Linux kernel that developed for such memory management. It is designed with some core mechanism (refer to Design for the detail), which can make it

• accurate (the monitoring output is useful enough for DRAM level memory management; It might not appropriate for CPU Cache levels, though),
• light-weight (the monitoring overhead is low enough to be applied online), and
• scalable (the upper-bound of the overhead is in constant range regardless of the size of target workloads).

DAMON therefore makes development of any access pattern based memory management available. For easier development of such system, DAMON further provides a feature called DAMOS (DAMon-based Operation Schemes). Using it, DAMON users can develop and run their access-aware memory management with no code but simple configurations. Simple mechanisms made on top of DAMOS achieve up to 12% performance improvement and 91% memory saving.

Detailed evaluation of DAMON and DAMON-based system optimizations are available at another post.

Demo Video

Demo Screenshot

Recent News

2023-03-14: The schedule for DAMON talk at OSSummit NA is available now.

2023-03-10: A DAMON talk proposal for OSSummit NA has accepted.

2023-03-06: DAMOS filters feature has introduced as one of the most significant changes for Linux v6.3 by an LWN article

2023-02-24: A preprint of an academic paper that compares their approach against DAMON has uploaded to ArXiv.

2023-02-13: LSF/MM/BPF topic proposal for DAMON has posted

2023-02-09: DAMON debugfs deprecation patchset has posted

Getting Started

You can start using DAMON by

• installing DAMON-enabled kernel and its user-space tool,
• following the tutorial of the user-space tool, and
• run the automated tests suite.

By following those, you will be able to know if DAMON works on your machine and how you can use it.

Install

DAMON is implemented in the Linux kernel, so you should install DAMON-enabled Linux kernel to use it. To check if you’re already running DAMON-enabled kernel, you could:

$ if grep CONFIG_DAMON /boot/config-$(uname -r); then echo "installed"; fi

If you are an Amazon Linux 2 user, you can install a DAMON-enabled Linux kernels which are based on v5.10 or v5.4 using amazon-linux-extras. For example, if you want to use DAMON with Amazon Linux v5.10 kernel, below commands will work.

$ sudo yum install -y amazon-linux-extras
$ sudo amazon-linux-extras install kernel-5.10
$ sudo reboot

Note that AL2 v5.10.y contains all DAMON features that available on mainline while AL2 5.4.y contains only a part of DAMON features, so AL2 v5.10.y is more recommended.

If your package system doesn’t support DAMON-enabled kernel, you can fetch a DAMON-merged Linux kernel source tree (it has merged in Linux mainline tree since v5.15), build, and install it. Note that you should enable kernel configuration options for DAMON, depending on your demands. For example:

$ cd $THE_FETCHED_DAMON_KERNEL_SOURCE_TREE
$ make olddefconfig
$ echo 'CONFIG_DAMON=y' >> /.config
$ echo 'CONFIG_DAMON_KUNIT_TEST=y' >> /.config
$ echo 'CONFIG_DAMON_VADDR=y' >> /.config
$ echo 'CONFIG_DAMON_PADDR=y' >> /.config
$ echo 'CONFIG_DAMON_VADDR_KUNIT_TEST=y' >> /.config
$ echo 'CONFIG_DAMON_SYSFS=y' >> /.config
$ echo 'CONFIG_DAMON_DBGFS=y' >> /.config
$ echo 'CONFIG_DAMON_DBGFS_KUNIT_TEST=y' >> /.config
$ echo 'CONFIG_DAMON_RECLAIM=y' >> /.config
$ echo 'CONFIG_DAMON_LRU_SORT=y' >> /.config
$ make -j$(nproc)
$ sudo make modules_install install

Source Code

There are several DAMON-merged Linux kernel source trees for different users. You may pick one among those based on your needs.

For users who want only a stable version of DAMON, below trees are available.

• Linus Torvalds’ mainline tree contains the latest stable version of DAMON since v5.15.
• Amazon Linux 2 v5.10.y tree and v5.4.y tree also contain stable versions of DAMON. Note that AL2 v5.10.y contains all DAMON features that available on the latest major releases of Linux while AL2 5.4.y contains only a part of DAMON features, so AL2 v5.10.y is more recommended.

If you have interests in DAMON features under development, below trees will be appropriate. These trees contain latest version of DAMON which having features under development.

• mm-unstable contains the latest DAMON patches, which are under testing with other unstable memory management subsystem patches. So this tree is likely unstable and frequently updated, but would be a good baseline for your DAMON development.
• damon/next contains the latest changes, which might not tested at all. So this tree is likely more unstable and frequently updated than mm-unstable. This tree also contains some changes that exist only for DAMON hacks itself rather than eventually be merged in the mainline.

For people who have interest in DAMON features under development but use LTS kernels as their baseline, there were trees that based on three latest LTS kernels.

NOTE: Below trees were continuously getting DAMON backports on latest 5.4.y, 5.10.y, and 5.15.y. But those are deprecated as of 2022-09-03, and therefore there will be no update to the trees.

• damon/for-v5.15.y
• damon/for-v5.10.y
• damon/for-v5.4.y

The source code of DAPTRACE, which is a prototype of DAMON, is also available.

User-space Tool

A user-space tool for DAMON, which is called DAMO, is available at Github and PyPi. You may start using DAMON by following the Getting Started of the tool for start.

Tests Package

There is a tests suite for correctness verification and performance evaluation of DAMON. Those are actively used for the development of DAMON. Using that, you can test DAMON on your machine with only a few simple commands or deeply understand the user interface of DAMON.

So, if you finished the tutorial but have no idea about what to do next, running the tests would be a good start. If any test fails, please report that to the maintainer via mail (sj@kernel.org or damon@lists.linux.dev) or github.

Official Document

The official document of DAMON in the latest mainline for users/sysadmins and kernel programmers are available. Document for DAMON in the damon/next tree, which focused on DAMON only and contains some more DAMON-specific contents, is also available.

Showcase Website

There is a showcase website that you can get more information of DAMON. The site hosts

• the documentation of DAMON that contains contents on damon/next tree,
• the heatmap format dynamic access pattern of various realistic workloads for heap area, mmap()-ed area, and stack area,
• the dynamic working set size distribution and chronological working set size changes, and
• the latest performance test results.

Evaluation Results

Evaluation of DAMON and DAMON-based system optimizations are available.

DAMON-based System Optimization Guide

A guide for DAMON-based system optimizations are also available.

Community

DAMON is maintained and developed by its own community, which is a sub-set of the linux kernel development community.

The community is mainly driven by the mailing list (damon@lists.linux.dev). All the patches are posted there and reviewed. Almost every discussions are also made there. You can subscribe to the mailing list via subspace.kernel.org following the instruction. You could also read the mails without subscription via lore.

The community also have an open, regular, and informal virtual bi-weekly meeting series for DAMON community called DAMON Beer/Coffee/Tea chat series.

Publications and Presentations

• SeongJae Park, Current Status and Future Plans of DAMON. In The Linux Kernel Summit, September 2022. Slides, Video, Link
• Jonathan Corbet, LRU-list manipulation with DAMON. In Linux Weekly News, August 2022. Article
• SeongJae Park, Current Status, Future Plans, and Possible Collaborations for DAMON. In The Linux Kernel Summit, September 2022. Link
• SeongJae Park, Madhuparna Bhowmik, Alexandru Uta, DAOS: Data Access-aware Operating System. In The 31st International ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC'22), June 2022. Paper, Slides, Poster
• SeongJae Park, Writing a fine-grained access pattern oriented lightweight kernel module using DAMON/DAMOS in 10 minutes. In The Linux Kernel Summit, September 2021. Slides, Video, Link
• Jonathan Corbet, Using DAMON for proactive reclaim. In Linux Weekly News, July 2021. Article
• SeongJae Park, DAMON: Data Access Monitoring Framework for Fun and Memory Management Optimizations, In The Linux Kernel Summit, August 2020. Slides, Video, Link
• Yunjae Lee, Yunhee Kim, and Heon. Y. Yeom, Lightweight Memory Tracing for Hot Data Identification, In Cluster computing, 2020. Paper
• Jonathan Corbet, Memory-management optimization with DAMON. In Linux Weekly News, February 2020. Article
• SeongJae Park, Yunjae Lee, Heon Y. Yeom, Profiling Dynamic Data Access Patterns with Controlled Overhead and Quality. In 20th ACM/IFIP International Middleware Conference Industry, December 2019. Paper
• SeongJae Park, Tracing Data Access Pattern with Bounded Overhead and Best-effort Accuracy. In The Linux Kernel Summit, September 2019. Slides, Link
• SeongJae Park, Yunjae Lee, Yunhee Kim, Heon Y. Yeom, Profiling Dynamic Data Access Patterns with Bounded Overhead and Accuracy. In IEEE International Workshop on Foundations and Applications of Self-* Systems (FAS* 2019), June 2019. Paper

News

2023-03-14: The schedule for DAMON talk at OSSummit NA is available now.

2023-03-10: A DAMON talk proposal for OSSummit NA has accepted.

2023-03-06: DAMOS filters feature has introduced as one of the most significant changes for Linux v6.3 by an LWN article

2023-02-24: A preprint of an academic paper that compares their approach against DAMON has uploaded to ArXiv.

2023-02-13: LSF/MM/BPF topic proposal for DAMON has posted

2023-02-09: DAMON debugfs deprecation patchset has posted

2022-12-29: DAMON development summary of 2022 has shared and featured by Phoronix.

2022-12-16: The DAMOS filtering for anon pages and/or memory cgroups have merged in mm tree.

2022-10-19: An RFC patchset for efficient query-like DAMON monitoring results have posted

2022-09-15: The video for my kernel summit DAMON talk this year is now available at Youtube

2022-09-09: The plan for the first in-person DAMON community meetup at LPC and the in-person office hour at OSSummit EU has announced

2022-09-06: AL2 5.10 kernel’s DAMON code has updated to that of v5.19

2022-08-30: AL2 5.10 kernel’s DAMON code has updated to that of v5.18

2022-08-22: LWN introduced DAMON-based LRU-lists manipulation (DAMON_LRU_SORT) in detail

2022-08-15: LWN introduced DAMON’s new features including ‘LRU_SORT’ as significant changes for Linux 6.0

2022-08-12: Bi-weekly DAMON Beer/Coffee/Tea Chat series for open, regular, and informal community syncups and discussions has announced.

2022-07-29: Current status, future plans, and possible collaborations for DAMON will be presented at the Kernel Summit 2022.

2022-06-26: The poster of the DAOS paper is available online.

2022-06-13: DAMON-based LRU-lists sorting patchset has posted and immediately merged in the -mm tree

2022-05-04: A paper introducing DAMON and related works have accepted by HPDC22

2022-05-03: Now DAMON has its own open mailing list

2022-04-29: Patches for DAMON online tuning have merged in -mm tree

2022-04-27: Android has backported and enabled building DAMON and DAMON_RECLAIM for the common kernel.

2022-04-27: Alibaba has shared thier own DAMON user space tool.

2022-02-28: The DAMON sysfs interface patchset has merged in -mm tree.

2022-02-17: An RFC patchset for sysfs-based DAMON’s new user interface has posted.

2022-01-20: A roadmap of DAMON has shared.

2022-01-09: Linux 5.16 is released. “DAMON-based proactive memory reclamation, operation schemes and physical memory monitoring” are marked as prominent features of the release by the Kernel newbies and LWN.

2021-12-23: A great blog post for DAMON-enabled kernel has uploaded

2021-11-07: DAMON patches for automated memory management optimization, the physical address space monitoring support, and proactive reclamation have merged in the mainline.

2021-11-01: DAMON has released with Linux v5.15.

2021-10-14: DAMON_RECLAIM patchset is merged in the Amazon Linux 5.10.y kernel tree.

2021-10-02: DAMOS patchset is merged in the -mm tree.

2021-09-23: DAMON and DAMOS are presented in the kernel summit. Slides, Video, Link

2021-09-16: DAMON development tree on kernel.org is created.

2021-09-08: DAMON patchset is merged in the Linus Torvalds’ tree, aka mainline

2021-09-07: DAMON/DAMOS will be presented at the Kernel Summit 2021

2021-08-31: DAMON user-space tool is uploaded to the official Python packages system, PyPi

2021-08-06: DAMON patchset is merged in the -mm tree

2021-07-27: LWN published a second article introducing DAMON patchset series

2021-06-11: DAMON-based proactive reclamation RFC patchset has shared on the hackernews and introduced by a Phoronix article

2021-05-31: DAMON-based proactive reclamation RFC patchset has posted

2021-05-26: DAMON-enabled Amazon Linux 2 kernels have deployed to all users

2021-05-07: DAMON has merged in the public source tree for Amazon Linux v5.4.y kernel

2021-04-05: damo now supports heatmap visualization on the terminal

2021-03-31: DAMON user-space tool (damo) is released as an individual open source project

2021-03-19: DAMON has merged in the public source tree for Amazon Linux v5.10.y kernel

2021-03-04: DAMON supports for two latest LTS kernels announced

2021-03-03: DAMON is merged in v5.10 based public Amazon Linux kernel tree

2021-02-25: An example usage of DAMON for profiling is shared

2021-01-07: A runtime system for DAMON-based optimal operation scheme finding is released

2020-12-03: Further plans around DAMON is shared

2020-11-17: A real-world user story of DAMON is shared

2020-09-26: The tests package for DAMON is released under GPL v2 license

2020-08-19: A demo video is available

2020-08-05: DAMON will be presented at the Kernel Summit 2020

2020-06-04: Physical Memory Monitoring is now available

2020-05-18: DAMON showcase website is announced

2020-05-13: DAMON official document is uploaded online

2020-02-20: DAMON has introduced by an LWN article

2020-02-10: The first RFC of Data Access Monitoring-based Operating Schemes (DAMOS) has posted to LKML

2020-01-23: The RFC of DAMON has introduced by LWN’s ‘Kernel patches of interest’

2020-01-20: The first RFC patchset of DAMON has posted to LKML