DAMON: Data Access Monitor
With increasingly data-intensive workloads and limited DRAM capacity, optimal memory management based on dynamic access patterns is becoming increasingly important. Such mechanisms are only possible if accurate and efficient dynamic access pattern monitoring is available.
DAMON is a Data Access MONitoring framework subsystem for the Linux kernel developed for such memory management. It is designed with some key mechanism (refer to Design for the detail) that make it
- accurate (the monitoring output is useful enough for DRAM level memory management; It might not be appropriate for CPU Cache levels, though),
- light-weight (the monitoring overhead is low enough to be applied online), and
- scalable (the upper-bound of the overhead is in constant range regardless of the size of target workloads).
Therefore, DAMON can be used to develop memory management based on any access pattern. To make it easier to develop such systems, DAMON provides a feature called DAMON-based Operation Schemes (DAMOS). This allows DAMON users to develop and execute access-aware memory management without code but with a simple setup. Simple mechanisms based on DAMOS can achieve up to 12% performance improvement and 91% memory savings.
Detailed evaluation of DAMON and DAMON-based system optimizations are available at another post.
Table Of Contents
- Demo Video
- Demo Screenshot
- Recent News
- Getting Started
- Source Code
- User-space Tool
- Tests Package
- Official Document
- Showcase Website
- Evaluation Results
- DAMON-based System Optimization Guide
- Profile-Guided Optimization Example
- Publications and Presentations
Below are only a short list of recent news. For complete list of the news, please refer to a dedecated post.
2024-02-20: DAMO v2.2.2 is released with a new command,
will hopefully help reproducing the real-world memory access pattern for
analysis and benchmarks.
2024-02-14: DAMON talk for OSSummit NA 2024 has been accepted and scheduled
2024-02-09: DAMON in Amazon Linux 5.10.y kernel has updated to that of v6.7 Linux kernel. Major updates on this change include
- DAMOS apply target regions tracing and
- Sampling interval granularity monitoring results generation and DAMOS.
You can start using DAMON by
- installing DAMON-enabled kernel and its user-space tool,
- following the tutorial of the user-space tool, and
- run the automated tests suite.
By following those, you will be able to know if DAMON works on your machine and how you can use it.
DAMON is implemented in the Linux kernel, so you should install DAMON-enabled Linux kernel to use it. To check if you’re already running DAMON-enabled kernel, you could:
$ if grep CONFIG_DAMON /boot/config-$(uname -r); then echo "installed"; fi
DAMON is enabled in Amazon Linux kernels that based on v5.4 or newer Linux kernels. If you’re using Amazon Linux 2 and not using DAMON-enabled kernel on the distro, you can install DAMON-enabled kernels using its package manager, amazon-linux-extras. For example, if you want to use DAMON with Amazon Linux v5.10 kernel, below commands will work.
$ sudo yum install -y amazon-linux-extras
$ sudo amazon-linux-extras install kernel-5.10
$ sudo reboot
Note that AL2 v5.10.y contains all DAMON features that available on mainline while AL2 5.4.y contains only a part of DAMON features. AL2 v5.10.y kernels are recommended for the reason.
If your package system doesn’t support DAMON-enabled kernel, you can fetch a DAMON-merged Linux kernel source tree (it has merged in Linux mainline tree since v5.15), build, and install it. Note that you should enable kernel configuration options for DAMON, depending on your demands. For example:
$ cd $THE_FETCHED_DAMON_KERNEL_SOURCE_TREE
$ make olddefconfig
$ echo 'CONFIG_DAMON=y' >> /.config
$ echo 'CONFIG_DAMON_VADDR=y' >> /.config
$ echo 'CONFIG_DAMON_PADDR=y' >> /.config
$ echo 'CONFIG_DAMON_SYSFS=y' >> /.config
$ echo 'CONFIG_DAMON_DBGFS=y' >> /.config
$ echo 'CONFIG_DAMON_RECLAIM=y' >> /.config
$ echo 'CONFIG_DAMON_LRU_SORT=y' >> /.config
$ make -j$(nproc)
$ sudo make modules_install install
There are several Linux kernel source trees having DAMON for different users. You may pick one among those based on your needs.
For users who want a stable version of DAMON, below trees are recommended.
- Linus Torvalds’ mainline tree contains the latest stable version of DAMON since v5.15.
- Amazon Linux kernel v5.10.y tree, v5.4.y tree and all trees that based on v5.15 or newer Linux kernel contain stable versions of DAMON.
Note that AL2 v5.10.y contains all DAMON features that available on the latest major releases of Linux while AL2 5.4.y contains only a part of DAMON features, so AL2 v5.10.y is more recommended.
If you have interests in DAMON features under development, below trees will be appropriate. These trees contain latest version of DAMON which having features under development.
- mm-unstable contains the latest DAMON patches, which are under testing with other unstable memory management subsystem patches. So this tree is likely unstable and frequently updated, but would be a good baseline for your DAMON development.
- damon/next contains the latest changes, which might not tested at all. So this tree is likely more unstable and frequently updated than mm-unstable. This tree also contains some changes that exist only for DAMON hacks itself rather than eventually be merged in the mainline.
For people who have interest in DAMON features under development but use LTS kernels as their baseline, three were trees that based on three latest LTS kernels.
NOTE: Below trees were continuously getting DAMON backports on latest 5.4.y, 5.10.y, and 5.15.y. But those are deprecated as of 2022-09-03, and therefore there will be no update to the trees.
The source code of DAPTRACE, which is a prototype of DAMON, is also available.
There is a tests suite for correctness verification and performance evaluation of DAMON. Those are actively used for the development of DAMON. Using that, you can test DAMON on your machine with only a few simple commands or deeply understand the user interface of DAMON.
So, if you finished the tutorial but have no idea about what to do next, running the tests would be a good start. If any test fails, please report that to the maintainer via mail (email@example.com or firstname.lastname@example.org) or github.
The official document of DAMON in the latest mainline for users/sysadmins and kernel programmers are available. Those for next major release are also available(users/sysadmins doc, kernel programmers doc).
There is a showcase website that you can get more information of DAMON. The site hosts
- the documentation of DAMON that contains contents on damon/next tree,
- the heatmap format dynamic access pattern of various realistic workloads for heap area, mmap()-ed area, and stack area,
- the dynamic working set size distribution and chronological working set size changes, and
- the latest performance test results.
Evaluation of DAMON and DAMON-based system optimizations are available.
DAMON-based System Optimization Guide
A guide for DAMON-based system optimizations are also available.
Profile-Guided Optimization Example
An example of DAMON-based profile-guided optimization is also available.
DAMON is maintained and developed by its own community, which is a sub-set of the linux kernel development community.
The community is mainly driven by the mailing list (email@example.com). All the patches are posted there and reviewed. Almost every discussions are also made there. You can subscribe to the mailing list via subspace.kernel.org following the instruction. You could also read the mails without subscription via lore.
The community also have an open, regular, and informal virtual bi-weekly meeting series for DAMON community called DAMON Beer/Coffee/Tea chat series.
Publications and Presentations
Below are featured and/or upcoming publications and presentations covering DAMON. For more complete list of those, please refer to another dedicated post.
- SeongJae Park, DAMO[N,S]?: Implementing Self-Driven Data Access-Aware Efficient Linux System. In Open Source Summit North America, Apr 2024. Link
- SeongJae Park, DAMON: Current Status and Future Plans. In Kernel Summit, Nov 2023. Slides, Video, Link
- SeongJae Park, Data Access Monitoring Operator (DAMO): User-Space Tool/Python Library for Access-Aware Profiling and Optimization of Your Linux Systems. In Open Source Summit Europe, Sep 2023. Slides, Video, Link
- SeongJae Park, DAMON, DAMOS, and DAMO: Kernel Subsystems and User-Space Tools for Data Access-Aware System Analysis/Optimizations. In Open Source Summit North America, May 2023. Slides, Video, Link
- SeongJae Park, DAMON updates and future plans. In Linux Storage | Filesystem | MM & BPF Summit, May 2023. Slides, Video, Link
- SeongJae Park, Current Status and Future Plans of DAMON. In The Linux Kernel Summit, September 2022. Slides, Video, Link
- Jonathan Corbet, LRU-list manipulation with DAMON. In Linux Weekly News, August 2022. Article
- SeongJae Park, Madhuparna Bhowmik, Alexandru Uta, DAOS: Data Access-aware Operating System. In The 31st International ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC'22), June 2022. Paper, Slides, Poster
- Jonathan Corbet, Using DAMON for proactive reclaim. In Linux Weekly News, July 2021. Article
- Jonathan Corbet, Memory-management optimization with DAMON. In Linux Weekly News, February 2020. Article
- SeongJae Park, Yunjae Lee, Heon Y. Yeom, Profiling Dynamic Data Access Patterns with Controlled Overhead and Quality. In 20th ACM/IFIP International Middleware Conference Industry, December 2019. Paper