DAMON: Data Access Monitor

Modern workloads tend to have huge working sets and low locality. Despite this trend, the capacity of DRAM has not been increased enough to accommodate such huge working sets. Therefore, memory management mechanisms optimized for such modern workloads are widely required today. For such optimizations, knowing the data access pattern of given workloads is essential. However, manually extracting such patterns from huge and complex workloads is exhaustive. Worse yet, existing memory access analysis tools incur unacceptably high overheads for unnecessarily detailed analysis results.

DAMON is a data access monitoring framework subsystem for the Linux kernel. It mitigates the problems with its core mechanisms (refer to Design for the detail), which make it

  • accurate (the monitoring output is useful enough for DRAM level memory management; It might not appropriate for CPU Cache levels, though),
  • light-weight (the monitoring overhead is low enough to be applied online), and
  • scalable (the upper-bound of the overhead is in constant range regardless of the size of target workloads).

Using this framework, therefore, the kernel’s memory management mechanisms can make advanced decisions. Experimental memory management optimization works that incurring high data accesses monitoring overhead could implemented again. In user space, meanwhile, users who have some special workloads can write personalized applications for better understanding and optimizations of their workloads and systems.

Demo Video

damo monitor for water_nsquared

Demo Screenshot

masim stairs heatmap in ascii

Recent News

2022-09-15: The video for my kernel summit DAMON talk this year is now available at Youtube

2022-09-09: The plan for the first in-person DAMON community meetup at LPC and the in-person office hour at OSSummit EU has announced

2022-09-06: AL2 5.10 kernel’s DAMON code has updated to that of v5.19

2022-08-30: AL2 5.10 kernel’s DAMON code has updated to that of v5.18

2022-08-22: LWN introduced DAMON-based LRU-lists manipulation (DAMON_LRU_SORT) in detail

2022-08-12: Bi-weekly DAMON Beer/Coffee/Tea Chat series for open, regular, and informal community syncups and discussions has announced.

Getting Started

You can start using DAMON by

By following those, you will be able to know if DAMON works on your machine and how you can use it.

Install

DAMON is implemented in the Linux kernel, so you should install DAMON-enabled Linux kernel to use it. To check if you’re already running DAMON-enabled kernel, you could:

$ grep CONFIG_DAMON /boot/config-$(uname -r)

If you are an Amazon Linux 2 user, you can install a DAMON-enabled Linux kernels which are based on v5.10 or v5.4 using amazon-linux-extras like below:

$ sudo yum install -y amazon-linux-extras
$ sudo amazon-linux-extras install kernel-<5.10 or 5.4>
$ sudo reboot

Note that AL2 v5.4.y contains only a part of DAMON features, so AL2 v5.10.y is more recommended.

If your package system doesn’t support DAMON-enabled kernel, you can fetch a DAMON-merged Linux kernel source tree, build, and install it. Note that you should enable kernel configuration options for DAMON, depending on your demands. For example:

$ cd $THE_FETCHED_DAMON_KERNEL_SOURCE_TREE
$ make olddefconfig
$ echo 'CONFIG_DAMON=y' >> /.config
$ echo 'CONFIG_DAMON_KUNIT_TEST=y' >> /.config
$ echo 'CONFIG_DAMON_VADDR=y' >> /.config
$ echo 'CONFIG_DAMON_PADDR=y' >> /.config
$ echo 'CONFIG_DAMON_VADDR_KUNIT_TEST=y' >> /.config
$ echo 'CONFIG_DAMON_SYSFS=y' >> /.config
$ echo 'CONFIG_DAMON_DBGFS=y' >> /.config
$ echo 'CONFIG_DAMON_DBGFS_KUNIT_TEST=y' >> /.config
$ echo 'CONFIG_DAMON_RECLAIM=y' >> /.config
$ echo 'CONFIG_DAMON_LRU_SORT=y' >> /.config
$ make -j$(nproc)
$ sudo make modules_install install

Source Code

There are several DAMON-merged Linux kernel source trees for different users. You may pick one among those based on your needs.

For users who want only a stable version of DAMON, below trees are available.

  • Linus Torvalds’ mainline tree contains the latest stable version of DAMON since v5.15.
  • Amazon Linux 2 v5.10.y tree and v5.4.y tree also contains stable versions of DAMON in the mainline, which back-ported on v5.10.y and v5.4.y, respectively. Note that AL2 v5.4.y contains only a part of DAMON features, so AL2 v5.10.y is more recommended.

If you have interests in DAMON features under development, below tree will be appropriate. This trees contain latest version of DAMON which having features under development/hacks.

  • damon/next contains the latest changes, which might not fully be tested. So this tree is likely unstable and frequently updated. This tree also contains some changes that exist only for DAMON hacks itself rather than eventually be merged in the mainline.

For people who have interest in DAMON features under development but use LTS kernels as their baseline, there were trees that based on three latest LTS kernels.

NOTE: these trees are deprecated as of 2022-09-03, and therefore there will be no update at all.

The source code of DAPTRACE, which is a prototype of DAMON, is also available.

User-space Tool

A user-space tool for DAMON, which is called DAMO is available at Github and PyPi. You may start using DAMON by following the Getting Started of the tool for start.

Tests Package

There is an automated tests suite for correctness verification and performance evaluation of DAMON. Those are actively used for the development of DAMON. Using that, you can test DAMON on your machine with only a few simple commands or deeply understand the user interface of DAMON.

So, if you finished the tutorial but have no idea about what to do next, running the tests would be a good start. If any test fails, please report that to the maintainer via mail (sj@kernel.org) or github.

Official Document

The official document of DAMON in the latest mainline for users/sysadmins and kernel programmers are available. That for DAMON in the damon/next tree, which focused on DAMON only and contains some more DAMON-specific contents, is also available.

Showcase Website

There is a showcase web site that you can get more information of DAMON. The site hosts

Community

DAMON is maintained and developed by its own community, which is a sub-set of the linux kernel development community.

The community is mainly driven by the mailing list (damon@lists.linux.dev). All the patches are posted there and reviewed. Almost every discussions are also made there. You can subscribe to the mailing list via subspace.kernel.org following the instruction, and read the mails via lore.

The community also have an open, regular, and informal virtual bi-weekly meeting series for DAMON community called DAMON Beer/Coffee/Tea chat series.

Publications and Presentations

  • SeongJae Park, Current Status and Future Plans of DAMON. In The Linux Kernel Summit, September 2022. Slides, Video, Link
  • Jonathan Corbet, LRU-list manipulation with DAMON. In Linux Weekly News, August 2022. Article
  • SeongJae Park, Current Status, Future Plans, and Possible Collaborations for DAMON. In The Linux Kernel Summit, September 2022. Link
  • SeongJae Park, Madhuparna Bhowmik, Alexandru Uta, DAOS: Data Access-aware Operating System. In The 31st International ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC'22), June 2022. Paper, Slides, Poster
  • SeongJae Park, Writing a fine-grained access pattern oriented lightweight kernel module using DAMON/DAMOS in 10 minutes. In The Linux Kernel Summit, September 2021. Slides, Video, Link
  • Jonathan Corbet, Using DAMON for proactive reclaim. In Linux Weekly News, July 2021. Article
  • SeongJae Park, DAMON: Data Access Monitoring Framework for Fun and Memory Management Optimizations, In The Linux Kernel Summit, August 2020. Slides, Video, Link
  • Yunjae Lee, Yunhee Kim, and Heon. Y. Yeom, Lightweight Memory Tracing for Hot Data Identification, In Cluster computing, 2020. Paper
  • Jonathan Corbet, Memory-management optimization with DAMON. In Linux Weekly News, February 2020. Article
  • SeongJae Park, Yunjae Lee, Heon Y. Yeom, Profiling Dynamic Data Access Patterns with Controlled Overhead and Quality. In 20th ACM/IFIP International Middleware Conference Industry, December 2019. Paper
  • SeongJae Park, Tracing Data Access Pattern with Bounded Overhead and Best-effort Accuracy. In The Linux Kernel Summit, September 2019. Slides, Link
  • SeongJae Park, Yunjae Lee, Yunhee Kim, Heon Y. Yeom, Profiling Dynamic Data Access Patterns with Bounded Overhead and Accuracy. In IEEE International Workshop on Foundations and Applications of Self-* Systems (FAS* 2019), June 2019. Paper

News

2022-09-15: The video for my kernel summit DAMON talk this year is now available at Youtube

2022-09-09: The plan for the first in-person DAMON community meetup at LPC and the in-person office hour at OSSummit EU has announced

2022-09-06: AL2 5.10 kernel’s DAMON code has updated to that of v5.19

2022-08-30: AL2 5.10 kernel’s DAMON code has updated to that of v5.18

2022-08-22: LWN introduced DAMON-based LRU-lists manipulation (DAMON_LRU_SORT) in detail

2022-08-15: LWN introduced DAMON’s new features including ‘LRU_SORT’ as significant changes for Linux 6.0

2022-08-12: Bi-weekly DAMON Beer/Coffee/Tea Chat series for open, regular, and informal community syncups and discussions has announced.

2022-07-29: Current status, future plans, and possible collaborations for DAMON will be presented at the Kernel Summit 2022.

2022-06-26: The poster of the DAOS paper is available online.

2022-06-13: DAMON-based LRU-lists sorting patchset has posted and immediately merged in the -mm tree

2022-05-04: A paper introducing DAMON and related works have accepted by HPDC22

2022-05-03: Now DAMON has its own open mailing list

2022-04-29: Patches for DAMON online tuning have merged in -mm tree

2022-04-27: Android has backported and enabled building DAMON and DAMON_RECLAIM for the common kernel.

2022-04-27: Alibaba has shared thier own DAMON user space tool.

2022-02-28: The DAMON sysfs interface patchset has merged in -mm tree.

2022-02-17: An RFC patchset for sysfs-based DAMON’s new user interface has posted.

2022-01-20: A roadmap of DAMON has shared.

2022-01-09: Linux 5.16 is released. “DAMON-based proactive memory reclamation, operation schemes and physical memory monitoring” are marked as prominent features of the release by the Kernel newbies and LWN.

2021-12-23: A great blog post for DAMON-enabled kernel has uploaded

2021-11-07: DAMON patches for automated memory management optimization, the physical address space monitoring support, and proactive reclamation have merged in the mainline.

2021-11-01: DAMON has released with Linux v5.15.

2021-10-14: DAMON_RECLAIM patchset is merged in the Amazon Linux 5.10.y kernel tree.

2021-10-02: DAMOS patchset is merged in the -mm tree.

2021-09-23: DAMON and DAMOS are presented in the kernel summit. Slides, Video, Link

2021-09-16: DAMON development tree on kernel.org is created.

2021-09-08: DAMON patchset is merged in the Linus Torvalds’ tree, aka mainline

2021-09-07: DAMON/DAMOS will be presented at the Kernel Summit 2021

2021-08-31: DAMON user-space tool is uploaded to the official Python packages system, PyPi

2021-08-06: DAMON patchset is merged in the -mm tree

2021-07-27: LWN published a second article introducing DAMON patchset series

2021-06-11: DAMON-based proactive reclamation RFC patchset has shared on the hackernews and introduced by a Phoronix article

2021-05-31: DAMON-based proactive reclamation RFC patchset has posted

2021-05-26: DAMON-enabled Amazon Linux 2 kernels have deployed to all users

2021-05-07: DAMON has merged in the public source tree for Amazon Linux v5.4.y kernel

2021-04-05: damo now supports heatmap visualization on the terminal

2021-03-31: DAMON user-space tool (damo) is released as an individual open source project

2021-03-19: DAMON has merged in the public source tree for Amazon Linux v5.10.y kernel

2021-03-04: DAMON supports for two latest LTS kernels announced

2021-03-03: DAMON is merged in v5.10 based public Amazon Linux kernel tree

2021-02-25: An example usage of DAMON for profiling is shared

2021-01-07: A runtime system for DAMON-based optimal operation scheme finding is released

2020-12-03: Further plans around DAMON is shared

2020-11-17: A real-world user story of DAMON is shared

2020-09-26: The tests package for DAMON is released under GPL v2 license

2020-08-19: A demo video is available

2020-08-05: DAMON will be presented at the Kernel Summit 2020

2020-06-04: Physical Memory Monitoring is now available

2020-05-18: DAMON showcase website is announced

2020-05-13: DAMON official document is uploaded online

2020-02-20: DAMON has introduced by an LWN article

2020-02-10: The first RFC of Data Access Monitoring-based Operating Schemes (DAMOS) has posted to LKML

2020-01-23: The RFC of DAMON has introduced by LWN’s ‘Kernel patches of interest’

2020-01-20: The first RFC patchset of DAMON has posted to LKML

Avatar
SeongJae Park
Kernel / Hypervisor Engineer

SeongJae Park is a programmer who loves to analyze and develop systems.

Related