DAMON: Data Access Monitor

Modern workloads tend to have huge working sets and low locality. Despite this trend, the capacity of DRAM has not been increased enough to accommodate such huge working sets. Therefore, memory management mechanisms optimized for such modern workloads are widely required today. For such optimizations, knowing the data access pattern of given workloads is essential. However, manually extracting such patterns from huge and complex workloads is exhaustive. Worse yet, existing memory access analysis tools incur unacceptably high overheads for unnecessarily detailed analysis results.

DAMON is a data access monitoring framework subsystem for the Linux kernel. It mitigates the problems with its core mechanisms (refer to Design for the detail), which make it

  • accurate (the monitoring output is useful enough for DRAM level memory management; It might not appropriate for CPU Cache levels, though),
  • light-weight (the monitoring overhead is low enough to be applied online), and
  • scalable (the upper-bound of the overhead is in constant range regardless of the size of target workloads).

Using this framework, therefore, the kernel’s memory management mechanisms can make advanced decisions. Experimental memory management optimization works that incurring high data accesses monitoring overhead could implemented again. In user space, meanwhile, users who have some special workloads can write personalized applications for better understanding and optimizations of their workloads and systems.

Demo Video

damo monitor for water_nsquared

Demo Screenshot

masim stairs heatmap in ascii

Recent News

2022-05-04: A paper introducing DAMON and related works have accepted by HPDC22

2022-05-03: Now DAMON has its own open mailing list

2022-04-29: Patches for DAMON online tuning have merged in -mm tree

2022-04-27: Android has backported and enabled building DAMON and DAMON_RECLAIM for the common kernel.

2022-04-27: Alibaba has shared thier own DAMON user space tool.

Getting Started

You can start using DAMON by

By following those, you will be able to check if DAMON works on your machine and know how you can use it.

Install

DAMON is implemented in the Linux kernel, so you should install DAMON-enabled Linux kernel to use it. To check if you’re already running DAMON-enabled kernel, you could:

$ grep CONFIG_DAMON /boot/config-$(uname -r)

If you are an Amazon Linux 2 user, you can install a DAMON-enabled Linux kernel which based on v5.4 or v5.10 using amazon-linux-extras like below:

$ sudo yum install -y amazon-linux-extras
$ sudo amazon-linux-extras install kernel-<5.4 or 5.10>
$ sudo reboot

If your package system doesn’t have DAMON-enabled kernel in the list, you can fetch a DAMON-merged Linux kernel source tree, build, and install it. Note that you should enable kernel configuration options for DAMON, depending on your demands. For example:

$ cd $THE_FETCHED_DAMON_KERNEL_SOURCE_TREE
$ make olddefconfig
$ echo 'CONFIG_DAMON=y' >> /.config
$ echo 'CONFIG_DAMON_KUNIT_TEST=y' >> /.config
$ echo 'CONFIG_DAMON_VADDR=y' >> /.config
$ echo 'CONFIG_DAMON_PADDR=y' >> /.config
$ echo 'CONFIG_DAMON_VADDR_KUNIT_TEST=y' >> /.config
$ echo 'CONFIG_DAMON_SYSFS=y' >> /.config
$ echo 'CONFIG_DAMON_DBGFS=y' >> /.config
$ echo 'CONFIG_DAMON_DBGFS_KUNIT_TEST=y' >> /.config
$ echo 'CONFIG_DAMON_RECLAIM=y' >> /.config
$ make -j$(nproc)
$ sudo make modules_install install

Source Code

There are several DAMON-merged Linux kernel source trees for different users. You may pick one among those based on your needs.

For users who want only a stable version of DAMON, below trees are available.

  • Linus Torvalds’ mainline tree contains the latest stable version of DAMON.
  • Amazon Linux 2 v5.10.y tree and v5.4.y tree also contains stable versions of DAMON, which back-ported on v5.10.y and v5.4.y, respectively.

If you have interest in DAMON features under development, below trees will be appropriate. Those trees contain latest version of DAMON which having features under development.

  • damon/master contains the tree that relatively stable. This tree is not so frequently updated.
  • damon/next contains the latest changes, which might not even be tested. So this tree is likely unstable and frequently updated.

For people who have interest in DAMON features under development but use LTS kernels as their baseline, there were trees that based on three latest LTS kernels. NOTE: these trees are deprecated, because all initial DAMON features are merged in the Linus’ tree during the v5.16 merge window.

The source code of DAPTRACE, which is a prototype of DAMON, is also available.

User-space Tool

A user-space tool for DAMON, which is called DAMO is available. It’s also available at PyPi. You may start using DAMON by following the Getting Started of the tool for start.

Tests Package

There is an automated tests suite for correctness verification and performance evaluation of DAMON. Those are actively used for the development of DAMON. Using that, you can test DAMON on your machine with only a few simple commands or deeply understand the user interface of DAMON.

So, if you finished the tutorial but have no idea about what to do next, running the tests would be a good start. If any test fails, please report that to the maintainer via mail (sj@kernel.org) or github.

Official Document

The official document of DAMON in the latest mainline for users/sysadmins and kernel programmers are available. Those for DAMON in the development tree (damon/next) are also available.

Showcase Website

There is a showcase web site that you can get more information of DAMON. The site hosts

  • the official documentation of DAMON in its development tree,
  • the heatmap format dynamic access pattern of various realistic workloads for heap area, mmap()-ed area, and stack area,
  • the dynamic working set size distribution and chronological working set size changes, and
  • the latest performance test results.

Publications and Presentations

  • SeongJae Park, Writing a fine-grained access pattern oriented lightweight kernel module using DAMON/DAMOS in 10 minutes. In The Linux Kernel Summit, September 2021. Slides, Video, Link
  • Jonathan Corbet, Using DAMON for proactive reclaim. In Linux Weekly News, July 2021. Article
  • SeongJae Park, DAMON: Data Access Monitoring Framework for Fun and Memory Management Optimizations, In The Linux Kernel Summit, August 2020. Slides, Video, Link
  • Yunjae Lee, Yunhee Kim, and Heon. Y. Yeom, Lightweight Memory Tracing for Hot Data Identification, In Cluster computing, 2020. Paper
  • Jonathan Corbet, Memory-management optimization with DAMON. In Linux Weekly News, February 2020. Article
  • SeongJae Park, Yunjae Lee, Heon Y. Yeom, Profiling Dynamic Data Access Patterns with Controlled Overhead and Quality. In 20th ACM/IFIP International Middleware Conference Industry, December 2019. Paper
  • SeongJae Park, Tracing Data Access Pattern with Bounded Overhead and Best-effort Accuracy. In The Linux Kernel Summit, September 2019. Slides, Link
  • SeongJae Park, Yunjae Lee, Yunhee Kim, Heon Y. Yeom, Profiling Dynamic Data Access Patterns with Bounded Overhead and Accuracy. In IEEE International Workshop on Foundations and Applications of Self-* Systems (FAS* 2019), June 2019. Paper

News

2022-05-04: A paper introducing DAMON and related works have accepted by HPDC22

2022-05-03: Now DAMON has its own open mailing list

2022-04-29: Patches for DAMON online tuning have merged in -mm tree

2022-04-27: Android has backported and enabled building DAMON and DAMON_RECLAIM for the common kernel.

2022-04-27: Alibaba has shared thier own DAMON user space tool.

2022-02-28: The DAMON sysfs interface patchset has merged in -mm tree.

2022-02-17: An RFC patchset for sysfs-based DAMON’s new user interface has posted.

2022-01-20: A roadmap of DAMON has shared.

2022-01-09: Linux 5.16 is released. “DAMON-based proactive memory reclamation, operation schemes and physical memory monitoring” are marked as prominent features of the release by the Kernel newbies and LWN.

2021-12-23: A great blog post for DAMON-enabled kernel has uploaded

2021-11-07: DAMON patches for automated memory management optimization, the physical address space monitoring support, and proactive reclamation have merged in the mainline.

2021-11-01: DAMON has released with Linux v5.15.

2021-10-14: DAMON_RECLAIM patchset is merged in the Amazon Linux 5.10.y kernel tree.

2021-10-02: DAMOS patchset is merged in the -mm tree.

2021-09-23: DAMON and DAMOS are presented in the kernel summit. Slides, Video, Link

2021-09-16: DAMON development tree on kernel.org is created.

2021-09-08: DAMON patchset is merged in the Linus Torvalds’ tree, aka mainline

2021-09-07: DAMON/DAMOS will be presented at the Kernel Summit 2021

2021-08-31: DAMON user-space tool is uploaded to the official Python packages system, PyPi

2021-08-06: DAMON patchset is merged in the -mm tree

2021-07-27: LWN published a second article introducing DAMON patchset series

2021-06-11: DAMON-based proactive reclamation RFC patchset has shared on the hackernews and introduced by a Phoronix article

2021-05-31: DAMON-based proactive reclamation RFC patchset has posted

2021-05-26: DAMON-enabled Amazon Linux 2 kernels have deployed to all users

2021-05-07: DAMON has merged in the public source tree for Amazon Linux v5.4.y kernel

2021-04-05: damo now supports heatmap visualization on the terminal

2021-03-31: DAMON user-space tool (damo) is released as an individual open source project

2021-03-19: DAMON has merged in the public source tree for Amazon Linux v5.10.y kernel

2021-03-04: DAMON supports for two latest LTS kernels announced

2021-03-03: DAMON is merged in v5.10 based public Amazon Linux kernel tree

2021-02-25: An example usage of DAMON for profiling is shared

2021-01-07: A runtime system for DAMON-based optimal operation scheme finding is released

2020-12-03: Further plans around DAMON is shared

2020-11-17: A real-world user story of DAMON is shared

2020-09-26: The tests package for DAMON is released under GPL v2 license

2020-08-19: A demo video is available

2020-08-05: DAMON will be presented at the Kernel Summit 2020

2020-06-04: Physical Memory Monitoring is now available

2020-05-18: DAMON showcase website is announced

2020-05-13: DAMON official document is uploaded online

2020-02-20: DAMON has introduced by an LWN article

2020-02-10: The first RFC of Data Access Monitoring-based Operating Schemes (DAMOS) has posted to LKML

2020-01-23: The RFC of DAMON has introduced by LWN’s ‘Kernel patches of interest’

2020-01-20: The first RFC patchset of DAMON has posted to LKML

Avatar
SeongJae Park
Kernel / Hypervisor Engineer

SeongJae Park is a programmer who loves to analyze and develop systems.

Related