Recently, I’ve helped customer to migrate their service from single VM to 1 vm per-service. I am able to manage minimum downtime when migrating the web data and database data to each VM. All VM using latest OS and latest stack version in the repository’s.

All new VM and the dedicated service is working as expected. Then we found increase of throughput in the front-end. Further investigation shows that there a bottleneck in from the database vm, where we found an “Out of Memory”. From the log and monitoring statistic, i found that the memory usage increase dramatically once the mysql have high access but not releasing the memory until the mysqld service OOM. Even though on old environment, there is no such issue with the same configuration. By the way, old database use MariaDB 10.2 and the new one is using MariaDB 10.4.12

I have tried to tune the configuration to reduce the memory footprint, reduce the connection also tune the system kernel. But the same issue keep happening.

After searching on the web about this unexpected behavior, i got some pople that have the same issue with but there is no specific solution.
From the bug report Old Mariadb, 10.2-3 there is a memory leak issue with “Unresolved” status. Here’s the link.

Further reading result, this condition may caused by the memory allocation handler. Using this keyword i manage to found someone that have the same issue and able to resolved it by changing the memory allocation library of their MariaDB server. Please use the link to find the related infromation here.

Based on the server status, i found that the MariaDB 10.4 still using system default for their memory allocation.

MariaDB [(none)]> SHOW VARIABLES LIKE 'version_malloc_library';
+------------------------+--------+
| Variable_name          | Value  |
+------------------------+--------+
| version_malloc_library | system |
+------------------------+--------+
1 row in set (0.00 sec)

Then i am install the jmalloc library to the system so we can use it as memory allocation library on MariaDB 10.4

apt-get update apt-get 
install -y libjemalloc-dev

Then update the env configuration to use new jmalloc library for MariaDB.

cd /etc/systemd/system/mariadb.service.d/
Edit any .conf in this directory by adding information at the end-line.
Environment="LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1"
systemctl daemon-reload
systemctl stop mariadb && systemctl start mariadb

Validate the new memory allocation library.

MariaDB [(none)]> SHOW VARIABLES LIKE 'version_malloc_library';
+------------------------+-------------------+
 | Variable_name          | Value             |
+------------------------+-------------------+
 | version_malloc_library | jemalloc 3.6.0-11 |
+------------------------+-------------------+
1 row in set (0.001 sec)

Thank You.

Previous ArticleNext Article

Leave a Reply

Your email address will not be published. Required fields are marked *

R.

Replacing Failde Raid disk Proxmox ZFS

This is the procedure on how to replace broken disk from zfs raid array. In this case we simulate to replace zfs raid-1.

  • Check zfs pool status
    #zfs status -v
  • Get the disk information and replaced it (Let’s say that the failed disk is /dev/sdb)
  • Clone the disk partition table from the health disk to replacement
    #sgdisk -R /dev/sdb /dev/sda [newdisk - existing disk]
  • Regenerate the partition UID
    #sgdisk -G /dev/sdb
  • Replacing the disk
    #zpool replace rpool /dev/sdb2 /dev/sdb2
  • Monitoring resilvering process, the
    #zpool status -v
  • For the disk where boot partition resided, make sure to update the grub configuration to mark the replaced disk
    #dpkg-reconfigure grup-pc
    #Make sure to check the new disk.

Thank You

L.

Linux backup using Rsync

This time is about using rsync to copy a Linux/Unix system to another host or backing it up. By using rsync, its more effective than using tool like dd since we can efficiently select the directory, attribute that we we want to backup or move.

Sometime ignorance is a blessing

The one who not so wise

rsync also can be use when the system is running. But you may need to do it with caution and make sure you understand the data state. When rsync with the system running, the data that have not commit the changes in files level may not transfer.

This method is work for migrating the the system to another hosts. Please be aware, that the target is the fresh installed operating system with the same version of the source.

Rsync full backup

# rsync -aAXHv --numeric-ids --info=progress2 --exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} / /path/to/backup

Rsync Clone

rsync -qaHAXS [SOURCE_DIR] [DESTINATION_DIR]
  • –numeric-ids, will disable mapping of user and group names; instead, numeric group and user IDs will be transfered. This is useful when backing up over SSH or when using a live system to backup different system disk.
  • –info=progress2,  will show the overall progress info and transfer speed instead of the list of files being transferred.
  • avoid crossing a filesystem boundary when recursing, add the option -x/--one-file-system. This will prevent backing up any mount point in the hierarchy.
  • -n or known as the option for the dry-run. To simulate the file transfers