Linux host system configuration

Installing Docker

Docker installation instructions vary slightly by distribution. Please follow the links below to docker.com to find the instructions specific to your distribution:

After installing Docker, because Malcolm should be run as a non-root user, add your user to the docker group with something like:

$ sudo usermod -aG docker yourusername

Following this, either reboot or log out then log back in.

Docker starts automatically on DEB-based distributions. On RPM-based distributions, you need to start it manually or enable it using the appropriate systemctl or service command(s).

You can test docker by running docker info, or (assuming you have internet access), docker run --rm hello-world.

Installing docker-compose

Please follow this link on docker.com for instructions on installing docker-compose.

Operating system configuration

The host system (ie., the one running Docker) will need to be configured for the best possible OpenSearch performance. Here are a few suggestions for Linux hosts (these may vary from distribution to distribution):

Append the following lines to /etc/sysctl.conf:

# the maximum number of open file handles
fs.file-max=2097152

# increase maximums for inotify watches
fs.inotify.max_user_watches=131072
fs.inotify.max_queued_events=131072
fs.inotify.max_user_instances=512

# the maximum number of memory map areas a process may have
vm.max_map_count=262144

# decrease "swappiness" (swapping out runtime memory vs. dropping pages)
vm.swappiness=1

# the maximum number of incoming connections
net.core.somaxconn=65535

# the % of system memory fillable with "dirty" pages before flushing
vm.dirty_background_ratio=40

# maximum % of dirty system memory before committing everything
vm.dirty_ratio=80

Depending on your distribution, create either the file /etc/security/limits.d/limits.conf containing:

# the maximum number of open file handles
* soft nofile 65535
* hard nofile 65535
# do not limit the size of memory that can be locked
* soft memlock unlimited
* hard memlock unlimited

OR the file /etc/systemd/system.conf.d/limits.conf containing:

[Manager]
# the maximum number of open file handles
DefaultLimitNOFILE=65535:65535
# do not limit the size of memory that can be locked
DefaultLimitMEMLOCK=infinity

Change the readahead value for the disk where the OpenSearch data will be stored. There are a few ways to do this. For example, you could add this line to /etc/rc.local (replacing /dev/sda with your disk block descriptor):

# change disk read-adhead value (# of blocks)
blockdev --setra 512 /dev/sda

Change the I/O scheduler to deadline or noop. Again, this can be done in a variety of ways. The simplest is to add elevator=deadline to the arguments in GRUB_CMDLINE_LINUX in /etc/default/grub, then running sudo update-grub2
If you are planning on using very large data sets, consider formatting the drive containing the opensearch volume as XFS.

After making all of these changes, do a reboot for good measure!