OpenSearch index management

Malcolm releases prior to v6.2.0 used environment variables to configure OpenSearch Index State Management policies.

Since then, OpenSearch Dashboards has developed and released plugins with UIs for Index State Management and Snapshot Management. Because these plugins provide a more comprehensive and user-friendly interface for these features, the old environment variable-based configuration code has been removed from Malcolm; with the exception of the code that uses the OPENSEARCH_INDEX_SIZE_PRUNE_LIMIT and OPENSEARCH_INDEX_SIZE_PRUNE_NAME_SORT variables in dashboards-helper.env, which deals with deleting the oldest network session metadata indices when the database exceeds a certain size.

Note that OpenSearch index state management and snapshot management only deals with disk space consumed by OpenSearch indices: it does not have anything to do with PCAP file storage. The MANAGE_PCAP_FILES environment variable in the arkime.env file can be used to allow Arkime to prune old PCAP files based on available disk space.

Using ILM/ISM with Arkime

Arkime allows setting index management policies with its sessions and history indices. The Malcolm environment variables for configuring this behavior are set in arkime.env. These variables can be used for both OpenSearch and Elasticsearch instances (OpenSearch Index State Management (ISM) and Elasticsearch Index Lifecycle Management (ILM), respectively).