To continue with the example of the
cooltool service added in the PCAP processors section, assuming that
cooltool generates some textual log files to be parsed and indexed into Malcolm.
Users will have have configured
cooltool in the
cooltool.Dockerfile and its section in the
docker-compose files to write logs into a subdirectory or subdirectories in a shared folder - bind mounted in such a way that both the
filebeat containers can access. Referring to the
zeek container as an example, this is how the
./zeek-logs folder is handled; both the
zeek services have
./zeek-logs in their
$ grep -P "^( - ./zeek-logs| [\w-]+:)" docker-compose.yml | grep -B1 "zeek-logs" filebeat: - ./zeek-logs:/data/zeek -- zeek: - ./zeek-logs/upload:/zeek/upload …
Access to the
cooltool logs must be provided in a similar fashion.
filebeat.yml by adding a new log input path pointing to the
cooltool logs to send them along to the
logstash container. This modified
filebeat.yml will need to be reflected in the
filebeat container via bind mount or by rebuilding it.
filebeat) sends logs to 1..n parse pipelines
In order to add a new parse pipeline for
cooltool after tweaking
filebeat.yml as described above, create a
cooltool directory under
logstash/pipelines that follows the same pattern as the
zeek parse pipeline. This directory will have an input file (tiny), a filter file (possibly large), and an output file (tiny). In the filter file, be sure to set the field
event.hash to a unique value to identify indexed documents in OpenSearch; the fingerprint filter may be useful for this.
Finally, in the
./config/logstash.env file, set a new
LOGSTASH_PARSE_PIPELINE_ADDRESSES environment variable to
cooltool-parse,zeek-parse,suricata-parse,beats-parse (assuming the pipeline address from the previous step was named
cooltool-parse) so that logs sent from
logstash are forwarded to all parse pipelines.
The following modifications must be made in order for Malcolm to parse new Zeek log files:
idfour-tuple, timestamp, etc., use the same convention used by existing Zeek logs in that file (e.g.,
logstash/pipelines/zeek/12_zeek_normalize.conffor values such as action (
event.action), result (
event.result), application protocol version (
scripts/zeek_script_to_malcolm_boilerplate.py may help by autogenerating these filters.
Malcolm’s Logstash instance will do a lot of enrichments automatically: see the enrichment pipeline, including MAC address to vendor by OUI, GeoIP, ASN, and a few others. In order to take advantage of these enrichments that are already in place, normalize new fields to use the same standardized field names Malcolm uses for things such as IP addresses, MAC addresses, etc. Additional enrichments may be added by creating new
.conf files containing Logstash filters in the enrichment pipeline directory and using either of the techniques in the Local modifications section to implement those changes in the
The logstash.Dockerfile installs the Logstash plugins used by Malcolm (search for
logstash-plugin install in that file). Additional Logstash plugins could be installed by modifying this Dockerfile and rebuilding the
logstash Docker image.