When Zeek extracts a file it observes being transfered in network traffic, the
file-monitor container picks up those extracted files and publishes to a ZeroMQ topic that can be subscribed to by any other process that wants to analyze that extracted file. In Malcolm at the time of this writing (as of the v5.0.0 release), currently implemented file scanners include ClamAV, YARA, capa and VirusTotal, all of which are managed by the
file-monitor container. The scripts involved in this code are:
zeek_carve_watcher.py’s topic and performs file scanning for the ClamAV, YARA, capa and VirusTotal engines and sends “hits” to another ZeroMQ sync on port 5988
zeek_carve_scanner.py’s topic and logs hits to a “fake” Zeek signatures.log file which is parsed and ingested by Logstash
Additional file scanners could either be added to the
file-monitor service, or to avoid coupling with Malcolm’s code you could simply define a new service as instructed in the Adding a new service section and write your own scripts to subscribe and publish to the topics as described above. While that might be a bit of hand-waving, these general steps take care of the plumbing around extracting the file and notifying your tool, as well as handling the logging of “hits”: you shouldn’t have to really edit any existing code to add a new carved file scanner.
EXTRACTED_FILE_PIPELINE_DEBUG_EXTRA environment variables in the
docker-compose files can be set to
true to enable verbose debug logging from the output of the Docker containers involved in the carved file processing pipeline.