A powerful, easily deployable network traffic analysis tool suite for network security monitoring
Carved file scanners
Similar to the PCAP processing pipeline described above, new tools can plug into Malcolm’s automatic file extraction and scanning to examine file transfers carved from network traffic.
When Zeek extracts a file it observes being transfered in network traffic, the file-monitor container picks up those extracted files and publishes to a ZeroMQ topic that can be subscribed to by any other process that wants to analyze that extracted file. In Malcolm at the time of this writing (as of the v5.0.0 release), currently implemented file scanners include ClamAV, YARA, capa and VirusTotal, all of which are managed by the file-monitor container. The scripts involved in this code are:
- shared/bin/zeek_carve_watcher.py - watches the directory to which Zeek extracts files and publishes information about those files to the ZeroMQ ventilator on port 5987
- shared/bin/zeek_carve_scanner.py - subscribes to
zeek_carve_watcher.py’s topic and performs file scanning for the ClamAV, YARA, capa and VirusTotal engines and sends “hits” to another ZeroMQ sync on port 5988 - shared/bin/zeek_carve_logger.py - subscribes to
zeek_carve_scanner.py’s topic and logs hits to a “fake” Zeek signatures.log file that is parsed and ingested by Logstash - shared/bin/zeek_carve_utils.py - various variables and classes related to carved file scanning
Additional file scanners could either be added to the file-monitor service; or to avoid coupling with Malcolm’s code users could simply define a new service as instructed in the Adding a new service section and write custom scripts to subscribe and publish to the topics as described above. While this might be a bit of hand-waving, these general steps take care of the plumbing around extracting the file and notifying a new tool, as well as handling the logging of “hits”: users shouldn’t have to really edit any existing code to add a new carved file scanner.
The EXTRACTED_FILE_PIPELINE_VERBOSITY environment variables in can be set to -v, -vv, etc., to increase the verbosity of debug logging from the output of the containers involved in the carved file processing pipeline.