A powerful, easily deployable network traffic analysis tool suite
When a PCAP is uploaded (either through Malcolm’s upload web interface or just copied manually into the ./pcap/upload/
directory), the pcap-monitor
container has a script that picks up those PCAP files and publishes to a ZeroMQ topic that can be subscribed to by any other process that wants to analyze that PCAP. In Malcolm at the time of this writing (as of the v5.0.0 release), there are two of those: the zeek
container and the arkime
container. In Malcolm, they actually both share the same script to read from that topic and run the PCAP through Zeek and Arkime, respectively. If you’re looking for an example to follow, the zeek
container is the less complicated of the two. So, if you were looking to integrate a new PCAP processing tool into Malcolm (named cooltool
for this example), the process would be something like:
zeek
and arkime
services use bind mounts to access the local ./pcap
directoryzeek
and arkime
use, if you like) which subscribes to the PCAP topic port (30441
as defined in pcap_utils.py) and handles the PCAP files published there, each PCAP file represented by a JSON dictionary with name
, tags
, size
, type
and mime
keys (search for FILE_INFO_
in pcap_utils.py). This script should be added to and run by your cooltool.Dockerfile
-generated container.While that might be a bit of hand-waving, these general steps take care of the PCAP processing piece: you shouldn’t have to really edit any existing code to add a new PCAP processor. You’re just creating a new container for the Malcolm appliance to the ZeroMQ topic and handle the PCAPs your tool receives.
The PCAP_PIPELINE_DEBUG
and PCAP_PIPELINE_DEBUG_EXTRA
environment variables in the docker-compose
files can be set to true
to enable verbose debug logging from the output of the Docker containers involved in the PCAP processing pipeline.