A powerful, easily deployable network traffic analysis tool suite
Malcolm provides an instance of NetBox, an open-source “solution for modeling and documenting modern networks.” The NetBox web interface is available at at https://localhost/netbox/ if you are connecting locally.
The design of a potentially deeper integration between Malcolm and Netbox is a work in progress.
Please see the NetBox page on GitHub, its documentation and its public demo for more information.
As Zeek logs and Suricata alerts are parsed and enriched (if the LOGSTASH_NETBOX_ENRICHMENT
environment variable in docker-compose.yml
is set to true
) the NetBox API will be queried for the associated hosts’ information. If found, the information retrieved by NetBox will be used to enrich these logs through the creation of the following new fields. See the NetBox API documentation and the NetBox documentation.
destination.…
destination.device.cluster
(/virtualization/clusters/
) (for Virtual Machine device types)destination.device.device_type
(/dcim/device-types/
)destination.device.id
(/dcim/devices/{id}
)destination.device.manufacturer
(/dcim/manufacturers/
)destination.device.name
(/dcim/devices/
)destination.device.role
(/dcim/device-roles/
)destination.device.service
(/ipam/services/
)destination.device.site
(/dcim/sites/
)destination.device.url
(/dcim/devices/
)destination.device.details
(full JSON object, only with LOGSTASH_NETBOX_ENRICHMENT_VERBOSE: 'true'
)destination.segment.id
(/ipam/vrfs/{id}
)destination.segment.name
(/ipam/vrfs/
)destination.segment.site
(/dcim/sites/
)destination.segment.tenant
(/tenancy/tenants/
)destination.segment.url
(/ipam/vrfs/
)destination.segment.details
(full JSON object, only with LOGSTASH_NETBOX_ENRICHMENT_VERBOSE: 'true'
)source.…
same as destination.…
related
fields (the same approach used in ECS)
related.device_type
related.device_name
related.manufacturer
related.role
related.segment
related.service
related.site
For Malcolm’s purposes, both physical devices and virtualized hosts will be stored as described above: the device_type
field can be used to distinguish between them.
NetBox has the concept of sites. Sites can have overlapping IP address ranges, of course. The value of the NETBOX_DEFAULT_SITE
variable in environment variable in docker-compose.yml
will be used as a query parameter for these enrichment lookups.
This feature was implemented as described in idaholab/Malcolm#132.
As Malcolm cross-checks network traffic with NetBox’s model (as described above), the resulting enrichment data (or lack thereof) can highlight devices and services observed in network traffic for which there is no corresponding entry in the list of inventoried assets.
These uninventoried devices and services are highlighted in two dashboards:
known_
logs and software
logs to provide a summary of the known devices and services in the network. The Uninventoried Observed Services and Uninventoried Observed Hosts tables show services and hosts (by IP address) that weren’t found when searched via the NetBox API.This feature was implemented as described in idaholab/Malcolm#133.
See idaholab/Malcolm#134.
The purpose of an asset management system is to document the intended state of a network: were Malcolm to actively and agressively populate NetBox with the live network state, a network configuration fault could result in an incorrect documented configuration. The Malcolm development team is investigating what data, if any, should automatically flow to NetBox based on traffic observed (enabled via the NETBOX_CRON
environment variable in docker-compose.yml
), and what NetBox inventory data could be used, if any, to enrich Malcolm’s network traffic metadata. Well-considered suggestions in this area are welcome.
See idaholab/Malcolm#135.
See idaholab/Malcolm#136.
The NetBox database may be backed up and restored using ./scripts/netbox-backup
and ./scripts/netbox-restore
, respectively. While Malcolm is running, run the following command from within the Malcolm installation directory to backup the entire NetBox database:
$ ./scripts/netbox-backup
NetBox configuration database saved to ('malcolm_netbox_backup_20230110-133855.gz', 'malcolm_netbox_backup_20230110-133855.media.tar.gz')
To clear the existing NetBox database and restore a previous backup, run the following command (substituting the filename of the netbox_….gz
you wish to restore) from within the Malcolm installation directory while Malcolm is running:
./scripts/netbox-restore --netbox-restore ./malcolm_netbox_backup_20230110-125756.gz
Note that some of the data in the NetBox database is cryptographically signed with the value of the SECRET_KEY
environment variable in the ./netbox/env/netbox.env
environment file. A restored NetBox backup will not work if this value is different from when it was created.