Hedgehog Linux is a Debian-based operating system built to

• monitor network interfaces
• capture packets to PCAP files
• detect file transfers in network traffic and extract and scan those files for threats
• generate and forward Zeek logs, Arkime sessions and other information to Malcolm

Table of Contents

• Sensor installation
  • Image boot options
  • Installer
• Boot
  • Kiosk mode
• Configuration
  • Interfaces, hostname, and time synchronization
    • Hostname
    • Interfaces
    • Time synchronization
  • Capture, forwarding, and autostart services
    • Capture
      • Automatic file extraction and scanning
    • Forwarding
      • arkime-capture: Arkime session forwarding
      • filebeat: Zeek and Suricata log forwarding
      • miscbeat: System metrics forwarding
    • Autostart services
• Zeek Intelligence Framework
• Appendix A - Generating the ISO
• Appendix B - Configuring SSH access
• Appendix C - Troubleshooting
• Appendix D - Hardening
  • STIG compliance exceptions
  • CIS benchmark compliance exceptions
  • Hardening compliance issues - work in progress
• Appendix E - Upgrades
• Copyright

Sensor installation

Image boot options

The Hedgehog Linux installation image, when provided on an optical disc, USB thumb drive, or other removable medium, can be used to install or reinstall the sensor software.

The boot menu of the sensor installer image provides several options:

• Live system and Live system (fully in RAM) may also be used to run the sensor in a "live USB" mode without installing any software or making any persistent configuration changes on the sensor hardware.
• Install Hedgehog Linux and Install Hedgehog Linux (encrypted) are used to install the sensor onto the current system. Both selections install the same operating system and sensor software, the only difference being that the encrypted option encrypts the hard disks with a password (provided in a subsequent step during installation) that must be provided each time the sensor boots. There is some CPU overhead involved in an encrypted installation, so it is recommended that encrypted installations only be used for mobile installations (eg., on a sensor that may be shipped or carried for an incident response) and that the unencrypted option be used for fixed sensors in secure environments.
• Install Hedgehog Linux (advanced configuration) allows you to configure installation fully using all of the Debian installer settings and should only be selected for advanced users who know what they're doing.
• Rescue system is included for debugging and/or system recovery and should not be needed in most cases.

Installer

The sensor installer is designed to require as little user input as possible. For this reason, there are NO user prompts and confirmations about partitioning and reformatting hard disks for use by the sensor. The installer assumes that all non-removable storage media (eg., SSD, HDD, NVMe, etc.) are available for use and ⛔🆘😭💀 will partition and format them without warning 💀😭🆘⛔.

The installer will ask for a few pieces of information prior to installing the sensor operating system:

• Root password – a password for the privileged root account which is rarely needed (only during the configuration of the sensors network interfaces and setting the sensor host name)
• User password – a password for the non-privileged sensor account under which the various sensor capture and forwarding services run
• Encryption password (optional) – if the encrypted installation option was selected at boot time, the encryption password must be entered every time the sensor boots

Each of these passwords must be entered twice to ensure they were entered correctly.

After the passwords have been entered, the installer will proceed to format the system drive and install Hedgehog Linux.

At the end of the installation process, you will be prompted with a few self-explanatory yes/no questions:

• Disable IPv6?
• Automatically login to the GUI session?
• Should the GUI session be locked due to inactivity?
• Display the Standard Mandatory DoD Notice and Consent Banner? (only applies when installed on U.S. government information systems)

Following these prompts, the installer will reboot and Hedgehog Linux will boot.

Boot

Each time the sensor boots, a grub boot menu will be shown briefly, after which the sensor will proceed to load.

Kiosk mode

The sensor automatically logs in as the sensor user account and runs in kiosk mode, which is intended to show an at-a-glance view of the its resource utilization. Clicking the ☰ icon in allows you to switch between the resource statistics view and the services view.

The kiosk's services screen (designed with large clickable labels for small portable touch screens) can be used to start and stop essential services, get a status report of the currently running services, and clean all captured data from the sensor.

Configuration

Kiosk mode can be exited by connecting an external USB keyboard and pressing Alt+F4, upon which the sensor user's desktop is shown.

Several icons are available in the top menu bar:

• Terminal - opens a command prompt in a terminal emulator
• Browser - opens a web browser
• Kiosk – returns the sensor to kiosk mode
• README – displays this document
• Sensor status – displays a list with the status of each sensor service
• Configure capture and forwarding – opens a dialog for configuring the sensor's capture and forwarding services, as well as specifying which services should autostart upon boot
• Configure interfaces and hostname – opens a dialog for configuring the sensor's network interfaces and setting the sensor's hostname
• Restart sensor services - stops and restarts all of the autostart services

Interfaces, hostname, and time synchronization

Hostname

The first step of sensor configuration is to configure the network interfaces and sensor hostname. Clicking the Configure Interfaces and Hostname toolbar icon (or, if you are at a command line prompt, running configure-interfaces) will prompt you for the root password you created during installation, after which the configuration welcome screen is shown. Select Continue to proceed.

You may next select whether to configure the network interfaces, hostname, or time synchronization.

Selecting Hostname, you will be presented with a summary of the current sensor identification information, after which you may specify a new sensor hostname. This name will be used to tag all events forwarded from this sensor in the events' host.name field.

Interfaces

Returning to the configuration mode selection, choose Interface. You will be prompted if you would like help identifying network interfaces. If you select Yes, you will be prompted to select a network interface, after which that interface's link LED will blink for 10 seconds to help you in its identification. This network interface identification aid will continue to prompt you to identify further network interfaces until you select No.

You will be presented with a list of interfaces to configure as the sensor management interface. This is the interface the sensor itself will use to communicate with the network in order to, for example, forward captured logs to an aggregate server. In order to do so, the management interface must be assigned an IP address. This is generally not the interface used for capturing data. Select the interface to which you wish to assign an IP address. The interfaces are listed by name and MAC address and the associated link speed is also displayed if it can be determined. For interfaces without a connected network cable, generally a -1 will be displayed instead of the interface speed.

Depending on the configuration of your network, you may now specify how the management interface will be assigned an IP address. In order to communicate with an event aggregator over the management interface, either static or dhcp must be selected.

If you select static, you will be prompted to enter the IP address, netmask, and gateway to assign to the management interface.

In either case, upon selecting OK the network interface will be brought down, configured, and brought back up, and the result of the operation will be displayed. You may choose Quit upon returning to the configuration tool's welcome screen.

Time synchronization

Returning to the configuration mode selection, choose Time Sync. Here you can configure the sensor to keep its time synchronized with either an NTP server (using the NTP protocol) or a local Malcolm aggregator or another HTTP/HTTPS server. On the next dialog, choose the time synchronization method you wish to configure.

If htpdate is selected, you will be prompted to enter the IP address or hostname and port of an HTTP/HTTPS server (for a Malcolm instance, port 9200 may be used) and the time synchronization check frequency in minutes. A test connection will be made to determine if the time can be retrieved from the server.

If ntpdate is selected, you will be prompted to enter the IP address or hostname of the NTP server.

Upon configuring time synchronization, a "Time synchronization configured successfully!" message will be displayed, after which you will be returned to the welcome screen.

Capture, forwarding, and autostart services

Clicking the Configure Capture and Forwarding toolbar icon (or, if you are at a command prompt, running configure-capture) will launch the configuration tool for capture and forwarding. The root password is not required as it was for the interface and hostname configuration, as sensor services are run under the non-privileged sensor account. Select Continue to proceed. You may select from a list of configuration options.

Capture

Choose Configure Capture to configure parameters related to traffic capture and local analysis. You will be prompted if you would like help identifying network interfaces. If you select Yes, you will be prompted to select a network interface, after which that interface's link LED will blink for 10 seconds to help you in its identification. This network interface identification aid will continue to prompt you to identify further network interfaces until you select No.

You will be presented with a list of network interfaces and prompted to select one or more capture interfaces. An interface used to capture traffic is generally a different interface than the one selected previously as the management interface, and each capture interface should be connected to a network tap or span port for traffic monitoring. Capture interfaces are usually not assigned an IP address as they are only used to passively “listen” to the traffic on the wire. The interfaces are listed by name and MAC address and the associated link speed is also displayed if it can be determined. For interfaces without a connected network cable, generally a -1 will be displayed instead of the interface speed.

Upon choosing the capture interfaces and selecting OK, you may optionally provide a capture filter. This filter will be used to limit what traffic the PCAP service (tcpdump) and the traffic analysis services (zeek and suricata) will see. Capture filters are specified using Berkeley Packet Filter (BPF) syntax. Clicking OK will attempt to validate the capture filter, if specified, and will present a warning if the filter is invalid.

Next you must specify the paths where captured PCAP files and logs will be stored locally on the sensor. If the installation worked as expected, these paths should be prepopulated to reflect paths on the volumes formatted at install time for the purpose storing these artifacts. Usually these paths will exist on separate storage volumes. Enabling the PCAP and log pruning autostart services (see the section on autostart services below) will enable monitoring of these paths to ensure that their contents do not consume more than 90% of their respective volumes' space. Choose OK to continue.

Automatic file extraction and scanning

Hedgehog Linux can leverage Zeek's knowledge of network protocols to automatically detect file transfers and extract those files from network traffic as Zeek sees them.

To specify which files should be extracted, specify the Zeek file carving mode:

If you're not sure what to choose, either of mapped (except common plain text files) (if you want to carve and scan almost all files) or interesting (if you only want to carve and scan files with mime types of common attack vectors) is probably a good choice.

Next, specify which carved files to preserve (saved on the sensor under /capture/bro/capture/extract_files/quarantine by default). In order to not consume all of the sensor's available storage space, the oldest preserved files will be pruned along with the oldest Zeek logs as described below with AUTOSTART_PRUNE_ZEEK in the autostart services section.

You'll be prompted to specify which engine(s) to use to analyze extracted files. Extracted files can be examined through any of three methods:

• scanning files with ClamAV; to enable this method, select ZEEK_FILE_SCAN_CLAMAV when specifying scanners for Zeek-carved files
• submitting file hashes to VirusTotal; to enable this method, select ZEEK_FILE_SCAN_VTOT when specifying scanners for Zeek-carved files, then manually edit /opt/sensor/sensor_ctl/control_vars.conf and specify your VirusTotal API key in VTOT_API2_KEY
• scanning files with Yara; to enable this method, select ZEEK_FILE_SCAN_YARA when specifying scanners for Zeek-carved files
• scanning portable executable (PE) files with Capa; to enable this method, select ZEEK_FILE_SCAN_CAPA when specifying scanners for Zeek-carved files

Files which are flagged as potentially malicious will be logged as Zeek signatures.log entries, and can be viewed in the Signatures dashboard in OpenSearch Dashboards when forwarded to Malcolm.

Finally, you will be presented with the list of configuration variables that will be used for capture, including the values which you have configured up to this point in this section. Upon choosing OK these values will be written back out to the sensor configuration file located at /opt/sensor/sensor_ctl/control_vars.conf. It is not recommended that you edit this file manually. After confirming these values, you will be presented with a confirmation that these settings have been written to the configuration file, and you will be returned to the welcome screen.

Forwarding

Select Configure Forwarding to set up forwarding logs and statistics from the sensor to an aggregator server, such as Malcolm.

There are five forwarder services used on the sensor, each for forwarding a different type of log or sensor metric.

capture: Arkime session forwarding

capture is not only used to capture PCAP files, but also the parse raw traffic into sessions and forward this session metadata to an OpenSearch database so that it can be viewed in Arkime viewer, whether standalone or as part of a Malcolm instance. If you're using Hedgehog Linux with Malcolm, please read Correlating Zeek logs and Arkime sessions in the Malcolm documentation for more information.

First, select the OpenSearch connection transport protocol, either HTTPS or HTTP. If the metrics are being forwarded to Malcolm, select HTTPS to encrypt messages from the sensor to the aggregator using TLS v1.2 using ECDHE-RSA-AES128-GCM-SHA256. If HTTPS is chosen, you must choose whether to enable SSL certificate verification. If you are using a self-signed certificate (such as the one automatically created during Malcolm's configuration), choose None.

Next, enter the OpenSearch host IP address (ie., the IP address of the aggregator) and port. These metrics are written to an OpenSearch database using a RESTful API, usually using port 9200. Depending on your network configuration, you may need to open this port in your firewall to allow this connection from the sensor to the aggregator.

You will be asked to enter authentication credentials for the sensor's connections to the aggregator's OpenSearch API. After you've entered the username and the password, the sensor will attempt a test connection to OpenSearch using the connection information provided.

Finally, you will be shown a dialog for a list of IP addresses used to populate an access control list (ACL) for hosts allowed to connect back to the sensor for retrieving session payloads from its PCAP files for display in Arkime viewer. The list will be prepopulated with the IP address entered a few screens prior to this one.

Finally, you'll be given the opportunity to review the all of the Arkime capture options you've specified. Selecting OK will cause the parameters to be saved and you will be returned to the configuration tool's welcome screen.

filebeat: Zeek and Suricata log forwarding

Filebeat is used to forward Zeek and Suricata logs to a remote Logstash instance for further enrichment prior to insertion into an OpenSearch database.

To configure filebeat, first provide the log path (the same path previously configured for log file generation).

You must also provide the IP address of the Logstash instance to which the logs are to be forwarded, and the port on which Logstash is listening. These logs are forwarded using the Beats protocol, generally over port 5044. Depending on your network configuration, you may need to open this port in your firewall to allow this connection from the sensor to the aggregator.

Next you are asked whether the connection used for log forwarding should be done unencrypted or over SSL. Unencrypted communication requires less processing overhead and is simpler to configure, but the contents of the logs may be visible to anyone who is able to intercept that traffic.

If SSL is chosen, you must choose whether to enable SSL certificate verification. If you are using a self-signed certificate (such as the one automatically created during Malcolm's configuration, choose None.

The last step for SSL-encrypted log forwarding is to specify the SSL certificate authority, certificate, and key files. These files must match those used by the Logstash instance receiving the logs on the aggregator. If Malcolm's auth_setup script was used to generate these files they would be found in the filebeat/certs/ subdirectory of the Malcolm installation and must be manually copied to the sensor (stored under /opt/sensor/sensor_ctl/logstash-client-certificates or in any other path accessible to the sensor account). Specify the location of the certificate authorities file (eg., ca.crt), the certificate file (eg., client.crt), and the key file (eg., client.key).

The Logstash instance receiving the events must be similarly configured with matching SSL certificate and key files. Under Malcolm, the BEATS_SSL variable must be set to true in Malcolm's docker-compose.yml file and the SSL files must exist in the logstash/certs/ subdirectory of the Malcolm installation.

Once you have specified all of the filebeat parameters, you will be presented with a summary of the settings related to the forwarding of these logs. Selecting OK will cause the parameters to be written to filebeat's configuration keystore under /opt/sensor/sensor_ctl/logstash-client-certificates and you will be returned to the configuration tool's welcome screen.

miscbeat: System metrics forwarding

The sensor uses Fluent Bit to gather miscellaneous system resource metrics (CPU, network I/O, disk I/O, memory utilization, temperature, etc.) and the Beats protocol to forward these metrics to a remote Logstash instance for further enrichment prior to insertion into an OpenSearch database. Metrics categories can be enabled/disabled as described in the autostart services section of this document.

This forwarder's configuration is almost identical to that of filebeat in the previous section. Select miscbeat from the forwarding configuration mode options and follow the same steps outlined above to set up this forwarder.

Autostart services

Once the forwarders have been configured, the final step is to Configure Autostart Services. Choose this option from the configuration mode menu after the welcome screen of the sensor configuration tool.

Despite configuring capture and/or forwarder services as described in previous sections, only services enabled in the autostart configuration will run when the sensor starts up. The available autostart processes are as follows (recommended services are in bold text):

• AUTOSTART_ARKIME - capture PCAP engine for traffic capture, as well as traffic parsing and metadata insertion into OpenSearch for viewing in Arkime. If you are using Hedgehog Linux along with Malcolm or another Arkime installation, this is probably the packet capture engine you want to use.
• AUTOSTART_CLAMAV_UPDATES - Virus database update service for ClamAV (requires sensor to be connected to the internet)
• AUTOSTART_FILEBEAT - filebeat Zeek and Suricata log forwarder
• AUTOSTART_FLUENTBIT_AIDE - Fluent Bit agent monitoring AIDE file system integrity checks
• AUTOSTART_FLUENTBIT_AUDITLOG - Fluent Bit agent monitoring auditd logs
• AUTOSTART_FLUENTBIT_KMSG - Fluent Bit agent monitoring the Linux kernel log buffer (these are generally reflected in syslog as well, which may make this agent redundant)
• AUTOSTART_FLUENTBIT_METRICS - Fluent Bit agent for collecting various system resource and performance metrics
• AUTOSTART_FLUENTBIT_SYSLOG - Fluent Bit agent monitoring Linux syslog messages
• AUTOSTART_FLUENTBIT_THERMAL - Fluent Bit agent monitoring system temperatures
• AUTOSTART_MISCBEAT - filebeat forwarder which sends system metrics collected by Fluent Bit to a remote Logstash instance (e.g., Malcolm's)
• AUTOSTART_NETSNIFF - netsniff-ng PCAP engine for saving packet capture (PCAP) files
• AUTOSTART_PRUNE_PCAP - storage space monitor to ensure that PCAP files do not consume more than 90% of the total size of the storage volume to which PCAP files are written
• AUTOSTART_PRUNE_ZEEK - storage space monitor to ensure that Zeek logs do not consume more than 90% of the total size of the storage volume to which Zeek logs are written
• AUTOSTART_SURICATA - Suricata traffic analysis engine
• AUTOSTART_SURICATA_UPDATES - Rule update service for Suricata (requires sensor to be connected to the internet)
• AUTOSTART_TCPDUMP - tcpdump PCAP engine for saving packet capture (PCAP) files
• AUTOSTART_ZEEK - Zeek traffic analysis engine

Note that only one packet capture engine (capture, netsniff-ng, or tcpdump) can be used.

Once you have selected the autostart services, you will be prompted to confirm your selections. Doing so will cause these values to be written back out to the /opt/sensor/sensor_ctl/control_vars.conf configuration file.

After you have completed configuring the sensor it is recommended that you reboot the sensor to ensure all new settings take effect. If rebooting is not an option, you may click the Restart Sensor Services menu icon in the top menu bar, or open a terminal and run:

/opt/sensor/sensor_ctl/shutdown && sleep 10 && /opt/sensor/sensor_ctl/supervisor.sh

This will cause the sensor services controller to stop, wait a few seconds, and restart. You can check the status of the sensor's processes by choosing Sensor Status from the sensor's kiosk mode, clicking the Sensor Service Status toolbar icon, or running /opt/sensor/sensor_ctl/status from the command line:

$ /opt/sensor/sensor_ctl/status 
arkime:arkime-capture            RUNNING   pid 6455, uptime 0:03:17
arkime:arkime-viewer             RUNNING   pid 6456, uptime 0:03:17
beats:filebeat                   RUNNING   pid 6457, uptime 0:03:17
beats:miscbeat                   RUNNING   pid 6458, uptime 0:03:17
clamav:clamav-service            RUNNING   pid 6459, uptime 0:03:17
clamav:clamav-updates            RUNNING   pid 6461, uptime 0:03:17
fluentbit-auditlog               RUNNING   pid 6463, uptime 0:03:17
fluentbit-kmsg                   STOPPED   Not started
fluentbit-metrics:cpu            RUNNING   pid 6466, uptime 0:03:17
fluentbit-metrics:df             RUNNING   pid 6471, uptime 0:03:17
fluentbit-metrics:disk           RUNNING   pid 6468, uptime 0:03:17
fluentbit-metrics:mem            RUNNING   pid 6472, uptime 0:03:17
fluentbit-metrics:mem_p          RUNNING   pid 6473, uptime 0:03:17
fluentbit-metrics:netif          RUNNING   pid 6474, uptime 0:03:17
fluentbit-syslog                 RUNNING   pid 6478, uptime 0:03:17
fluentbit-thermal                RUNNING   pid 6480, uptime 0:03:17
netsniff:netsniff-enp1s0         STOPPED   Not started
prune:prune-pcap                 RUNNING   pid 6484, uptime 0:03:17
prune:prune-zeek                 RUNNING   pid 6486, uptime 0:03:17
supercronic                      RUNNING   pid 6490, uptime 0:03:17
suricata                         RUNNING   pid 6501, uptime 0:03:17
tcpdump:tcpdump-enp1s0           STOPPED   Not started
zeek:capa                        RUNNING   pid 6553, uptime 0:03:17
zeek:clamav                      RUNNING   pid 6512, uptime 0:03:17
zeek:logger                      RUNNING   pid 6554, uptime 0:03:17
zeek:virustotal                  STOPPED   Not started
zeek:watcher                     RUNNING   pid 6510, uptime 0:03:17
zeek:yara                        RUNNING   pid 6548, uptime 0:03:17
zeek:zeekctl                     RUNNING   pid 6502, uptime 0:03:17

Zeek Intelligence Framework

To quote Zeek's Intelligence Framework documentation, "The goals of Zeek’s Intelligence Framework are to consume intelligence data, make it available for matching, and provide infrastructure to improve performance and memory utilization. Data in the Intelligence Framework is an atomic piece of intelligence such as an IP address or an e-mail address. This atomic data will be packed with metadata such as a freeform source field, a freeform descriptive field, and a URL which might lead to more information about the specific item." Zeek intelligence indicator types include IP addresses, URLs, file names, hashes, email addresses, and more.

Hedgehog Linux doesn't come bundled with intelligence files from any particular feed, but they can be easily included into your local instance. Before Zeek starts, Hedgehog Linux configures it such that intelligence files will be automatically included in its local policy. Subdirectories under /opt/sensor/sensor_ctl/zeek/intel which contain their own __load__.zeek file will be @load-ed as-is, while subdirectories containing "loose" intelligence files will be loaded automatically with a redef Intel::read_files directive.

Note that Hedgehog Linux does not manage updates for these intelligence files. You should use the update mechanism suggested by your feeds' maintainers to keep them up to date. Adding and deleting intelligence files under this directory will take effect upon restarting Zeek.

Appendix A - Generating the ISO

Official downloads of the Hedgehog Linux installer ISO are not provided: however, it can be built easily on an internet-connected Linux host with Vagrant:

• Vagrant
  • vagrant-reload plugin
  • vagrant-sshfs plugin
  • bento/debian-11 Vagrant box

The build should work with either the VirtualBox provider or the libvirt provider:

• VirtualBox provider
  • vagrant-vbguest plugin
• libvirt
  • vagrant-libvirt provider plugin
  • vagrant-mutate plugin to convert bento/debian-11 Vagrant box to libvirt format

To perform a clean build the Hedgehog Linux installer ISO, navigate to your local Malcolm working copy and run:

$ ./sensor-iso/build_via_vagrant.sh -f
…
Starting build machine...
Bringing machine 'default' up with 'virtualbox' provider...
…

Building the ISO may take 90 minutes or more depending on your system. As the build finishes, you will see the following message indicating success:

…
Finished, created "/sensor-build/hedgehog-6.2.0.iso"
…

Alternately, if you have forked Malcolm on GitHub, workflow files are provided which contain instructions for GitHub to build the docker images and Hedgehog and Malcolm installer ISOs, specifically sensor-iso-build-docker-wrap-push-ghcr.yml for the Hedgehog ISO. The resulting ISO file is wrapped in a Docker image that provides an HTTP server from which the ISO may be downloaded.

Appendix B - Configuring SSH access

SSH access to the sensor's non-privileged sensor account is only available using secure key-based authentication which can be enabled by adding a public SSH key to the /home/sensor/.ssh/authorized_keys file as illustrated below:

sensor@sensor:~$ mkdir -p ~/.ssh

sensor@sensor:~$ ssh analyst@172.16.10.48 "cat ~/.ssh/id_rsa.pub" >> ~/.ssh/authorized_keys
The authenticity of host '172.16.10.48 (172.16.10.48)' can't be established.
ECDSA key fingerprint is SHA256:...
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.10.48' (ECDSA) to the list of known hosts.
analyst@172.16.10.48's password:

sensor@sensor:~$ cat ~/.ssh/authorized_keys
ssh-rsa AAA...kff analyst@SOC

SSH access should only be configured when necessary.

Appendix C - Troubleshooting

Should the sensor not function as expected, first try rebooting the device. If the behavior continues, here are a few things that may help you diagnose the problem (items which may require Linux command line use are marked with †)

• Stop / start services – Using the sensor's kiosk mode, attempt a Services Stop followed by a Services Start, then check Sensor Status to see which service(s) may not be running correctly.
• Sensor configuration file – See /opt/sensor/sensor_ctl/control_vars.conf for sensor service settings. It is not recommended to manually edit this file unless you are sure of what you are doing.
• Sensor control scripts – There are scripts under /opt/sensor/sensor_ctl/ to control sensor services (eg., shutdown, start, status, stop, etc.)
• Sensor debug logs – Log files under /opt/sensor/sensor_ctl/log/ may contain clues to processes that are not working correctly. If you can determine which service is failing, you can attempt to reconfigure it using the instructions in the Configure Capture and Forwarding section of this document.
• sensorwatch script – Running sensorwatch on the command line will display the most recently modified PCAP and Zeek log files in their respective directories, how much storage space they are consuming, and the amount of used/free space on the volumes containing those files.

Appendix D - Hardening

Hedgehog Linux targets the following guidelines for establishing a secure configuration posture:

• DISA STIG (Security Technical Implementation Guides) ported from DISA RHEL 7 STIG v1r1 to a Debian 9 base platform
• CIS Debian Linux 9 Benchmark with additional recommendations by the hardenedlinux/harbian-audit project

STIG compliance exceptions

Currently there are 158 compliance checks that can be verified automatically and 23 compliance checks that must be verified manually.

Hedgehog Linux claims the following exceptions to STIG compliance:

Please review the notes for these additional rules. While not claiming an exception, they may be implemented or checked in a different way than outlined by the RHEL STIG as Hedgehog Linux is not built on RHEL or for other reasons.

In addition, DISA STIG rules SV-86663r1, SV-86695r2, SV-86759r3, SV-86761r3, SV-86763r3, SV-86765r3, SV-86595r1, and SV-86615r2 relate to the SELinux kernel which is not used in Hedgehog Linux, and are thus skipped.

CIS benchmark compliance exceptions

Currently there are 271 checks to determine compliance with the CIS Debian Linux 9 Benchmark.

Hedgehog Linux claims exceptions from the recommendations in this benchmark in the following categories:

1.1 Install Updates, Patches and Additional Security Software - When the Hedgehog Linux sensor appliance software is built, all of the latest applicable security patches and updates are included in it. How future updates are to be handled is still in design.

1.3 Enable verify the signature of local packages - As the base distribution is not using embedded signatures, debsig-verify would reject all packages (see comment in /etc/dpkg/dpkg.cfg). Enabling it after installation would disallow any future updates.

2.14 Add nodev option to /run/shm Partition, 2.15 Add nosuid Option to /run/shm Partition, 2.16 Add noexec Option to /run/shm Partition - Hedgehog Linux does not mount /run/shm as a separate partition, so these recommendations do not apply.

2.18 Disable Mounting of cramfs Filesystems, 2.19 Disable Mounting of freevxfs Filesystems, 2.20 Disable Mounting of jffs2 Filesystems, 2.21 Disable Mounting of hfs Filesystems, 2.22 Disable Mounting of hfsplus Filesystems, 2.23 Disable Mounting of squashfs Filesystems, 2.24 Disable Mounting of udf Filesystems - Hedgehog Linux is not compiling a custom Linux kernel, so these filesystems are inherently supported as they are part Debian Linux's default kernel.

4.6 Disable USB Devices - The ability to copy data captured by the sensor to a mounted USB mass storage device is a requirement of the system.

6.1 Ensure the X Window system is not installed, 6.2 Ensure Avahi Server is not enabled, 6.3 Ensure print server is not enabled - A locked-down X Windows session is required for the sensor's kiosk display. The library packages libavahi-common-data, libavahi-common3, and libcups2 are dependencies of some of the X components used by Hedgehog Linux, but the avahi and cups services themselves are disabled.

6.17 Ensure virus scan Server is enabled, 6.18 Ensure virus scan Server update is enabled - As this is a network traffic capture appliance rather than an end-user device and will not be internet-connected, regular user files will not be created. A virus scan program would impact device performance and would be unnecessary.

7.2.4 Log Suspicious Packets, 7.2.7 Enable RFC-recommended Source Route Validation, 7.4.1 Install TCP Wrappers - As this is a network traffic capture appliance sniffing packets on a network interface configured in promiscuous mode, these recommendations do not apply.

Password-related recommendations under 9.2 and 10.1 - The library package libpam-pwquality is used in favor of libpam-cracklib which is what the compliance scripts are looking for. Also, as a sensor running Hedgehog Linux is intended to be used as an appliance rather than a general user-facing software platform, some exceptions to password enforcement policies are claimed.

9.3.13 Limit Access via SSH - Hedgehog Linux does not create multiple regular user accounts: only root and a sensor service account are used. SSH access for root is disabled. SSH login with a password is also disallowed: only key-based authentication is accepted. The sensor service account accepts no keys by default. As such, the AllowUsers, AllowGroups, DenyUsers, and DenyGroups values in sshd_config do not apply.

9.5 Restrict Access to the su Command - Hedgehog Linux does not create multiple regular user accounts: only root and a sensor service account are used.

10.1.10 Set maxlogins for all accounts and 10.5 Set Timeout on ttys - Hedgehog Linux does not create multiple regular user accounts: only root and a sensor service account are used.

12.10 Find SUID System Executables, 12.11 Find SGID System Executables - The few files found by these scripts are valid exceptions required by Hedgehog Linux's system requirements.

Please review the notes for these additional guidelines. While not claiming an exception, Hedgehog Linux may implement them in a manner different than is described by the CIS Debian Linux 9 Benchmark or the hardenedlinux/harbian-audit audit scripts.

4.1 Restrict Core Dumps - Hedgehog Linux disables core dumps using a configuration file for ulimit named /etc/security/limits.d/limits.conf. The audit script checking for this does not check the limits.d subdirectory, which is why this is incorrectly flagged as noncompliant.

5.4 Ensure ctrl-alt-del is disabled - Hedgehog Linux disables the ctrl+alt+delete key sequence by executing systemctl disable ctrl-alt-del.target during installation and the command systemctl mask ctrl-alt-del.target at boot time.

6.19 Configure Network Time Protocol (NTP) - While time synchronization is supported on Hedgehog Linux, an exception is claimed for this rule as the network sensor device may be configured to sync to servers in a different way than specified in the benchmark.

7.4.4 Create /etc/hosts.deny, 7.7.1 Ensure Firewall is active, 7.7.4.1 Ensure default deny firewall policy, 7.7.4.3 Ensure default deny firewall policy, 7.7.4.4 Ensure outbound and established connections are configured - Hedgehog Linux is configured with an appropriately locked-down software firewall (managed by "Uncomplicated Firewall" ufw). However, the methods outlined in the CIS benchmark recommendations do not account for this configuration.

8.1.1.2 Disable System on Audit Log Full, 8.1.1.3 Keep All Auditing Information, 8.1.1.5 Ensure set remote server for audit service, 8.1.1.6 Ensure enable_krb5 set to yes for remote audit service, 8.1.1.7 Ensure set action for audit storage volume is fulled, 8.1.1.9 Set space left for auditd service, a few other audit-related items under section 8.1, 8.2.5 Configure rsyslog to Send Logs to a Remote Log Host - As maximizing availability is a system requirement, audit processing failures will be logged on the device rather than halting the system. Because Hedgehog Linux is intended to be used as an appliance rather than a general network host, notifications about its status are sent in system logs forwarded to the OpenSearch database on the aggregator. auditd is set up to syslog when this storage volume is reached. Fluent Bit offloads audit records to an OpenSearch database on another system, though this is not detected by the CIS benchmark compliance scripts. Local logs are generated when the network connection is broken, and it resumes automatically. Syslog messages are also similarly forwarded.

8.7 Verifies integrity all packages - The script which verifies package integrity only "fails" because of missing (status ??5?????? displayed by the utility) language ("locale") files, which are removed as part of Hedgehog Linux's trimming-down process. All non-locale-related system files pass intergrity checks.

Hardening compliance issues - work in progress

Hedgehog Linux has recently replaced several Beats forwarders, including auditbeat, with Fluent Bit. While auditd logs can be configured to be forwarded to an OpenSearch database on an external aggregator, requirements for file integrity checks are in the progress of being implemented with AIDE.

Until that work is complete, Hedgehog Linux is not in compliance with the following items:

• STIG

• CIS

8.4.1 Install aide package and 8.4.2 Implement Periodic Execution of File Integrity

Appendix E - Upgrades

At this time there is not an "official" upgrade procedure to get from one release of Hedgehog Linux to the next. Upgrading the underlying operating system packages is generally straightforward, but not all of the Hedgehog Linux components are packaged into .deb archives yet as they should be, so for now it's a manual (and kind of nasty) process to Frankenstein an upgrade into existance. The author of this project intends to remedy this at some future point when time and resources allow.

If possible, it would save you a lot of trouble to just re-ISO your Hedgehog installation and start fresh, backing up the files (in /opt/sensor/sensor_ctl) first and reconfiguring or restoring them as needed afterwards.

However, if reinstalling the system is not an option, here is the basic process for doing a manual upgrade of Hedgehog Linux. It should be understood that this process is very likely to break your system, and there is no guarantee of any kind that any of this will work, or that these instructions are even complete or any support whatsoever regarding them. Really, it will be much easier if you re-ISO your installation. But for the brave among you, here you go. ⛔🆘😭💀

Prerequisites

• A good understanding of the Linux command line
• An existing installation of Hedgehog Linux with internet access
• A copy of the Hedgehog Linux ISO for the version approximating the one you're upgrading to (i.e., the latest version), and
  • Either a separate VM with that ISO installed OR
  • A separate Linux workstation where you can manually mount that ISO to pull stuff off of it

Upgrade

1. Obtain a root shell

   • su -

2. Temporarily set the umask value to Debian default instead of the more restrictive Hedgehog Linux default. This will allow updates to be applied with the right permissions.

   • umask 0022

3. Create backups of some files

   • cp /etc/apt/sources.list /etc/apt/sources.list.bak

4. Set up alternate package sources, if needed

   • In an offline/airgapped scenario, you could use apt-mirror to mirror Debian repos and bandersnatch to mirror PyPI sources, or combine them with Docker. If you were to do this, you'd probably want to make the following changes (and revert them after the upgrade):
     • create /etc/apt/apt.conf.d/80ssl-exceptions to ignore self-signed certificate warnings from using your apt-mirror

Acquire::https {
  Verify-Peer "false";
  Verify-Host "false";
}

    + modify `/etc/apt/source.list` to point to your apt-mirror:

deb https://XXXXXX:443/debian buster main contrib non-free
deb https://XXXXXX:443/debian-security buster/updates main contrib non-free
deb https://XXXXXX:443/debian buster-updates main contrib non-free
deb https://XXXXXX:443/debian buster-backports main contrib non-free

1. Update underlying system packages with apt-get

   • apt-get update && apt-get dist-upgrade

2. If there were new system deb packages added to this release of Hedgehog Linux (you might have to manually compare on GitHub), install them. If you're not sure, of course, you could just install everything, like this (although you may have to tweak some version numbers or something if the base distribution of your Hedgehog branch is different than main; in this example I'm not jumping between Debian releases, just upgrading within a release):

$ for LIST in apps desktopmanager net system; do curl -L -J -O https://raw.github.com/idaholab/Malcolm/main/sensor-iso/config/package-lists/$LIST.list.chroot; done
...
$ apt-get install $(cat *.list.chroot)

1. Update underlying python packages with python3 -m pip

   • apt-get install -y build-essential git-core pkg-config python3-dev
   • python3 -m pip list --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1 | xargs -r -n1 python3 -m pip install -U
     • if this fails for some reason, you may need to reinstall pip first with python3 -m pip install --force -U pip
     • some very old builds of Hedgehog Linux had separate Python 3.5 and 3.7 installations: in this case, you'd need to do this for both python3 -m pip and python3.7 -m pip (or whatever python3.x you have)
   • If there were new python packages added to this release of Hedgehog Linux (you might have to manually compare on GitHub), install them. If you are using a PyPI mirror, replace XXXXXX here with your mirror's IP. The colorama package is used here as an example, your package list might vary.
     • python3 -m pip install --no-compile --no-cache-dir --force-reinstall --upgrade --index-url=https://XXXXXX:443/pypi/simple --trusted-host=XXXXXX:443 colorama

2. Okay, now things start to get a little bit ugly. You're going to need access to the ISO of the release of Hedgehog Linux you're upgrading to, as we're going to grab some packages off of it. On another Linux system, build it.

3. Use a disk image mounter to mount the ISO, or if you want to just install the ISO in a VM and grab the files we need off of it, that's fine too. But I'll go through the example as if I've mounted the ISO.

4. Navigate to the /live/ directory, and mount the filesystem.squashfs file

   • sudo mount filesystem.squashfs /media/squash -t squashfs -o loop
   • OR
   • squashfuse filesystem.squashfs /home/user/media/squash

5. Very recent builds of Hedgehog Linux keep some build artifacts in /opt/hedgehog_install_artifacts/. You're going to want to grab those files and throw them in a temporary directory on the system you're upgrading, via SSH or whatever means you devise.

root@hedgehog:/tmp# scp -r user@otherbox:/media/squash/opt/hedgehog_install_artifacts/ ./
user@otherbox's password: 
filebeat-tweaked-7.6.2-amd64.deb                                                100%   13MB  65.9MB/s   00:00    
arkime_2.2.3-1_amd64.deb                                                        100%  113MB  32.2MB/s   00:03    
netsniff-ng_0.6.6-1_amd64.deb                                                   100%  330KB  52.1MB/s   00:00    
zeek_3.0.20-1_amd64.deb                                                         100%   26MB  63.1MB/s   00:00

1. Blow away the old zeek package, we're going to start clean with that one particularly. The others should be fine to upgrade in place.

root@hedgehog:/opt# apt-get --purge remove zeek
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  zeek*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 160 MB disk space will be freed.
Do you want to continue? [Y/n] y
(Reading database ... 118490 files and directories currently installed.)
Removing zeek (3.0.20-1) ...
dpkg: warning: while removing zeek, directory '/opt/zeek/spool' not empty so not removed
dpkg: warning: while removing zeek, directory '/opt/zeek/share/zeek/site' not empty so not removed
dpkg: warning: while removing zeek, directory '/opt/zeek/lib' not empty so not removed
dpkg: warning: while removing zeek, directory '/opt/zeek/bin' not empty so not removed
root@hedgehog:/opt# rm -rf /opt/zeek*

1. Install the new .deb files. You're going to have some warnings, but that's okay.

root@hedgehog:/tmp# dpkg -i hedgehog_install_artifacts/*.deb
(Reading database ... 118149 files and directories currently installed.)
Preparing to unpack .../filebeat-tweaked-7.6.2-amd64.deb ...
Unpacking filebeat (7.6.2) over (6.8.4) ...
dpkg: warning: unable to delete old directory '/usr/share/filebeat/kibana/6/dashboard': Directory not empty
dpkg: warning: unable to delete old directory '/usr/share/filebeat/kibana/6': Directory not empty
Preparing to unpack .../arkime_2.2.3-1_amd64.deb ...
Unpacking arkime (2.2.3-1) over (2.0.1-1) ...
Preparing to unpack .../netsniff-ng_0.6.6-1_amd64.deb ...
Unpacking netsniff-ng (0.6.6-1) over (0.6.6-1) ...
Preparing to unpack .../zeek_3.0.20-1_amd64.deb ...
Unpacking zeek (3.0.20-1) over (3.0.0-1) ...
Setting up filebeat (7.6.2) ...
Installing new version of [...]
[...]
Setting up arkime (2.2.3-1) ...
READ /opt/arkime/README.txt and RUN /opt/arkime/bin/Configure
Setting up netsniff-ng (0.6.6-1) ...
Setting up zeek (3.0.20-1) ...
Processing triggers for systemd (232-25+deb9u12) ...
Processing triggers for man-db (2.7.6.1-2) ...

1. Fix anything that might need fixing as far as the deb package requirements go

   • apt-get -f install

2. We just installed a Zeek .deb, but the third-part plugins packages and local config weren't part of that package. So we're going to rsync those from the other box where we have the ISO and filesystem.squashfs mounted as well:

root@hedgehog:/tmp# rsync -a user@otherbox:/media/squash/opt/zeek/ /opt/zeek 
user@otherbox's password: 

root@hedgehog:/tmp# ls -l /opt/zeek/share/zeek/site/
total 52
lrwxrwxrwx  1 root root    13 May  6 21:52 bzar -> packages/bzar
lrwxrwxrwx  1 root root    22 May  6 21:50 cve-2020-0601 -> packages/cve-2020-0601
-rw-r--r--  1 root root  2031 Apr 30 16:02 extractor.zeek
-rw-r--r--  1 root root 39134 May  1 14:20 extractor_params.zeek
lrwxrwxrwx  1 root root    14 May  6 21:52 hassh -> packages/hassh
lrwxrwxrwx  1 root root    12 May  6 21:52 ja3 -> packages/ja3
-rw-rw-r--  1 root root  2005 May  6 21:54 local.zeek
drwxr-xr-x 13 root root  4096 May  6 21:52 packages
lrwxrwxrwx  1 root root    27 May  6 21:52 zeek-EternalSafety -> packages/zeek-EternalSafety
lrwxrwxrwx  1 root root    26 May  6 21:52 zeek-community-id -> packages/zeek-community-id
lrwxrwxrwx  1 root root    27 May  6 21:51 zeek-plugin-bacnet -> packages/zeek-plugin-bacnet
lrwxrwxrwx  1 root root    25 May  6 21:51 zeek-plugin-enip -> packages/zeek-plugin-enip
lrwxrwxrwx  1 root root    29 May  6 21:51 zeek-plugin-profinet -> packages/zeek-plugin-profinet
lrwxrwxrwx  1 root root    27 May  6 21:52 zeek-plugin-s7comm -> packages/zeek-plugin-s7comm
lrwxrwxrwx  1 root root    24 May  6 21:52 zeek-plugin-tds -> packages/zeek-plugin-tds

1. The zeekctl component of zeek doesn't like being run by an unprivileged user unless the whole directory is owned by that user. As Hedgehog Linux runs everything it can as an unprivileged user, we're going to reset zeek to a "clean" state after each reboot. Zeek's config files will get regenerated when Zeek itself is started. So, now make a complete backup of /opt/zeek as it's going to have its ownership changed during runtime:

root@hedgehog:/tmp# rsync -a /opt/zeek/ /opt/zeek.orig

root@hedgehog:/tmp# chown -R sensor:sensor /opt/zeek/*

root@hedgehog:/tmp# chown -R root:root /opt/zeek.orig/*

root@hedgehog:/tmp# ls -l /opt/ | grep zeek
drwxr-xr-x  8 root   root    4096 May  8 15:48 zeek
drwxr-xr-x  8 root   root    4096 May  8 15:48 zeek.orig

1. Grab other new scripts and stuff from our mount of the ISO using rsync:

root@hedgehog:/tmp# rsync -a user@otherbox:/media/squash/usr/local/bin/ /usr/local/bin
user@otherbox's password: 

root@hedgehog:/tmp# ls -l /usr/local/bin/ | tail
lrwxrwxrwx 1 root root        18 May  8 14:34 zeek -> /opt/zeek/bin/zeek
-rwxr-xr-x 1 root staff    10349 Oct 29  2019 zeek_carve_logger.py
-rwxr-xr-x 1 root staff    10467 Oct 29  2019 zeek_carve_scanner.py
-rw-r--r-- 1 root staff    25756 Oct 29  2019 zeek_carve_utils.py
-rwxr-xr-x 1 root staff     8787 Oct 29  2019 zeek_carve_watcher.py
-rwxr-xr-x 1 root staff     4883 May  4 17:39 zeek_install_plugins.sh

root@hedgehog:/tmp# rsync -a user@otherbox:/media/squash/opt/yara-rules/ /opt/yara-rules
user@otherbox's password: 

root@hedgehog:/tmp# rsync -a user@otherbox:/media/squash/opt/capa-rules/ /opt/capa-rules
user@otherbox's password: 

root@hedgehog:/tmp# ls -l /opt/ | grep '\-rules'
drwxr-xr-x  8 root   root    4096 May  8 15:48 capa-rules
drwxr-xr-x  8 root   root  24576  May  8 15:48 yara-rules

root@hedgehog:/tmp# for BEAT in filebeat; do rsync -a user@otherbox:/media/squash/usr/share/$BEAT/kibana/ /usr/share/$BEAT/kibana; done
user@otherbox's password: 
user@otherbox's password: 

root@hedgehog:/tmp# rsync -avP --delete user@otherbox:/media/squash/etc/audit/rules.d/ /etc/audit/rules.d/
user@otherbox's password: 

root@hedgehog:/tmp# rsync -avP --delete user@otherbox:/media/squash/etc/sudoers.d/ /etc/sudoers.d/
user@otherbox's password: 

root@hedgehog:/tmp# chmod 400 /etc/sudoers.d/*

1. Set capabilities and symlinks for network capture programs to be used by the unprivileged user:

commands:

chown root:netdev /usr/sbin/netsniff-ng && \
  setcap 'CAP_NET_RAW+eip CAP_NET_ADMIN+eip CAP_IPC_LOCK+eip CAP_SYS_ADMIN+eip' /usr/sbin/netsniff-ng
chown root:netdev /opt/zeek/bin/zeek && \
  setcap 'CAP_NET_RAW+eip CAP_NET_ADMIN+eip CAP_IPC_LOCK+eip' /opt/zeek/bin/zeek
chown root:netdev /sbin/ethtool && \
  setcap 'CAP_NET_RAW+eip CAP_NET_ADMIN+eip' /sbin/ethtool
chown root:netdev /opt/zeek/bin/capstats && \
  setcap 'CAP_NET_RAW+eip CAP_NET_ADMIN+eip' /opt/zeek/bin/capstats
chown root:netdev /usr/bin/tcpdump && \
  setcap 'CAP_NET_RAW+eip CAP_NET_ADMIN+eip' /usr/bin/tcpdump
chown root:netdev /opt/arkime/bin/capture && \
  setcap 'CAP_NET_RAW+eip CAP_NET_ADMIN+eip CAP_IPC_LOCK+eip' /opt/arkime/bin/capture

ln -s -f /opt/zeek/bin/zeek /usr/local/bin/
ln -s -f /usr/sbin/netsniff-ng /usr/local/bin/
ln -s -f /usr/bin/tcpdump /usr/local/bin/
ln -s -f /opt/arkime/bin/capture /usr/local/bin/
ln -s -f /opt/arkime/bin/npm /usr/local/bin
ln -s -f /opt/arkime/bin/node /usr/local/bin
ln -s -f /opt/arkime/bin/npx /usr/local/bin

example:

root@hedgehog:/tmp# chown root:netdev /usr/sbin/netsniff-ng && \
>   setcap 'CAP_NET_RAW+eip CAP_NET_ADMIN+eip CAP_IPC_LOCK+eip CAP_SYS_ADMIN+eip' /usr/sbin/netsniff-ng
root@hedgehog:/tmp# chown root:netdev /opt/zeek/bin/zeek && \
>   setcap 'CAP_NET_RAW+eip CAP_NET_ADMIN+eip CAP_IPC_LOCK+eip' /opt/zeek/bin/zeek
root@hedgehog:/tmp# chown root:netdev /sbin/ethtool && \
>   setcap 'CAP_NET_RAW+eip CAP_NET_ADMIN+eip' /sbin/ethtool
root@hedgehog:/tmp# chown root:netdev /opt/zeek/bin/capstats && \
>   setcap 'CAP_NET_RAW+eip CAP_NET_ADMIN+eip' /opt/zeek/bin/capstats
root@hedgehog:/tmp# chown root:netdev /usr/bin/tcpdump && \
>   setcap 'CAP_NET_RAW+eip CAP_NET_ADMIN+eip' /usr/bin/tcpdump
root@hedgehog:/tmp# chown root:netdev /opt/arkime/bin/capture && \
>   setcap 'CAP_NET_RAW+eip CAP_NET_ADMIN+eip CAP_IPC_LOCK+eip' /opt/arkime/bin/capture
root@hedgehog:/tmp# ln -s -f /opt/zeek/bin/zeek /usr/local/bin/
root@hedgehog:/tmp# ln -s -f /usr/sbin/netsniff-ng /usr/local/bin/
root@hedgehog:/tmp# ln -s -f /usr/bin/tcpdump /usr/local/bin/
root@hedgehog:/tmp# ln -s -f /opt/arkime/bin/capture /usr/local/bin/
root@hedgehog:/tmp# ln -s -f /opt/arkime/bin/npm /usr/local/bin
root@hedgehog:/tmp# ln -s -f /opt/arkime/bin/node /usr/local/bin
root@hedgehog:/tmp# ln -s -f /opt/arkime/bin/npx /usr/local/bin

1. Back up unprivileged user sensor-specific config and scripts:

   • mv /opt/sensor/ /opt/sensor_upgrade_backup_$(date +%Y-%m-%d)

2. Grab unprivileged user sensor-specific config and scripts from our mount of the ISO using rsync and change its ownership to the unprivileged user:

root@hedgehog:/tmp# rsync -av user@otherbox:/media/squash/opt/sensor /opt/
user@otherbox's password: 
receiving incremental file list
created directory ./opt
sensor/
[...]

sent 1,244 bytes  received 1,646,409 bytes  470,758.00 bytes/sec
total size is 1,641,629  speedup is 1.00

root@hedgehog:/tmp# chown -R sensor:sensor /opt/sensor*

root@hedgehog:/tmp# ls -l /opt/ | grep sensor
drwxr-xr-x  4 sensor sensor  4096 May  6 22:00 sensor
drwxr-x---  4 sensor sensor  4096 May  8 14:33 sensor_upgrade_backup_2020-05-08

1. Leave the root shell and cd to /opt

root@hedgehog:~# exit
logout

sensor@hedgehog:~$ whoami
sensor

sensor@hedgehog:~$ cd /opt

1. Compare the old and new control_vars.conf files

sensor@hedgehog:opt$ diff sensor_upgrade_backup_2020-05-08/sensor_ctl/control_vars.conf sensor/sensor_ctl/control_vars.conf 
1,2c1,2
< export CAPTURE_INTERFACE=enp0s3
< export CAPTURE_FILTER="not port 5044 and not port 5601 and not port 8005 and not port 9200 and not port 9600"
---
> export CAPTURE_INTERFACE=xxxx
> export CAPTURE_FILTER=""
4c4
[...]

Examine the differences. If there aren't any new export variables, then you're probably safe to just replace the default version of control_vars.conf with the backed-up version:

sensor@hedgehog:opt$ cp sensor_upgrade_backup_2020-05-08/sensor_ctl/control_vars.conf sensor/sensor_ctl/control_vars.conf 
cp: overwrite 'sensor/sensor_ctl/control_vars.conf'? y

If there are major differences or new variables, continue on to the next step, in a minute you'll need to run capture-config to configure from scratch anyway.

1. Restore certificates/keystores for forwarders from the backup sensor_ctl path to the new one

sensor@hedgehog:opt$ for BEAT in filebeat miscbeat; do cp /opt/sensor_upgrade_backup_2020-05-08/sensor_ctl/$BEAT/data/* /opt/sensor/sensor_ctl/$BEAT/data/; done

sensor@hedgehog:opt$ cp /opt/sensor_upgrade_backup_2020-05-07/sensor_ctl/filebeat/{ca.crt,client.crt,client.key} /opt/sensor/sensor_ctl/logstash-client-certificates/

1. Despite what we just did, you may consider running capture-config to re-configure capture, forwarding, and autostart services from scratch anyway. You can use the backed-up version of control_vars.conf to refer back to as a basis for things you might want to restore (e.g., CAPTURE_INTERFACE, CAPTURE_FILTER, PCAP_PATH, ZEEK_LOG_PATH, your autostart settings, etc.).

2. Once you feel confident you've completed all of these steps, issue a reboot on the Hedgehog

Post-upgrade

Once the Hedgehog has come back up, check to make sure everything is working:

• /opt/sensor/sensor_ctl/status should return RUNNING for the things you set to autorun (no FATAL errors)
• sensorwatch should show current writes to Zeek log files and PCAP files (depending on your configuration)
• tail -f /opt/sensor/sensor_ctl/log/* should show no egregious errors
• zeek --version, zeek -N local and capture --version ought to run and print out version information as expected
• if you are forwarding to a Malcolm aggregator, you should start seeing data momentarily

Copyright

Hedgehog Linux - part of Malcolm - is Copyright 2022 Battelle Energy Alliance, LLC, and is developed and released through the cooperation of the Cybersecurity and Infrastructure Security Agency of the U.S. Department of Homeland Security.

See License.txt for the terms of its release.

Contact information of author(s):

malcolm@inl.gov