Maintaining the Tanium Appliance

Perform regular maintenance tasks to ensure that the Tanium Appliance infrastructure remains in good health. For maintenance tasks to be performed on a routine basis, see Perform monthly maintenance and Perform quarterly maintenance. If an appliance is not performing as expected, you might need to troubleshoot issues or change settings. See Troubleshooting for related procedures.

Use the Appliance Maintenance menu to perform most maintenance tasks, such as backups, system resets, TanOS upgrades, and system reboots or shutdowns.

Configure Tanium Appliance monitoring

Perform any of the following tasks to facilitate monitoring the health of your Tanium deployment and Appliance infrastructure. For example, if your organization has a syslog server or SNMP manager, you can integrate it with the Appliance for monitoring. If these monitoring solutions reveal issues that require resolution, see Troubleshooting.

Configure TanOS alerts

TanOS can send alerts to a syslog server or to an email recipient. For optimal results, configure an SMTP email recipient. If the syslog server fails, the SMTP recipient receives a failure notification every 15 minutes until either the failure is resolved or syslog forwarding is disabled. See Configure alerts.

Configure syslog forwarding

You can forward Appliance logs to a remote syslog server. The syslog forwarding configuration is separate from the syslog alert configuration. For the differences, and the steps to configure syslog forwarding, see Configuring syslog.

Configure SNMP

You can configure integration with an SNMP manager to collect and analyze Appliance information. After you configure credentials, the user tansnmp can make a remote SNMP connection to the Appliance or to the Integrated Dell Remote Access Controller (iDRAC) interface of a physical Appliance to conduct SNMP polling from a remote host or SNMP manager. See Configuring SNMP.

Perform monthly maintenance

If these tasks reveal issues that require resolution, see Troubleshooting.

Review the Health Check report

The Health Check report provides information on the health of the Appliance operating system, hardware, users, network, services, applications, database replication, RAID security, Postgres SSL, and virtual machine (if applicable).

  1. Run the report. See Run the Health Check.

  2. Review the output for actionable items, which are summarized at the end of the output.

    For example, the output might indicate that the End User License Agreement (EULA) is not accepted.

Monitor Appliance performance (optional)

See the following tasks for the steps to run commands for viewing Appliance performance information:

  • Run a SAR command to view statistical information such as CPU load, memory paging, memory utilization, swap usage, and network input/output (I/O).
  • Run the iotop command to view I/O utilization by process.
  • Run the perf top command to view CPU usage by function.
  • Run the htop command to view detailed information about each running process, such as memory and CPU consumption. The output provides an interface whereby you can navigate among values and tabs by keyboard and mouse.

Perform quarterly maintenance

Verify the grub key backup

You can use the grub key during the boot sequence to diagnose and recover from failure conditions. During recovery, you must provide the key to Tanium Support for a technician to extract the grub password.

  1. Verify that a backup of the latest key resides in a safe location off the Appliance.

    A new backup is required whenever the key password is regenerated. See Change the grub key password.

  2. Export the key and save it in a safe location if no backup exists or if the current backup is not the latest. See Export the grub key.

Review and update TanOS user accounts

  1. On each appliance, review the TanOS system users to ensure that they can access the Appliance operating system and that they have the appropriate authentication settings. For example, users who authenticate through passwords must comply with the password policy of your organization. See Modify the local authentication service security policy.

    The predefined roles for TanOS system users include:

    • tanadmin: Users with this role can access all TanOS console menus. It is useful to have more than one tanadmin user in case you forget the password for the initial tanadmin user that is created during Appliance setup.
    • tancopy: Users with this role can copy files to and from the /incoming and /outgoing directories on the Appliance.
    • tanuser: Users with this role can access only status menus in the TanOS console.

    For details and procedures, see Reference: User Administration menu.

  2. Verify that the predefined tanremote user account is present if you configured an Integrated Dell Remote Access Controller (iDRAC) interface on the physical Appliance. The account provides remote access to the iDRAC virtual console. This is useful for diagnosing hardware and network interface issues if the TanOS system becomes unavailable. For details and procedures, see Manage the iDRAC interface.

Backup overview

TanOS contains the options to perform core and comprehensive backups. physical Tanium Appliances and virtual Tanium Appliances with inactive partitions also have the option to back up the active partition to the inactive partition. On virtual Tanium Appliances and cloud-based Tanium Appliances, you can also take a snapshot of the appliance image. For core and comprehensive backups, you can schedule automatic backups or perform a manual backup. The following sections describe the available options:

Configure and run automatic backups

Use TanOS to schedule automatic backups for the Tanium Appliance. Through TanOS, you can schedule core and comprehensive backups.

After you perform the initial setup for an appliance, a core backup is scheduled by default. The TanOS health check reports an error that automatic backups cannot complete until you set up an encryption key. To remove the error from the health check, you can either add an encryption key or disable the scheduled core backup. For more information on the TanOS health check, see Run the Health Check.

The general process to set up an automatic backup includes the following steps:

  1. Add an encryption key for all backups.
  2. Configure a core or comprehensive backup.
  3. Test the automatic backup.
  4. Schedule the automatic backup.

When you schedule an automatic backup, TanOS prompts you to select a remote host to which to save the recovery bundle. TanOS also saves the recovery bundle to the /outgoing directory, where you can use SFTP to download the recovery bundle.

Add encryption key for the backups

Encrypt all core and comprehensive backup recovery bundles with a key pair. Encryption is required for both automatic and manual backups.

  1. Use OpenSSL to generate a public/private key pair in a PEM file. Enter a passphrase when prompted.

    openssl genrsa -aes256 -out ssl-pvk.pem 3072

  2. (FIPS mode only) For customers that need to decrypt backup bundles on a FIPS-enabled appliance, encrypt your backup key using the following command:

    openssl pkcs8 -topk8 -in OLD_FILENAME.pem -v2 aes-256-cbc -out NEW_FILENAME.pem'

  3. Extract the public key from the PEM file. Enter your passphrase when prompted.

    openssl pkey -in ssl-pvk.pem -pubout -out ssl-pub.pem

  4. Extract the identifier for the public key. This identifier is visible in the backup file and can be useful to find a particular public key.

    openssl pkey -pubin -in ssl-pub.pem -outform DER | openssl dgst -sha1

  5. Copy the contents of the ssl-pub.pem file (the public key) to the clipboard.

  6. Sign in to the TanOS console as a user with the tanadmin role.

  7. Enter B to go to the Appliance Maintenance menu. ClosedView screen

  8. Enter 1 to go to the Backup menu. ClosedView screen

  9. Enter E, paste the public key from the clipboard, and press Ctrl-D. ClosedView screen

  10. Press Enter to go to the Backup menu.

  11. To test the encryption, perform a manual core backup using the steps described in Perform a core or comprehensive backup.

    1. After the backup completes, download the recovery bundle. Note that you are not prompted to set a password.
    2. Extract the recovery bundle. The folder contains a README.txt file that describes how to decrypt the recovery bundle.

Configure an automatic backup

In a new installation with a Tanium role installed, an automatic core backup is scheduled to run nightly at 2:01 AM UTC. You can edit the backup, disable the backup, or configure an automatic comprehensive backup.

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B to go to the Appliance Maintenance menu. ClosedView screen
  3. Enter 1 to go to the Backup menu. ClosedView screen
  4. Enter C to configure an automatic backup. ClosedView screen
  5. Select the type of backup to configure:
    • To configure a core backup, enter N.
    • To configure a comprehensive backup, enter F.
  6. Follow the prompts to enable the backup and to specify file transfer options.
    • Automatic backups always save the recovery bundles to the /outgoing directory for download with SFTP.
    • You can specify a username and IP address for a destination server to reach with secure copy protocol (SCP). If you set up a file transfer with SCP, copy the public SSH key of the TanOS user that you are using to configure and run the backup to the remote user's ~/.ssh/authorized_keys file on the remote system. Ensure proper privileges on the remote system; you may need to run CHMOD 600 on the ~/.ssh/authorized_keys file.
      To locate a TanOS user's public SSH key to copy and store in the remote user's authorized keys file, sign in to the TanOS menu and navigate to User Administration C > TanOS U > user # > Key Pair P. The public SSH key is presented for copying.

      The backup files are stored on the remote system at /home/<remote user>/ .

      For information on managing TanOS user SSH keys, see Manage SSH keys.

Test an automatic backup

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B to go to the Appliance Maintenance menu. ClosedView screen
  3. Enter 1 to go to the Backup menu. ClosedView screen
  4. Enter A to go to the Run Now menu. ClosedView screen
  5. Select the type of backup to run.
  6. Verify the backup settings and enter Yes to run the backup.
  7. Verify the backup completes successfully. ClosedView screen
    • If the backup exports the recovery bundle to a remote server with SCP, sign in to the remote server and verify the recovery bundle exists.
    • Extract the recovery bundle. The folder contains a README.txt file that describes how to decrypt the recovery bundle.

Schedule an automatic backup

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B to go to the Appliance Maintenance menu. ClosedView screen
  3. Enter 1 to go to the Backup menu. ClosedView screen
  4. Enter S to go to the Schedule menu. ClosedView screen
  5. Select the type of backup to view the schedule settings. The schedule settings include the current settings and the pending settings. ClosedView screen
    • Enter 1 to disable the backup.
    • Enter 2 to enable the backup.
    • Enter 4 to enter the days of the month to run the backup. You can enter a date range or comma-separated days.
    • Enter 5 to enter the days of the week to run the backup. You can enter a range or comma-separated values.
    • Enter 6 to select the time to run the backup. Enter the hours and minutes in UTC time.
    • To confirm the pending settings, enter 7 to activate the settings. The active settings update to match the pending settings.

    6. If you enter R and do not activate the settings, the changes do not save.

View log files after a backup

If a backup fails, you can review the TanOS log file for additional information.

  1. Sign in to the TanOS console as a user with the tanadmin role.

  2. Enter 3 to go to the Tanium Support Menu menu.
  3. Enter 1 to go to the Tanium Log Files menu.
  4. Enter 1 to go to the TanOS Appliance menu.
  5. Enter 1 to go to the TanOS Log menu.
  6. Enter V to view the log file.

Configure and run manual backups

You must encrypt all backups with a key pair. Encryption is required for both automatic and manual backups. For steps on how to set up encryption, see Add encryption key for the backups.

Perform a partition sync (physical Tanium Appliance and virtual Tanium Appliance only)

In an array, start the partition sync on the primary appliance only. The array automatically performs the sync on all other servers in the array.

A virtual Tanium Appliance contains only one partition by default. Appliances with only one partition do not contain the option to perform a partition sync. You can add a secondary partition to perform a partition sync.

To protect data consistency, the partition sync job disables (shuts down) the Tanium Server, Tanium database server, and other related services for the duration of the partition sync. Make sure that manually performing a partition sync does not disrupt solution processes.

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B to go to the Appliance Maintenance menu. ClosedView screen
  3. Enter 1 to go to the Backup menu. ClosedView screen
  4. Enter P to go to the Partition Sync menu. ClosedView screen
  5. Follow the prompts to complete the backup.
  6. After the backup completes, press Enter to go to the Backup menu.

For information on how to change the active partition to the inactive partition, see Change the active partition.

Perform a core or comprehensive backup

Complete the following steps to perform a manual backup of the Tanium Appliance:

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B to go to the Appliance Maintenance menu. ClosedView screen
  3. Enter 1 to go to the Backup menu.
  4. Use the menu to create a backup:
    • Enter N to go to the Backup off-box Core menu.
    • Enter F to go to the Backup off-box Comprehensive menu.
  5. Follow the prompts to confirm the backup and to specify file transfer options. You can save the backup file to the /outgoing directory for download with SFTP, and you can specify a username and IP address for a destination server that can be reached with secure copy protocol (SCP). ClosedView screen
  6. After the backup completes, press Enter to go to the Backup menu.

If a backup fails, you can review the TanOS log file for additional information.

  1. Sign in to the TanOS console as a user with the tanadmin role.

  2. Enter 3 to go to the Tanium Support Menu menu.
  3. Enter 1 to go to the Tanium Log Files menu.
  4. Enter 1 to go to the TanOS Appliance menu.
  5. Enter 1 to go to the TanOS Log menu.
  6. Enter V to view the log file.

Manage Tanium database backups

Beginning with TanOS 1.6.3, Tanium database backups are included with core backups and comprehensive backups. TanOS contains the option to manage Tanium database backups produced prior to TanOS 1.6.3.

To select a specific backup from the last 7 days, navigate to the List Tanium Database Backups menu.

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B to go to the Appliance Maintenance menu. ClosedView screen
  3. Enter 1 to go to the Backup menu.
  4. Enter L to list the Tanium database backups.
  5. Follow the prompts to export, rename, or delete the backup.

Configure alerts

TanOS can send alerts to a syslog server or to an email recipient. For optimal results, configure an SMTP email recipient. If the syslog server fails, the SMTP recipient receives a failure notification every 15 minutes until the failure is resolved or syslog forwarding is disabled.

Severity level is a global setting that applies to both Syslog and SMTP alerts.

Configure alerts

Use the Configure Alerts menu to set the alert severity threshold to info, warn, or error.

  • Info: Includes all alerts
  • Warn: Includes all error and warning alerts
  • Error: Includes error alerts
  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B to go to the Appliance Maintenance menu. ClosedView screen
  3. Enter 2 to go to the Alerting menu. ClosedView screen
  4. Enter 3 to go to the Configure Alerts menu. ClosedView screen
  5. Use the menu to set a severity level and enable/disable alerting.

Configure syslog destination

The syslog alert configuration is separate from the syslog configuration in the Appliance Configuration menu. This configuration sends alerts for the alert threshold severity. The syslog configuration in the Appliance Configuration menu sends all logs.

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B to go to the Appliance Maintenance menu. ClosedView screen
  3. Enter 2 to go to the Alerting menu. ClosedView screen
  4. Enter 1 to go to the Configure Syslog Destination menu. ClosedView screen
  5. Enter 2 and follow the prompts to configure a syslog destination. ClosedView screen
  6. Enter 1 to enable syslog alerts. The Configure Syslog Destination menu updates to show the current status. ClosedView screen
  7. Enter 3 to send a test alert to the syslog server.

The test alert appears in the syslog server logs.

Configure SMTP destination

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B to go to the Appliance Maintenance menu. ClosedView screen
  3. Enter 2 to go to the Alerting menu. ClosedView screen
  4. Enter 2 to go to the Configure SMTP Destination menu. ClosedView screen
  5. Enter 2 and follow the prompts to configure the SMTP destination. ClosedView screen
  6. Enter 1 to enable SMTP alerts. The Configure SMTP Destination menu updates to show the current status. ClosedView screen
  7. Enter 3 to send a test alert to the mail recipient.

Upgrade TanOS

See Upgrade TanOS.

Request a shell access key

You can request OS shell access to examine OS processes and files written to the file system. See Examine Tanium and TanOS files.

Clean up generated files

Clean directories to clear up disk space or clear logs to make it easier to work with new entries in the log viewer.

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B to go to the Appliance Maintenance menu. ClosedView screen
  3. Enter A to go to the Clean directories menu. ClosedView screen
  4. Use the menu to delete files that have been generated in the SFTP /incoming and /outgoing directories, core dump files, application logs, and so on.

Reboot or shut down

Tasks that you complete with TanOS menus typically do not require you to reboot the system. A reboot might be required during troubleshooting workflows.

Shutdown turns off the system and powers down the appliance.

On a physical Tanium Appliance, you must have physical or iDRAC access to the appliance to power it on. Do not perform a system shutdown unless you are prepared to power the appliance back on.

Reboot

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B to go to the Appliance Maintenance menu. ClosedView screen
  3. Enter B to go to the Reboot/Shutdown menu. ClosedView screen
  4. Enter 1 to go to the Reboot menu. ClosedView screen
  5. Follow the prompts to reboot the appliance.

Shut down

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B to go to the Appliance Maintenance menu. ClosedView screen
  3. Enter B to go to the Reboot/Shutdown menu. ClosedView screen
  4. Enter 2 to go to the Shutdown menu. ClosedView screen
  5. Follow the prompts to shut down the appliance.

Exit maintenance mode

Some maintenance procedures that you perform with TanOS menus prompt you to enter maintenance mode to ensure Tanium services are not affected by the maintenance operation. When the operation completes, exit maintenance mode to resume normal operations.

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B to go to the Appliance Maintenance menu. ClosedView screen
  3. Enter C to go to the Maintenance Mode menu.
  4. Enter 1 to clear any maintenance actions.

Enable alternate partitions (physical Tanium Appliance and virtual Tanium Appliance)

A virtual Tanium Appliance contains only one partition set by default. You can add an alternate (inactive) partition set to use as a backup partition.

Enabling an alternate partition allows you to perform a partition sync, as described in Perform a partition sync (physical Tanium Appliance and virtual Tanium Appliance only).

On a virtual Tanium Appliance, the option to enable partitions only appears if you have a single partition.

Enabling an alternate partition is a long-running operation. This operation performs an initial partition sync, which may take a long time depending on the configuration of your storage subsystem.

  1. If needed, modify the virtual image to add disk storage.

    This action is not reversible. Storage that you add to the appliance is permanently allocated. Do not attempt to remove disk storage from an appliance, as the appliance becomes unusable.

  2. Sign in to the TanOS console as a user with the tanadmin role.

  3. Enter B to go to the Appliance Maintenance menu. ClosedView screen
  4. Enter E and follow the prompts to enable alternate partitions.
    If your virtual Tanium Appliance does not have enough space to enable the alternate partition, the TanOS console displays the minimum amount of space needed.

Disable alternate partitions (physical Tanium Appliance and virtual Tanium Appliance)

Use this option on virtual Tanium Appliances to remove the alternate (inactive) partition set. Use this option if you do not need the alternate partition. After you remove the alternate partition set, you can allocate the unused storage to the primary partition.

Do not disable the alternate partition set with the intent to reclaim disk storage. Disk storage on the appliance is permanently allocated. Do not attempt to remove disk storage from an appliance, as the appliance becomes unusable.

Disabling an alternate partition used for a partition sync removes the partition sync backup.

In TanOS 1.6.1 and later, virtual images contain only one partition set by default. If you upgrade from a previous version of TanOS, the existing partition configuration is preserved. This menu option only appears if your virtual Tanium Appliance has an alternate partition set.

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B to go to the Appliance Maintenance menu. ClosedView screen
  3. Enter D and follow the prompts to remove all alternate partitions. ClosedView screen

Use the Increase storage option to reallocate the storage from the deleted partition set.

Increase storage

On cloud-based Tanium Appliances and virtual Tanium Appliances, you can add a disk to the virtual image or increase the size of the existing virtual disk to increase the amount of storage that is available to TanOS.

This action is not reversible. Storage that you add to the appliance is permanently allocated. Do not attempt to remove disk storage from an appliance, as the appliance becomes unusable.

  1. Modify the virtual image to add a disk or increase the size of the existing virtual disk.
  2. Sign in to the TanOS console as a user with the tanadmin role.
  3. Enter B to go to the Appliance Maintenance menu. ClosedView screen
  4. Enter I to go to the Increase Storage menu. ClosedView screen
  5. Follow the prompts to add the disk storage. ClosedView screen

    If your virtual Tanium Appliance has an inactive partition set, any new storage is evenly allocated across the active (/OPT) and inactive (/ALTOPT) partitions.

Manage OS services

Use this menu to start, stop, restart, enable, and view status details for the network time protocol daemon (chronyd) and SSH daemon (sshd) services.

  1. Sign in to the TanOS console as a user with the tanadmin role.

  2. Enter B to go to the Appliance Maintenance menu. ClosedView screen

  3. Enter S to go to the OS Services menu.ClosedView screen

  4. Select a service to open the Service Control menu:

    • To manage chronyd, enter 1.ClosedView screen

    • To manage sshd, enter 2.ClosedView screen

  5. Use the menu to select an action to start, stop, restart, enable, or view status details for the service.

  6. Follow the prompts to perform the action.

Perform advanced maintenance tasks

Consult with Tanium Support before you use advanced options. For more information, see Support for Tanium Appliances.

Install a firmware update

When you Run the Health Check , you might see messages alerting you to perform a firmware update.

Use the Advanced Maintenance menu to stage and apply firmware updates. On a physical Tanium Appliance, the updates include iDRAC firmware updates, PERC firmware updates, NIC firmware updates, and BIOS firmware updates.

Updating a firmware update is a major task. The process can take from 10-30 minutes, depending on model. Allow the firmware update to complete before attempting any other tasks with the appliance. Do NOT manually power off or reboot the appliance.

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B to go to the Appliance Maintenance menu. ClosedView screen
  3. Enter X to go to the Advanced Maintenance menu. ClosedView screen
  4. Enter 1 to go to the Firmware Update menu. ClosedView screen
  5. Follow the prompts to update the iDRAC, PERC, NIC, and BIOS firmware. ClosedView screen

Perform a TanOS reset

The Appliance Reset menu allows you to perform a software reset which erases the Tanium application software and data.

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B to go to the Appliance Maintenance menu. ClosedView screen
  3. Enter X to go to the Advanced Maintenance menu. ClosedView screen
  4. Enter 2 to go to the Reset menu. ClosedView screen
  5. Enter 1 to perform a software reset.
  6. Follow the prompts to initiate the reset. ClosedView screen

Re-install ACLs

If you experience issues copying to or from the tancopy/incoming or /outgoing directories, you can use this menu to reapply the access control lists (ACL) for those directories.

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B to go to the Appliance Maintenance menu. ClosedView screen
  3. Enter X to go to the Advanced Maintenance menu. ClosedView screen
  4. Enter 3 to reapply the ACLs.

View the TanOS partition sync log

This menu only appears if the appliance contains an active partition and an inactive partition. This menu is not available on cloud-based Tanium Appliances.

  1. Sign in to the TanOS console as a user with the tanadmin role.
  2. Enter B to go to the Appliance Maintenance menu. ClosedView screen
  3. Enter X to go to the Advanced Maintenance menu. ClosedView screen
  4. Enter 5 to view the TanOS partition sync log file.