Shutdown VM’s on ESXi 7.0 using NUT on pfSense

Summary

Utilizing a UPS for your network devices and servers not only keeps your system up and running when there is an interruption to the utility power but it also protects you from unnecessary reboots during a temporary power outage. However, unless you have a larger backup system, the battery will eventually drain and your system will go down. To preserve power and keep critical services running longer, or to shutdown servers before the UPS battery drains, a plan to shutdown servers is necessary.

I use pfSense as my main router and a UPS to protect my network and servers from such cases and wanted an automated way to shutdown my servers when the UPS got too low. Since I use NUT (Network UPS Tool) to monitor my UPS in pfSense, I utilized the advanced settings in the UPS Service to provide clients a way to monitor the UPS. I installed a VIB on my ESXi VMware host to monitor the UPS status and react accordingly when there is a UPS alert. This post assumes you have the NUT UPS package installed on pfSense, and pfSense is currently monitoring an attached UPS and working properly. The following describes how I got it to work.

Setup NUT server on pfSense…

Setup

First thing to do is set up the UPS Service to to allow other devices to monitor the UPS attached to pfSense. In the pfSense GUI go to the services tab, open the UPS Service and under the UPS Settings tab click Display Advanced.

In the box labeled “Extra Arguments to driver (optional)” box add the following lines:

ignorelb
override.battery.charge.warning = 25
override.battery.charge.low = 20

This will ignore the UPS default “low battery” signal and instead go by the actual battery %. The other two lines will warn of low battery at 25% and send the low battery signal at 20%. The low battery signal will trigger the shutdown. You can adjust the values according to your system but this is a typical middle ground to start with.

Next, in the box labeled “Additional configuration lines for upsmon.conf” add the following lines:

RUN_AS_USER root
NOTIFYFLAG NOCOMM     SYSLOG 
NOTIFYFLAG COMMOK     SYSLOG 
NOTIFYFLAG COMMBAD    SYSLOG 
POLLFREQ 60 
POLLFREQALERT 60 
DEADTIME 180

This entry is optional but will ensure the NUT monitor service runs as root. The other options are meant to suppress the notifications sent to your terminal and reduce the amount of time it polls and alerts; they will still be logged in your system logs to help with debugging if necessary.

Next, in the box labeled “Additional configuration for ups.conf” add the following lines:

user = root

This will ensure the NUT service itself runs as root.

Next, in the box labeled “Additional configuration lines for upsd.conf” add the following lines:

LISTEN pfsense.ip.address.here

Change “pfsense.ip.address.here” to the IP address of your pfSense router or network interface you want to broadcast the NUT server on. This will allow clients to monitor the status of the UPS through the UPS Daemon running on pfSense by connecting via the IP address of pfSense on port 3493.

Next, in the box labeled “Additional configuration lines for upsd.users” add the following lines:

[ups_remote_client]
password = P@ssw0rd
upsmon slave

This is what the client, or in this case, the ESXi host will use to connect to the UPS Daemon running on pfSense. [ups_remote_client] can be whatever name you want to choose to identify the connection; the password can be whatever you choose for the client credentials to connect to the daemon.

Finally, restart the UPS Service to enable the updated configuration.

Test pfSense Configuration

SSH into the pfSense and run upsc UPS_name@localhost command, where UPS_name is the name on the UPS Status page. If the current status is displayed then the NUT server is setup correctly.

Prepare the ESXi 7.0 host…

On the ESXi host, a VIB will need to be uploaded and installed. In order to install the VIB the acceptance level of the host will need to be set to the Community acceptance level and enable SSH access to the ESXi host.

Host->Manage => Security & Users => Acceptance level->Edit settings = change to Community

Host => Actions->Services = Enable Secure Shell (SSH)

Download the module that will be copied to the ESXi host to install the VIB.

Alternately, you can get the file from the creator’s site. (You will need translator since the site is in French). Rene’s Diary There’s some extra info to read, as well.

Setup ESXi Host and Configuration…

Setup

Now that the ESXi host is ready for setup, Copy the NutClient-ESXi-2.8.0-2.3.0.i386.tar.gz file to the /tmp directory of the ESXi host. This can be accomplished using SCP from a Linux client, the WinSCP tool from a Windows client, or my favorite choice, FileZilla. SSH into the ESXi host as root or a user who has administrator rights and navigate to the /tmp directory and run the following commands to install the VIB. The process will take a few minutes. Once complete, the Installation Result will display and upon successful install you can exit the SSH session.

/tmp # tar -xzf NutClient-ESXi-2.8.0-2.3.0.i386.tar.gz
/tmp # sh upsmon-install.sh
Installation Result
   Message: Operation finished successfully.
   Reboot Required: false
   VIBs Installed: Margar_bootbank_upsmon_2.8.0-2.3.0
   VIBs Removed:
   VIBs Skipped:

Configuration

Go back to the ESXi GUI to configure the NUT client before the service is started. Actually, the service won’t start until the configuration is complete. Right click each option and select edit to configure the settings as needed to meet your requirements.

If the configuration variables do not appear in the administration interface after installation, issue the /etc/init.d/hostd restart command on the ESXi host in the SSH session. Be careful not to have any jobs in progress if you do this (taking snapshots, VMotion, etc.).

Host->Manage->System => Advanced settings

# in the search bubble type UserVars.Nut to display the options that can be configured

• UserVars.NutUpsName  : Name of the UPS on the NUT server (in the form inverter_name@server_name_or_ip). Multiple inverters can be entered separated by a space. There will be no system shutdown until the last inverter still standing gives the shutdown command.
• UserVars.NutUser  : Name of the NUT server connection account
• UserVars.NutPassword  : NUT server login account password
• UserVars.NutFinalDelay  : Seconds to wait after receiving low battery event to perform system shutdown
• UserVars.NutSendMail  : To be set to 1 so that the NUT client sends an e-mail at each important inverter event
• UserVars.NutMailTo  : Email address to send inverter events to
• UserVars.NutMinSupplies  : For multi-inverter systems. The number of UPSs that must be able to power the system before initiating a shutdown. This number must be less than or equal to the number of ups defined in UserVars.NutUpsName . If you violate this constraint, the client will not start. With only one inverter, leave the value at 1.

For the sake of following the examples used in this post, the following settings will be set. Be sure to set to the values used in your environment.

Note that each time these parameters are modified, it will be necessary to stop/start the service to take them into account. For a more in-depth configuration, see below for links to references.

Start Service

The NutClient service should be running but if its not then start the service and set the start and stop policy.

Host->Manage->Services = NutClient

Confirm Firewall

In the NutClient service row, the last item is a link to the firewall rule. Click the link to confirm the rule was loaded during the VIB install. If rule is not found then SSH into the ESXi host and run esxcli network firewall refresh command to load the firewall rule. Run esxcli network firewall ruleset list to list the firewall rules to confirm the NutServer rule is loaded and enabled.

The NUT client and server use port 3493/tcp, so additional configuration may have to be setup on your internal network or clients if you have a firewall configured. This may be on your client or across vlans.

Test ESXi Configuration

To test the configuration , SSH into the ESXi host and run the command /opt/nut/bin/upsc UPS_name@pfsense.ip.address.here , where UPS_name is the name on the UPS Status page. If the current status is displayed then the NUT client is setup correctly and monitoring the UPS status from the NUT server on pfSense.

When the ESXi host receives a low battery signal, and after the UserVars.NutFinalDelay value in seconds has elapsed, the VM’s on the ESXi host will shutdown (or suspend) in respect to the settings and order set on the ESXi host.

Host->Manage->System => Autostart

To do a manual test, SSH into the ESXi host and run /opt/nut/sbin/upsmon -c fsd command. The shutdown procedure is immediately started. This can be useful to confirm the shutdown procedure is as expected, and to estimate the time required for the server to shutdown.

Conclusion

With a bit of research I was effectively able to allow the ESXi host to monitor the UPS and shutdown active VM’s after a UPS alert. With the NUT server setup, this can be expanded to other clients that could benefit from monitoring the UPS by installing or configuring the NUT client service. Let me know if this was helpful or any correction you find to be useful for others.

References

NUT manual pages – User manual pages for all the items that pertain to NUT

Understand VMware Acceptance Levels – A good read to understand acceptance levels for Hosts and VIBs

Introduction to Network UPS Tools – A great reference with examples for a more in-depth configuration

Rene’s Diary – Source for the VIB with additional resources (in French will need to translate)

UPS management with NUT – the initial inspiration for this post and expanded to meet my needs