[GFS] IPMI_Fencing Config
What is IPMI?
Intelligent Platform Management Interface
Advantages
Low cost.
Available on most recent server systems.
Disadvantages
There is no way to retain power to the IPMI BMC if power to the host is lost. If power is lost to the host, fencing will fail. This will delay cluster recovery. It is recommended to either have a second fencing device (external power switch) or a server with multiple power supplies when using IPMI over LAN for fencing.
Some IPMI implementations share an ethernet interface with the host operating system. If this interface is the same one used for the cluster communication, a loss of connectivity to this port will cause a need for fencing but also cause fencing to fail, delaying cluster recovery. It is recommended that the ethernet interface the BMC uses be reserved exclusively for BMC use; that is, the host operating system should not use the BMC’s NIC.
Prerequisites
You will need the OpenIPMI packages and the ipmitool utility in order to use this mini-howto. On Red Hat Enterprise Linux 5 and CentOS 5, the packages you need are:
OpenIPMI
OpenIPMI-tools
You will also need some free IP addresses which you assign to IPMI devices.
This mini-HOWTO is for machines which have not had IPMI-over-LAN set up before.
Basic Configuration
See if you can start ipmi:
[root@et-virt05 ~]# service ipmi start
Starting ipmi drivers: [ OK ]
If it fails (FAILED), go have a beer or something, because the rest of this howto will not help you. If it succeeds, you should see something like this (may differ by hardware) in dmesg output:
ipmi message handler version 39.1
IPMI System Interface driver.
ipmi_si: Trying SMBIOS-specified kcs state machine at i/o address 0xca8, slave address 0x20, irq 0
ipmi: Found new BMC (man_id: 0x0002a2, prod_id: 0x0000, dev_id: 0x20)
IPMI kcs interface initialized
ipmi device interface
If that works, proceed to configure IPMI for your hardware.
Dell, Intel, and IBM machines (typical)
HP machines with iLO MP
Also, pay attention to the known bugs:
Known Bugs
IPMI on Dell/IBM machines
LAN/LAN+ Configuration
We need to configure the LAN device for network access. First, take a look at the configuration:
[root@et-virt05 ~]# ”’ipmitool lan print 1”’
Set in Progress : Set Complete
Auth Type Support : NONE MD2 MD5 PASSWORD
Auth Type Enable : Callback : MD2 MD5
: User : MD2 MD5
: Operator : MD2 MD5
: Admin : MD2 MD5
: OEM : MD2 MD5
IP Address Source : Static Address
IP Address : 0.0.0.0 <— need to set this
Subnet Mask : 0.0.0.0 <— also need to set this
MAC Address : 00:13:72:50:05:f0
SNMP Community String : public
IP Header : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
Default Gateway IP : 0.0.0.0 <— Might need to set this
Default Gateway MAC : 00:00:00:00:00:00
Backup Gateway IP : 0.0.0.0
Backup Gateway MAC : 00:00:00:00:00:00
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
Cipher Suite Priv Max : Not Available
Now we need to set the IP address, netmask, gateway, etc. of the LAN interface. Things to remember:
The IP address must not be used anywhere else on the network.
If you are using DHCP for the host, you may usually not use DHCP for IPMI-over-LAN and vice-versa. This is because of the fact that some machines share a MAC address between the IPMI-over-LAN device and the device used by the operating system. I used static addresses on the IPMI devices.
[root@et-virt05 ~]# ipmitool lan set 1 ipsrc static
[root@et-virt05 ~]# ipmitool lan set 1 ipaddr 10.10.10.52
Setting LAN IP Address to 10.10.10.52
[root@et-virt05 ~]# ipmitool lan set 1 netmask 255.255.255.0
Setting LAN Subnet Mask to 255.255.255.0
[root@et-virt05 ~]# ipmitool lan set 1 arp respond on
[root@et-virt05 ~]# ipmitool lan set 1 arp generate on
[root@et-virt05 ~]# ipmitool lan set 1 arp interval 5
Finally, we have to enable the interface:
[root@et-virt05 ~]# ipmitool lan set 1 user
[root@et-virt05 ~]# ipmitool lan set 1 access on
User Configuration
First, we need to list the users and get the user ID of the administrative user for IPMI:
[root@et-virt05 ~]# ipmitool user list
ID Name Callin Link Auth IPMI Msg Channel Priv Limit
2 root true true true ADMINISTRATOR
If you don’t already know the password for the ‘root’ user, you can set it from the command line as well. This should NOT be the same as the root user password for the system). Note the 2 in the command line – this matches the administrator / root account from above:
[root@et-virt05 ~]# ipmitool user set password 2
Password for user 2: mypassword
Password for user 2: mypassword
Preliminary Testing
The easiest thing to check is the power status. Note – you CAN NOT issue IPMI-over-lan commands from the same machine. That is, you must perform the following test from a different machine than the one you have just configured.
[root@et-virt06 yum]# ipmitool -H 10.10.10.52 -I lan -U root -P mypassword chassis power status
Chassis Power is on
Fencing Configuration
Now, we have a working IPMI configuration, we need to add it to cluster.conf. Here is what the above configuration would look like, given that et-virt05 had an IPMI lan interface set to 10.10.10.52 and et-virt06 had its IPMI lan interface set to 10.10.10.62:
…
<clusternodes>
<clusternode name=”et-virt05″ nodeid=”1″ votes=”1″>
<fence>
<method name=”1″>
<device name=”virt05-ipmi”/>
</method>
</fence>
</clusternode>
<clusternode name=”et-virt06″ nodeid=”3″ votes=”1″>
<fence>
<method name=”1″>
<device name=”virt06-ipmi”/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice agent=”fence_ipmilan” ipaddr=”10.10.10.52″ login=”root” name=”virt05-ipmi” passwd=”mypassword”/>
<fencedevice agent=”fence_ipmilan” ipaddr=”10.10.10.62″ login=”root” name=”virt06-ipmi” passwd=”mypassword”/>
</fencedevices>
…
IPMI with HP ILO MP
The ILO MP (Integrated Lights-Out Management Processor), which is present on some HP Integrity servers, has the ability to handle IPMI-over-LAN requests. Users that wish to configure fencing for machines with ILO MP will therefore want to use the fence_ipmilan agent. Note that the fence_ilo agent will not work with the ILO MP interface.
The first step is to enable IPMI from with ILO MP. You will need to login to the ILO MP console in order to set this option. Once you are connected to the ILO MP console, enter the ‘Command Menu’ (CM). You can then enable IPMI by using the following command:
MP:CM> SA -lanipmi e
New Set Access Configuration (* modified values):
Remote/Modem : OS SESSION
Telnet : Enabled
Web SSL : Enabled
SSH : Enabled
* IPMI over LAN : Enabled
Warning: enabling IPMI over LAN will allow an anonymous user login
(an IPMI user with null name and password). To secure access please
set the password of the null user using an IPMI tool.
Confirm? (Y/[N]): Y
-> Set Access Configuration has been updated.
-> Command successful.
Check to see that you can connect to your ILO MP device via IPMI using the ipmitool.
ipmitool -I lan -P”” -H <hostname> user list
Password:
ID Name Callin Link Auth IPMI Msg Channel Priv Limit
1 true false true ADMINISTRATOR
Note that the above command passes a null password (-P “”) since initially the password is set to null. To set the password, use ipmitool with the ‘user set password’ command.
ipmitool -I lan -P “” -H <hostname> user set password 1 <password>
This command will set the password for userid “1”, which is the only IPMI user. Note that userid “1” does not have an associated username. This is a limitation of the ILO MP device and it is not possible to set a username for userid “1”. This is very important. The IPMI BMC on a ILO MP device will not recognize usernames, and therefore you will never want to pass a username to either ipmitool or fence_ipmilan.
Fencing Configuration
Configuring your cluster to use fence_ipmilan with an ILO MP device is the same as described above with one small exception. As mentioned, you will not want to define a username since the ILO MP device has no notion of usernames. Therefore, when setting up your fencedevice in the cluster.conf file, do not pass a “login=” parameter to the fence agent.
…
<clusternodes>
<clusternode name=”mynode-01″ nodeid=”1″ votes=”1″>
<fence>
<method name=”1″>
<device name=”ilo-mp-ipmi”/>
</method>
</fence>
</clusternode>
<clusternode name=”mynode-02″ nodeid=”2″ votes=”1″>
<fence>
<method name=”1″>
<device name=”ilo-mp-ipmi”/>
</method>
</fence>
</clusternode>
</clusternodes>
<fencedevices>
<fencedevice agent=”fence_ipmilan” ipaddr=”10.0.0.1″ name=”ilo-mp-ipmi” passwd=”mypassword”/>
<fencedevice agent=”fence_ipmilan” ipaddr=”10.0.0.2″ name=”ilo-mp-ipmi” passwd=”mypassword”/>
</fencedevices>
…
Known Bugs
The IPMI Specification, Version 2.0, Section 28.3 states:
[3:0] – chassis control
0h = power down. Force system into soft off (S4/S45) state. This is for
‘emergency’ management power down actions. The command
does not initiate a clean shut-down of the operating system prior to
powering down the system.
The source code for ipmitool correctly passes 0x30 to the BMC. However, most BMCs seem to ignore the fact that this operation is explicitly not a clean shutdown and generate an ACPI power-off event. This causes the operating system to initiate a clean shut down which is precisely what we do not want. Because of this, it is important to disable operating system registration of ACPI. For most users, simply stopping acpid will be sufficient:
[root@et-virt05 ~]# chkconfig –level 2345 acpid off
[root@ayanami lhh]# /sbin/service acpid stop
Stopping acpi daemon: [ OK ]
If power off is not immediate during testing of IPMI, it may be necessary to disable ACPI entirely during boot.
