\\
**[[cluster:0|Back]]**

===== IPoIB =====

Redoing our RHEL5.5 HP Proliant blade servers with CentOS 6.7 using [[cluster:144|Warewulf Golden Image]] provisioning.

Not quite there yet, but I'll document here how Infiniband was installed. These compute nodes are connect to a Voltaire interconnect, and aging quite a bit.

First install a vanilla basic server and load packages you need. Then add the packages for Infiniband.

<code>

# Install and reboot
yum groupinstall "Infiniband Support"
yum install infiniband-diags perftest qperf opensm
chkconfig rdma on

# for openhpc 
yum install inifinipath-psm

yum install opensm
chkconfig opensm on

yum install tcl tk
yum install infiniband-diags

shutdown -r now

# after reboot
lsmod | grep ib

# and the output (ipoib is the important one)

ib_ipoib               80391  0
ib_ucm                 12121  0
ib_uverbs              39106  6 rdma_ucm,ib_ucm
ib_umad                11802  8
ib_cm                  36996  3 ib_ipoib,ib_ucm,rdma_cm
mlx4_ib               137138  1
ib_sa                  24060  5 ib_ipoib,rdma_ucm,rdma_cm,ib_cm,mlx4_ib
ib_mad                 39811  4 ib_umad,ib_cm,mlx4_ib,ib_sa
ib_core                81507  11 ib_ipoib,rdma_ucm,ib_ucm,ib_uverbs,ib_umad,rdma_cm,ib_cm,iw_cm,mlx4_ib,ib_sa,ib_mad
ib_addr                 8304  3 rdma_ucm,rdma_cm,ib_core
ipv6                  335525  74 ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6,ib_ipoib,mlx4_ib,ib_addr
mlx4_core             226123  2 mlx4_ib,mlx4_en

</code>

Connect the cable(s) to you ports and check that the reboot discovered the ports.

<code>

# Check IB ports
for i in `ls /sys/class/infiniband/*/ports/*/state`; do echo $i; cat $i; done

# Your output may vary
/sys/class/infiniband/mlx4_0/ports/1/state
4: ACTIVE
/sys/class/infiniband/mlx4_0/ports/2/state
1: DOWN

# some test commands
ibhosts
iblinkinfo
ibstatus

</code>

Edit ''/etc/sysconfig/network-scripts/ifcfg-ib0'', something like

<code>

DEVICE=ib0
TYPE=InfiniBand
UUID=eac9f00a-245d-4c88-b56f-1bcb6e6ed933
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
HWADDR=80:00:00:48:FE:80:00:00:00:00:00:00:00:02:C9:03:00:07:3B:DD
CONNECTED_MODE=no
IPADDR=10.11.103.31
PREFIX=16
DEFROUTE=node
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME="System ib0"

# then start interface 
ifup ib0

# check the route, then mount /home on this interface

route

# Your output may vary

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.11.0.0       *               255.255.0.0     U     0      0        0 ib0
link-local      *               255.255.0.0     U     1002   0        0 eth0
link-local      *               255.255.0.0     U     1004   0        0 ib0
192.168.0.0     *               255.255.0.0     U     0      0        0 eth0
default         greentail       0.0.0.0         UG    0      0        0 eth0

# home from sharptail
10.11.103.42:/home     /home   nfs  defaults 0 0 


</code>

\\
**[[cluster:0|Back]]**