ATLASHN NETWORKING SETUP

This procedure explains how to configure the networking for the head node.

• Hardware – server with two network cards
• Services – running on headnode
• Condor collector/negotiator
• Xrootd redirector
• Ganglia Metad server
• Ganglia Gmond client
• Apache Web server

• NIC #1 yellow network address, with MAC registered in IHW
• NIC #2 yellow network address, virtual interface in IHW

1) Stop xrootd, cmsd and condor

/sbin/service xrootd stop
/sbin/service cmsd stop
/sbin/service condor stop

2) Edit the “service interface” (In this case: eth1): /etc/sysconfig/network-scripts/ifcfg-eth1

# Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
DEVICE=eth1
HWADDR=00:26:B9:4E:B8:AD
ONBOOT=no
HOTPLUG=no
BOOTPROTO=static
NETMASK=255.255.255.128
IPADDR=146.139.33.75
GATEWAY=146.139.33.1

3) Edit the primary interface (In this case: eth5) /etc/sysconfig/network-scripts/ifcfg-eth5

# Intel Corporation Unknown device 151c
DEVICE=eth5
BOOTPROTO=static
IPADDR=146.139.33.66
NETMASK=255.255.255.128
ONBOOT=yes
#ONBOOT=no
HWADDR=00:1b:21:d4:51:61
GATEWAY=146.139.33.1

4) Activate the service interface

/etc/sysconfig/network-scripts/ifup eth1

4b) Add the following to the routing table

146.139.33.0/25 dev eth5  proto kernel  scope link  src 146.139.33.66 
146.139.33.0/25 dev eth1  proto kernel  scope link  src 146.139.33.75 
192.168.122.0/24 dev virbr0  proto kernel  scope link  src 192.168.122.1 
169.254.0.0/16 dev eth1  scope link 
default via 146.139.33.1 dev eth5

(Note the virtual interfaces come from the installation of the SL5 virtualization software)

5) Remove routing reference for eth5

ip route delete 146.139.33.0/25 dev eth5  proto kernel  scope link  src 146.139.33.66 ; ip route delete default via 146.139.33.1 dev eth5;   ip route add default via 146.139.33.1 dev eth1

(Note there are 3 ip commands on one line, which is important if you connected to the machine via ssh)

6) Commands to test your handy work

ip route show
ip link show

7) Test how the routing looks

ip route get to 146.139.33.68

146.139.33.68 dev eth1 src 146.139.33.75 cache mtu 1500 advmss 1460 hoplimit 64

ip route get to 130.202.173.21

130.202.173.28 via 146.139.33.1 dev eth1 src 146.139.33.75 cache mtu 1500 advmss 1460 hoplimit 64

ip route get to 130.202.173.16

130.202.173.16 via 146.139.33.1 dev eth1 src 146.139.33.75 cache mtu 1500 advmss 1460 hoplimit 64

8) Check the contents of the working routing table

ip route show
146.139.33.0/25 dev eth1  proto kernel  scope link  src 146.139.33.75 
192.168.122.0/24 dev virbr0  proto kernel  scope link  src 192.168.122.1 
169.254.0.0/16 dev eth1  scope link 
default via 146.139.33.1 dev eth1

9) Check to see if a nameserver is reachable

ping 146.139.254.3
dig atlas28.hep.anl.gov

10) Modifications to xrootd configuration file: /etc/xrootd/xrootd-clustered.cfg must have the following lines

#redirector name
set head = atlashn.hep.anl.gov

11) Create the /atlas subdirectory and set the ownership to “xrootd:xrootd”

mkdir /atlas
chown xrootd:xrootd /atlas

12) Set the hostname in the /proc file system to be atlashn.hep.anl.gov and check your work

echo "atlashn.hep.anl.gov" > /proc/sys/kernel/hostname
hostname
uname -a

13) Open the firewall to ports 1094 and 1213 and condor ports by adding these lines to /etc/sysconfig/iptables

-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 1094 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 1213 -j ACCEPT
-A RH-Firewall-1-INPUT  -s 146.139.33.66 -m state --state ESTABLISHED,NEW -p tcp -m tcp -j ACCEPT
-A RH-Firewall-1-INPUT  -s 146.139.33.75 -m state --state ESTABLISHED,NEW -p tcp -m tcp -j ACCEPT
-A RH-Firewall-1-INPUT  -s 130.202.173.21 -m state --state ESTABLISHED,NEW -p tcp -m tcp -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 9000:9999 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 9000:9999 -j ACCEPT
-A RH-Firewall-1-INPUT -s  146.139.33.75 -m state --state ESTABLISHED,NEW -p tcp -m tcp -j ACCEPT
#

Then restart iptables

/sbin/service iptables restart

14) Restart xrootd/cmsd

/sbin/service xrootd start
/sbin/service cmsd start

15) Modifications to condor file

• Move /etc/condor/condor_config.local (with reference to condor_config.cluster) in place
• Move /etc/condor/condor_config.cluster in place, which should contain this line:

  CONDOR_HOST=atlashn.hep.anl.gov

• Move /etc/condor/condor_config.head in place

16) Create the file /etc/condor/condor_config.head.atlashn with the following two lines

# set the network interface for the primary interface
NETWORK_INTERFACE = 146.139.33.75

17) Condor Security: Since we are using a share secret file (PASSWORD) make sure root owns the file

ls -l /var/lib/condor/condor_credential

-rw——- 1 root condor 256 Jan 3 07:44 /var/lib/condor/condor_credential

18) start condor

/sbin/service condor start