U of O Beowulf Building Notes

Nodes

• Get copy of Jack Radjewski's Beowulf Super Computer HOWTO
• Partition WDAC11200 drive with

  /       850M
  swap1   128M
  swap2   128M
  

• Full install of Redhat 5.0
• Install SMC Etherpower II driver
• Run 'netcfg' in X to setup network configuration or edit /etc/HOSTNAME /etc/sysconfig/network and /etc/sysconfig/network-scripts/ifcfg-eth0 to include the following lines:
  

  /etc/sysconfig/network
  NETWORKING=yes
  FORWARD_IPV4=no
  HOSTNAME=n001
  GATEWAYDEV=eth0
  GATEWAY=10.0.21.1
  
  /etc/sysconfig/network-scripts/ifcfg-eth0
  DEVICE=eth0
  USERCTL=no
  ONBOOT=yes
  BOOTPROTO=none
  BROADCAST=10.0.20.255
  NETWORK=10.0.20.0
  NETMASK=255.255.255.0
  IPADDR=10.0.20.2
  

• Modify /etc/fstab to NFS mount the /usr filesystem from the server:

  dolly.uoregon.edu:/usr /usr   nfs   ro           0 0
  dolly.uoregon.edu:/home /home nfs   defaults     0 0
  

• Edit /etc/inetd.conf to comment out all services except in.rshd
• Put the following lines in /etc/hosts.allow:

  sshd: 10.0.20.1
  xntpd: 10.0.20.1
  rdistd: 10.0.20.1
  pvmd: 10.0.20.1
  in.rshd: 10.0.20.1
  

  /etc/hosts.deny should be ALL: ALL
• Install ssh client and server from RPM
• Delete all but 2 mingettys from /etc/inittab
• Build new kernel with desired options. Patch with 'patch -p0 < location/to/patch/patch_name'
• Add root alias in /etc/aliases for sendmail
root: hak@darkwing, joelja@darkwing, joe@oregon
• Remove annoying rm, cp, mv aliases in /root/.bashrc
• Install xntp from RPM. Then run xntpconf to configure xntp.
• Now that the install is the way you want it, clone the master drive and copy it to all the other drives.
• Edit /etc/hosts and /root/.rhosts to include root logins and a host entry for the new node.

Server

• edit /etc/hosts to include nodes
• build and configure rdist to copy /etc/passwd and shadow to nodes
• install sshd and ssh-clients
• Install alpha-like sendmail config
• Setup rdist with the following config:

  #
  #       distfile file to rdist files from 
  #       dolly.uoregon.edu --> clones @10.0.20.*
  #
  #       created by hak@oregon 980115 
  #       (modified from nemeth, et. al p.389)
  #
  
          HOSTS = ( 10.0.20.2 10.0.20.3 10.0.20.4 10.0.20.5 10.0.20.6 10.0.20.7 10
  .0.20.8 10.0.20.9 10.0.20.10 )
          FILES = ( /etc/passwd /etc/shadow )
  
          all:
          ${FILES} -> ${HOSTS}
  #notify root@dolly.uoregon.edu ;
  

• install PVM kernel patches
• make sure there isn't a route added for eth1 in /etc/sysconfig/static-routes. This will hose the private network connection to the nodes.
  eth0 net 128.223.32.0 netmask 255.255.254.0 gw 128.223.32.1

• /etc/sysconfig/network
  NETWORKING=yes
  FORWARD_IPV4=no
  HOSTNAME=dolly
  GATEWAYDEV=
  GATEWAY=128.223.32.1
  
  /etc/sysconfig/network-scripts/ifcfg-eth0
  DEVICE=eth0
  USERCTL=no
  ONBOOT=yes
  BOOTPROTO=none
  BROADCAST=128.223.33.255
  NETWORK=128.223.32.0
  NETMASK=255.255.254.0
  IPADDR=128.223.32.248
  
  /etc/sysconfig/network-scripts/ifcfg-eth1
  DEVICE=eth1
  USERCTL=no
  ONBOOT=yes
  BOOTPROTO=none
  BROADCAST=10.0.20.255
  NETWORK=10.0.20.0
  NETMASK=255.255.255.0
  IPADDR=10.0.20.1
  

• add the following to /etc/exports and '/etc/rc.d/rc3.d/S46nfs restart'

  /usr            10.0.20.2/255.255.255.0(ro,secure) # usr filesystem for clones
  /home           10.0.20.2/255.255.255.0(rw,root_squash,secure) # home filesystem for clones
  

• since the kernel on the server is smp the precompiled smc epic module has unresolved symbols. to fix this grab the source, epic100.c and compile with the following string.

   
  gcc -D__KERNEL__ -I/usr/src/linux-2.0.31/include -Wall -Wstrict-prototypes -O2 -fomit-frame-pointer -fno-strength-reduce -D__SMP__ -pipe -m486 -malign-loops=2 -malign-jumps=2 -malign-functions=2 -DCPU=686 -D__SMP__ -DMODULE -DMODVERSIONS -include /usr/src/linux-2.0.31/inclu de/linux/modversions.h  -c -o epic100.o epic100.c 
  

• Also the server has 128 mb of ram, only 64 mb is detected automaticaly, so add the command:

  append mem="128M"
  

to each kernel configuration in the lilo.conf and re-run lilo. reboot for changes to take affect.

• install fortran compiler (make sure to rpm -e the installed gcc and dependencies:
  EGCS compiler
• The sendmail configuration forwards all email to darkwing except for email destined to root. Systems didn't want to receive email for dolly since they don't administer it. We deliver email for root locally and then use the aliases file to deliver it to Joe Hans and Joel. The config files to handle dolly's email are on darkwing and dolly.