BWA (Business Warehouse Accelerator)
From Tecniq
This page is intended to provide help and tips on the Hardware and Operating Systems (SuSE Linux Enterprise Server) used in IBM's BWA solution. For help concerning the BWA software (TREX) try the SAP Developers Network and SAP notes.
Contents |
GPFS
Installing GPFS Hint
GPFS is a licensed product. You or your company must own a legitimate license before installing.
The installation for GPFS is pretty straight forward. However, on Linux you need to build, from source, the portability layer. The portability layer is a set of binaries that are compiled against the running kernel that a required for GPFS to function.You must ensure you have the following three packages installed.
- kernel-source
- gcc
- xorg-x11-devel
Installing of these three packages and their dependencies through yast will satisfy the building of the portability layer.
RSH or SSH?
GPFS nodes must all be able to login and copy files to each other as root without being prompted for a password. There are two methods that can be used to achieve this, these are RSH and SSH. Traditionally RSH has been used. RSH offers password free shell and file copy (rcp) services based on a trusted hosts that are listed in the file .rhosts that resides in a users home directory. RSH is a clear text protocol, encryption is not available. As RSH is clear text is can easily be sniffed and as the protocol is based on hostnames it can easily be spoofed, thus using RSH is not good security practice. SSH offers both remote shell and copy (scp) utilities based on SSL encryption. Each host creates a pair of keys, a private key and a public key. The public key of each node should be shared with all nodes in the cluster (including itself). When done all the nodes in the cluster will be able to login to each other password free as the desired user.
Creating a GPFS cluster with ssh
The default remote shell and copy tools when creating a cluster are rsh and rcp respectively. To use ssh and scp you must first ensure that all nodes can communicate password free as root using ssh. Once this is the case you create the cluster using the mmcrcluster command and use the -r and -R flags to specify the remote shell and remote copy programs. See the following example.
# mmcrcluster -n /var/mmfs/config/nodes.cluster -p bwablade01 -s bwablade10 -r `which ssh` -R `which scp` -C gpfsFS1 –A
In the example command a cluster is being created. The -n flag is followed by the file containing the node definitions, the -p specifies the primary server and the -s specifies the secondary server. The -r flag specifies the remote shell tool, backticks have been used with the which commands have to print the path for ssh. The -R flags specifies the remote copy tool, again backticks are used to print the path to scp. The -C flag is used to name the cluster, in this case "gpfsFS1", and finally the -A flags specifies that the GPFS daemons are started when the system is booted.
mmdsh
GPFS has its own distributed shell than can be to issue commands to all the nodes in the cluster. It uses the variable WCOLL (Working Collective), which should be the path to a text file that contains a list of the nodes in the cluster. Look at the section "Useful Variables" for more information on setting this up.
mmdsh is useful for issuing commands but not very pretty at collecting data, the output from the command is not ordered particularly nicely. It is possible to pipe the output to the sort command to tidy things up a bit, but a preferred method for nicely formatted output is to use a for loop. The following example shows mmdsh in use.
# mmdsh rcportmap stop bwa02.testsite: Shutting down RPC portmap daemon..done bwa01.testsite: Shutting down RPC portmap daemon..done
Running a command that returns a single line from each node looks fine.
bwa01:~ # mmdsh tail -n 5 /var/log/messages bwa02.testsite: Nov 20 12:31:25 bwa02 sshd[17256]: Accepted publickey for root from 192.168.205.150 port 59855 ssh2 bwa01.testsite: Nov 20 09:24:11 bwa01 syslog-ng[2362]: STATS: dropped 0 bwa02.testsite: Nov 20 12:31:36 bwa02 sshd[17279]: Accepted publickey for root from 192.168.205.150 port 59857 ssh2 bwa02.testsite: Nov 20 12:31:53 bwa02 sshd[17302]: Accepted publickey for root from 192.168.205.150 port 46368 ssh2 bwa02.testsite: Nov 20 12:32:11 bwa02 sshd[17325]: Accepted publickey for root from 192.168.205.150 port 46370 ssh2 bwa02.testsite: Nov 20 12:32:20 bwa02 sshd[17352]: Accepted publickey for root from 192.168.205.150 port 46372 ssh2 bwa01.testsite: Nov 20 10:24:12 bwa01 syslog-ng[2362]: STATS: dropped 0 bwa01.testsite: Nov 20 11:24:12 bwa01 syslog-ng[2362]: STATS: dropped 0 bwa01.testsite: Nov 20 12:00:01 bwa01 /usr/sbin/cron[11840]: (root) CMD ( /usr/lib64/sa/sa2 -A) bwa01.testsite: Nov 20 12:24:12 bwa01 syslog-ng[2362]: STATS: dropped 0
Commands that return multiple lines from each node are often out of order, this could make it confusing to read.
OS Hardening
Coming Soon.
Bash
Bash is the default shell in Linux and is the main way to interact with the BWA system. By adding a few variables to bash the managing of a BWA system can be made a little more streamlined. Only the root user should have access to the following variables therefore the variables will be set in root's .bashrc file. Edit root's .bashrc file with the a text editor of your choice. By default there is not .bashrc file on SLES so you need to create it. Below is the what your new root's .bashrc file should look like.
NODES="`cat /var/mmfs/config/nodes.list`" ONODES="`cat /var/mmfs/config/nodes.list | grep -v $HOSTNAME`" PATH="$PATH:/usr/lpp/mmfs/bin" WCOLL="/var/mmfs/config/nodes.list" export NODES ONODES PATH WCOLL
NODES
The NODES variable is a list of all the nodes in the cluster. It can be used to send commands in a for loop like mmdsh can. The advantage of the for loop is that the output is always in node order and multiple commands can be sent to each node and the use of pipes for text process is correct parsed. The following example shows how a for loop can run multiple commands with text processing.
# for i in $NODES ; do ssh $i "( echo \$HOSTNAME ; grep -i usb /var/log/messages | tail -n 10 ; echo )" ; done bwa01 Nov 4 09:53:22 bwa01 kernel: usb 1-2: new full speed USB device using uhci_hcd and address 4 Nov 4 09:53:22 bwa01 kernel: usb 1-2: new device found, idVendor=04b3, idProduct=4002 Nov 4 09:53:22 bwa01 kernel: usb 1-2: new device strings: Mfr=1, Product=2, SerialNumber=3 Nov 4 09:53:22 bwa01 kernel: usb 1-2: Product: IBM MM2 Nov 4 09:53:22 bwa01 kernel: usb 1-2: Manufacturer: IBM Nov 4 09:53:22 bwa01 kernel: usb 1-2: SerialNumber: 0000003012214 Nov 4 09:53:22 bwa01 kernel: usb 1-2: configuration #1 chosen from 1 choice Nov 4 09:53:22 bwa01 kernel: input: USB HID v1.10 Keyboard [IBM IBM MM2] on usb-0000:00:1d.0-2 Nov 4 09:53:22 bwa01 kernel: input: USB HID v1.10 Mouse [IBM IBM MM2] on usb-0000:00:1d.0-2 Nov 4 09:53:27 bwa01 kernel: usb 1-2: USB disconnect, address 4 bwa02 Oct 21 12:53:51 bwa01 kernel: usb 2-2: new device strings: Mfr=1, Product=2, SerialNumber=3 Oct 21 12:53:51 bwa01 kernel: usb 2-2: Product: IBM MM2 Oct 21 12:53:51 bwa01 kernel: usb 2-2: Manufacturer: IBM Oct 21 12:53:51 bwa01 kernel: usb 2-2: SerialNumber: 0000003012214 Oct 21 12:53:51 bwa01 kernel: usb 2-2: configuration #1 chosen from 1 choice Oct 21 12:53:51 bwa01 kernel: input: USB HID v1.10 Keyboard [IBM IBM MM2] on usb-0000:00:1d.0-2 Oct 21 12:53:51 bwa01 kernel: input: USB HID v1.10 Mouse [IBM IBM MM2] on usb-0000:00:1d.0-2 Oct 21 12:56:46 bwa02 kernel: usb-storage: device scan complete Oct 21 12:57:51 bwa02 kernel: usb 1-2: USB disconnect, address 3 Oct 26 13:15:06 bwa02 kernel: usb-storage: device scan complete
This loop prints the name of the host then prints the last 10 instances of the term (non case sensitive) 'usb' from the file /var/log/message by sending the STDOUT of grep to tail.
ONODES
The ONODES variable is a list of all nodes in the system except the current nodes. This is useful when making a change to a local file that need to be replicated on all nodes in the system. For example, If a change is made to /etc/hosts on one node and this change needs to be replicated to the other nodes in the system the follow commands could be executed.
for i in $ONODES ; do scp /etc/hosts $i:/etc/hosts ; done.
The commands would copy /etc/hosts to every nodes in the cluster except the node the command was run on.
PATH
Path is a standard variable in bash, it is a list or directories the bash uses to search for commands. The path variable is often different for different users, for example the root user would normally have /sbin and /usr/sbin it its path, an unpriviledged user would not.
When GPFS is installed the binaries are all installed to /usr/lpp/mmfs/bin/. This directory is not in the standard path so needs to be added unless you don't mind typing the entire path every time you want to run a GPFS command. As the vast majority of the GPFS commands are only runnable as root there is no point to changing the PATH variable for all users.
The line PATH=$PATH:/usr/lpp/mmfs/bin instructs bash to search /usr/lpp/mmfs/bin after all of the directories listed before it.
WCOLL
The WCOLL variable is known as the "Working Collective" and is the variable used by mmdsh to identify the nodes in a cluster. WCOLL be the path to a file that contains a list of hostnames of devices you want including in the "Working Collective"
