Software configuration tips for Raspberry Pi clusters & parallel-ssh command

I missed that linux.conf.au 2021 took place on January 23-25 2021, and while browsing the schedule I noticed a talk entitled “Building Raspberry Pi Supercomputers” by Federico Lucifredi, Product Management Director for Ceph Storage at Red Hat.

In the talk, he mostly focuses on the software part, and besides some basic steps, I learned about some new commands that be useful to people managing clusters of Raspberry Pi or other Linux boards or hosts.

Configuring a cluster

He used Picocluster image in his example, but for people wanting to use 64-bit OS, he recommends Ubuntu or Fedora images until Raspberry Pi OS 64-bit becomes stable. The first part of the configuration is making sure all the main user is the same on all board, disable SSH for root, and configure run levels (X not needed on clusters). Networking is configured with fixed IP addresses for Ethernet, and DHCP for WiFi.

He also configured ssh without password (i.e. public/private keys), but using ssh-copy-id command to enable the keys on all boards, as well as NTP with timedatectl command, and an NFS share on the master node to share data.

Some of the useful commands from the Picocluster image include:

  • restartAllNodes.sh used to restart all of the nodes in the cluster.
  • stopAllNodes.sh used to stop or shut down all of the nodes in the cluster.
  • genKeys.sh used to generate an SSH key and distribute it to all nodes.
  • testAllNodes.sh runs df -h on each node to indicate that each node is up and running.

Parallell-ssh command

parallel-ssh commands - software for raspberry pi clusters

But the command that interested me the most from the talk was parallel-ssh which can be installed on the master node as follows:


The manpage describes pssh as “a program for executing ssh in parallel on a number of hosts. It provides features such as sending input to all of the processes, passing a password to ssh, saving output to files, and timing out”.

Let’s take the first example from Federico talk:


The nodes file contains the list of host in the following format:


I tried it with two Linodes I own and already configured with private/public keys:


It just returns whether the command worked without output from the command. If we can output we need to add inline:


The third command just checks which hosts are alive with a ping command to the master host IP address in Federico’s cluster configuration:


The fourth command is also using ping but this time to test DNS connectivity wih output from the command on each other:


We can also check the ping time, and it’s easy to find out which host is based in the US and which one in Europe.

The last command from the slide is specific to Raspberry Pi, and used to check whether it’s possible to read the temperature (without actually returning it), so I’ll skip it.

I can see many really useful use case for parallel-ssh. For example, if you had to install a program on multiple boards, you’d only need to type the command once, and the output would show where the installation was successful, and potentially where it failed.

You can see the full talk in the video above, as sadly Federico did not share the presentation slides, at least not yet.  as well as the slides that also include examples about the Message Passing Interface (MPI) open library standard for distributed memory parallelization, but it was quickly skipped in the video due to time constraints.

Share this:

Support CNX Software! Donate via PayPal or cryptocurrencies, become a Patron on Patreon, or buy review samples

Subscribe
Notify of
guest
The comment form collects your name, email and content to allow us keep track of the comments placed on the website. Please read and accept our website Terms and Privacy Policy to post a comment.
5 Comments
oldest
newest
Markus
Markus
11 months ago

I also find cssh (https://linux.die.net/man/1/cssh) very useful. Especially when you have to run a similar (but not identical) sequence of commands on a number of machines.

Wendy Miller
Wendy Miller
11 months ago

Interesting Article! Can’t wait to try some of this. And I’m not Wendy, browser auto filled that (facepalm)

Dave
Dave
11 months ago

Ugh, don’t bother with parallel-ssh. Ansible has replaced it completely, will do the same things you want there along with a whole lot more you can get it to do as you need it.

willy
willy
11 months ago

You seem to be confusing administration and automation. But you’re not the only one, it’s frequent.

willy
willy
11 months ago

I’m using pdsh for the same thing. I tried pssh a while ago but didn’t figure how to get the output from the machines so I didn’t seek further, since if it requires to read a man page to use it once in a while, it’s not as convenient: $ pssh "-H miqi-"{2..9} grep eth0 /proc/net/dev [1] 07:35:17 [SUCCESS] miqi-6 [2] 07:35:17 [SUCCESS] miqi-2 [3] 07:35:17 [SUCCESS] miqi-7 [4] 07:35:17 [SUCCESS] miqi-8 [5] 07:35:17 [SUCCESS] miqi-3 [6] 07:35:17 [SUCCESS] miqi-4 [7] 07:35:17 [SUCCESS] miqi-5 [8] 07:35:17 [SUCCESS] miqi-9 $ pdsh "-w miqi-"{2..9} grep eth0 /proc/net/dev miqi-7:   eth0: 1669987727 196272757    0… Read more »

Advertisement