Advanced Node Management

but first, a rant

Full disclosure- I have a horse in this race. I have personal interest in seeing skywire succeed. Skywire is needed, and there is no viable potential substitute at present. Skywire is also the fundamental mechanism of skycoin distribution at this point; and the skyminer itself is the flagship hardware product of skycoin.

I have long lamented the official update and installation procedures as being unreliable, prone to error, etc. In the testnet, there were no binary releases. In some ways this was better because it forced people to learn how to compile golang binaries (which is easy); but the official installation documentation and the setup of the skywire installation itself in skybian was, candidly, a hot mess. In the testnet, skywire was pre-set up in /usr/local, in the skybian images. This is a violation of the spec for that path, and a bad idea for other reasons, but I won’t get into that.

The current setup of skywire in skybian is not what I would term an ‘installation’ because a package of skywire is not used. A script places binaries in the image, as well as other things. The wiki documentaton in skywire is not descriptive of the skybian installation. The skybian installation is only defined in skybian, and I would have to look at the source code to even know what is going on there or to troubleshoot it. Many errors are manifesting as a result of not using packages.

Here is the problem I see with the official installation and updates in a nutshell:

A user uses skyimager to make images, writes them on a microSD, and then tries to run skywire. Let's assume they figure out how to access the hypervisor web interface and it does work for them initially. The first time it does not work, they are required to do a series of things they may have never done before, the first of which is establishing the SSH session. People get stuck right there! People also get stuck in a loop of just reflashing to try to fix their issue because they don't really know how to troubleshoot.

That is the issue in a nutshell, so what is the solution?

  • Canonize a standard installation and update process which uses packages
  • The installation and update process should be informative of troubleshooting
  • Require users have basic linux skill

These are realistic goals to move things forward.

I never had any issues figuring out how to update on my own. The proficiency with which I could update (which lent a lot to testing and generating feedback / issues) was built upon the efforts I spent developing arch packages of this software, and maintaining builds in the Arch User Repos

The whole reason I was able to do this is thanks to the archlinux packaging tools (makepkg) and the arch user repos, which allows users to submit and maintain builds of software. Archlinux has a very good packaging system, that’s probably the best feature of the operating system. The provided packaging and repository creation tools are further extended in their functionality and ease-of-use by AUR wrappers such as yay.

What this means is that archlinux provides precompiled packages which are officially maintained, but also a database of user submitted builds for software, the AUR. Yay acts as a wrapper for pacman and makepkg, meaning that one can install any software from either the official repositries or the AUR with the same command, and it greatly simplifies the management of software from both officially configured repositories and from the AUR in the same command via this standardization.

If you look at skywire in the AUR you can see a build and a set of scripts. Pull requests have been merged to the source code which would let skywire-cli generate a config file with the correct default paths for the package’s installation. A script is provided which automatically generates the skywire configuration file and starts a hypervisor. Much time and effort has been put into this packaging..

To continue this diatribe-

I reject the notion that skywire needs to update itself. It doesn’t, after all, install itself; and we should not desire that. The updater implementation has been the problem all along, as the underlying software is very high quality. I was shocked from very early on when the skywire ‘updater’ was announced; and I railed against the notion that skywire would usurp the function of the package manager to update itself. This causes a lot of problems, the first of which is that the installation and updates become divergent processes.

A recent example of problems which are caused by not using packages is the 0.4.3 release of skywire. This is a release I would argue should have never happened. The difference between 0.4.3 and 0.4.2 is one directory. It went unnoticed that the apps were not working in the official skybian images for several months since they had been released. The apps had been placed in /usr/bin instead of /usr/bin/apps, which did not agree with the path specified in the config file. The only way to fix it is to edit the file by hand. When the issue in question spans thousands of nodes in a global network, users should not be required to edit .json manually.

So why did this happen? Why did it not affect the binary release package of skywire I maintain? The reason is simple. When I updated the version of skywire in the PKGBUILD and ran the build to test, the build errored at that point where the expected path was not found. At that point I spent 10 seconds fixing it in the build.

a perfect and decentralised system

The point of packaging is to simplify the installation and updates. If you are using my debian package repository you know how easy it is to install skywire. But apt isn’t working well for everyone, especially users in china. As well, my APT repo on its own is a single point of failure and it does require some resources to serve these packages.

If the skywire network was running archlinux natively, these issues would be so far in the past that they would be barely remembered. That may not be in the cards at this point, as making correct archARM images for orange pis remains an elusive goal for me.

The archlinux implementation of updates for node clusters is trivially simple, and I believe it more or less achieves perfection:

  • Skywire can be compiled locally from source or using a binary release using the builds I have submitted to and maintain in the AUR
  • The built package can be added to a repository and served on the local network to update the cluster
  • nodes use the hypervisor board as an update mirror for all the system updates

Package management, packaging, repository creation; all of these things are integral to archlinux and are already included with pacman

The first point, easily building skywire into a package from source, is all one really needs. The rest is just reducing bandwidth and speeding up the updates.

the next best thing for armbian

The skywire network in its current state consists mainly of various pi computers around the world which are connected to the existing internet infrastructure. The official hardware is the orange pi prime, and the operating system which is used is modified armbian (.deb based). There are also a host of other raspberry pis and some very creative types are using tv boxes.

Deb-based operating systems, such as armbian and raspbian, have the advantage of pre-made images with good hardware support

The disadvantage of .deb distros:

  • stale packages (old)
  • no easy to use packaging system

On the other hand, archlinuxARM does not provide images, but a very up to date filesystem, which includes the packaging tools necessary to create .deb packages through a system I have devised.

Why would you want to create your own packages? Because compiling from source is the most secure but also optimizes the binary for the specific hardware it is compiled on. Why have a local repository for your skywire cluster? To administer updates of the aforementioned packages. If you are compiling from source (even just using the binary release) and distributing the updates to the nodes you have achieved minimum bandwidth. And since you know what you are doing enough to do that, everything gets easier to maintain, because you have a system in place which can locally package software.

I have not been able to create a correct archARM image for orange pi prime. Raspberry pi will run archlinux arm natively. A link will be provided to images in the future, as well as much shorter documentation for the same local repository configuration.

In the event that archlinuxarm cannot be made to run natively on the hardware, it should be accessed via chroot on one board, the hypervisor. The process for linux users is as follows:

What follows is a technical overview of the process for creating your own packages and maintaining your skywire node cluster.

Technical Overview - installing Skywire from AUR on ArchlinuxARM

It’s worthwhile to outline the process for doing this on archlinux(arm) because the process for Armbian includes almost the entire process for doing this on archlinux; the parts which are not the same are simply translated to the debian format.

First, you’ll want to create the microSD card that will boot on your board https://archlinuxarm.org/platforms

As previously stated, this process as it applies to creating images will be further documented and images provided for raspberry pis and other supported boards at a later date. Try to figure it out on your own until then. If your board is officially supported you should have no issue.

When the board is booted, ssh to it.

I am going out on a limb here, but I recommend using the endeavourOS-arm installer for a graphical interactive installer session, which will yield a slightly better configured final product. (we won’t bother doing this in the chroot for the armbian method)

pacman-key --init
pacman-key --populate archlinuxarm
pacman -Syu git libnewt wget

The install script will be installing Numerous Packages, to make the downloading go faster:

nano /etc/pacman.conf

Scroll down to #ParallelDownloads = 5

Uncomment this line by removing the leading # and if desired, change 5 to 8 or however many parallel downloads you want to run. For my ISP 8 works well.

Save the file, then

systemctl reboot

Then ssh to the board again log back in with root.

Now we’re going to install EndeavourOS on the device by typing the following commands:

git clone https://github.com/endeavouros-arm/install-script.git
cd install-script
chmod +x *.sh
./endeavour-ARM-install-V2.X.sh

You will be greeted with a graphical installer session; similar to any linux installer.

Complete the steps. You only need a server installation for skywire, but if you have a board with more ram you can install a desktop if desired. Reboot once you have finished the installation and log in as the user you created.

All the tools of endeavourOS are at hand and the system is updated and ready to install skywire, which we can do now with just a single command:

yay -S skywire

When the postinstall script is executed, skywire will be configured and started as hypervisor, with no additional configuration necessary.

Installing skywire on additional nodes

Now, let’s say you want to let your other boards update from the one you just installed skywire on.

There are two parts to this; as there are two types of software now present on the board that skywire was just installed on:

  • pre-packaged software provided by repositories which are configured in /etc/pacman.conf
  • the skywire package which was created locally from the AUR.
Hypervisor update-sharing configuration

On the hypervisor board, host the package cache on the local network. Since you have golang installed, you can use a golang statc fileserver. All that is necessary is to symlink the sync repository db files to the package cache first. Refer to the read-only cache method of doing this outlined in the archwiki.

Symlink the sync repository .db files to the package cache

sudo ln -s /var/lib/pacman/sync/*.db /var/cache/pacman/pkg

Now your hypervisor can act as a mirror for all updates it receives from official repositories. But what about skywire?

Simple enough, you just need to create a local package repository which includes skywire, and then include the built skywire package in pacman’s cache. Refer to custom local package repository in the archwiki.

sudo repo-add /var/lib/pacman/sync/local.db.tar.gz ~/.cache/yay/skywire/*.pkg.tar.xz
sudo ln -s ~/.cache/yay/skywire/*.pkg.tar.xz /var/cache/pacman/pkg/
sudo repo-add /var/lib/pacman/sync/local.db.tar.gz ~/.cache/yay/skywire/*.pkg.tar.zst
sudo ln -s ~/.cache/yay/skywire/*.pkg.tar.zst /var/cache/pacman/pkg/
sudo ln -s /var/lib/pacman/sync/*.db /var/cache/pacman/pkg

host the repository on the local network

echo 'package main

import (
  "flag"
  "log"
  "net/http"
  )

  func main() {
    port := flag.String("p", "8079", "port to serve on")
    directory := flag.String("d", ".", "the directory of static file to host")
    flag.Parse()

    http.Handle("/", http.FileServer(http.Dir(*directory)))

    log.Printf("Serving %s on HTTP port: %s\n", *directory, *port)
    log.Fatal(http.ListenAndServe(":"+*port, nil))
  }
  ' > srvcache.go

  sudo go run srvcache.go -d /var/cache/pacman/pkg
Configuring visors to use the updates shared from the hypervisor

Now, Create another microSD card with archlinuxARM, boot it up, and after you have initialized and populated the pacman-keyring;

edit /etc/pacman.d/mirrorlist

nano /etc/pacman.d/mirrorlist

add the ip address of the board which is providing updates at the top of this file on every node which is to update from this one (i.e. only same architecture); matching the format of the other entries:

Server = http://192.168.0.2:8079

Save the file and exit.

then add the following to /etc/pacman.conf

[local]
SigLevel = Never
Server = http://192.168.0.2:8079

save the file, and then attempt to update the board

pacman -Syy && pacman -Syu skywire

all the software on the board should now be updated, and skywire installed; leaving just the small matter of configuring the skywire installation on these boards to use the hypervisor instance on the board providing updates.

The instructions for doing this are printed to the screen after skywire has been installed.

Be sure to reboot the board after updates which include the linux kernel

Future updates to your cluster

on the hypervisor-

  • update the hypervisor with: yay -Syy && yay -Syu
  • install / update skywire on the hypervisor with: yay -S skywire (the previous command should have updated skywire if it was installed already and a new version became available)
  • add the updated skywire package to the local repository with repo-add
  • symlink the sync.db files and include skywire package in pacman’s cache
  • host the package cache on the local network

on the visors, all that should be necessary to update skywire is to pass a full system update

  • pacman -Syy && pacman -Syu

Technical Overview - using Armbian

Download Armbian

download and flash the correct armbian image for your board to the microSD

https://www.armbian.com/download/

write the image to the microSD card using dd (or dcfldd)

dd if=/path/to/armbian.img of=/dev/sdX

When the command completes, insert the microSD card to the pi and boot the image by applying power.

SSH to the board

You may need to check your router web interface to identify the ip address of the pi in order to ssh to the board.

your router web interface can typically be accessed via one of the following links: http://192.168.0.1 http://192.168.1.1 http://192.168.2.1

If you cannot access your router’s web interface:

  • look on the side of the router for the web interface access point IP address.
  • check the network you are currently connected to for the router or gateway IP address.

Once you have accessed the wb interface, look for a list of LAN DHCP clients. You should be able to spot the board as one of these

open a terminal (or command prompt on windows) and establish the ssh session to the board:

ssh root@192.168.0.2

if your router will do hostname routing, you may be able to access it via the hostname instead of the ip address

ssh root@armbian

Update Armbian

if you are in china, this may not work.

as root or using sudo:

apt update
apt upgrade

Download archlinuxARM filesystem

download the archlinuxARM root filesystem that matches your board’s architecture

https://archlinuxarm.org/platforms

64-bit ARM / armv8 / aarch64

wget http://os.archlinuxarm.org/os/ArchLinuxARM-aarch64-latest.tar.gz

armv7

wget http://os.archlinuxarm.org/os/ArchLinuxARM-armv7-latest.tar.gz

armv5

wget http://os.archlinuxarm.org/os/ArchLinuxARM-armv5-latest.tar.gz

Set up chroot

extract the root filesystem

mkdir root
bsdtar -xpf ArchLinuxARM-aarch64-latest.tar.gz -C root
cd root

set up a chroot

mount -t proc /proc proc/
mount -t sysfs /sys sys/
mount --rbind /dev dev/

Configure chrooted filesystem

first, initialize and populate pacman key and check network connectivity

pacman-key--init
pacman-key --populate

update via pacman, install base-devel, sudo, git, golang

pacman -Syy && pacman -Syu base-devel git sudo go

Create a user in the Chroot

create a user and grant the user sudo permissions in the chroot

adduser -m user

visudo

change to that user

su - user

Creating the skywire .deb package

manually install yay

mkdir -p ~/.cache/yay
cd ~/.cache/yay
git clone https://aur.archlinux.org/yay-git
cd yay-git
makepkg -scif

use yay to install skywire and dpkg in the chroot

yay -S skywire dpkg

uninstall skywire from the chroot (we just wanted sources and dependencies)

yay -R skywire

create the debian package

cd ~/.cache/yay/skywire-bin
makepkg -p deb.PKGBUILD

Exit the chroot

you can exit the chroot here

cd /
exit
exit
umount dev/
umount proc/
umount sys/

Install skywire

install the built skywire .deb package (on the hypervisor) which is in the home dir of the user you created in the chroot

dpkg -i ~/root/home/user/.cache/yay/skywire/*.deb

Setting up a local APT repository

add the built .deb package to a local repository for updating your nodes

you can download the script I use to do this. Be sure to modify it first to suit your needs.

mkdir -p ~/repo && cd ~/repo
wget https://deb.magnetosphere.net/create-deb-repo.sh
nano create-deb-repo.sh

modify the script and place the package you created in that directory

cp ~/root/home/user/.cache/yay/skywire/*.deb .
./create-deb-repo.sh

at this point, host the ~/repo directory on the local network.

You can use whatever webserver you like for this. If you cannot think of one, here is a simple one in golang:

echo 'package main

import (
	"flag"
	"log"
	"net/http"
)

func main() {
	port := flag.String("p", "8079", "port to serve on")
	directory := flag.String("d", ".", "the directory of static file to host")
	flag.Parse()

	http.Handle("/", http.FileServer(http.Dir(*directory)))

	log.Printf("Serving %s on HTTP port: %s\n", *directory, *port)
	log.Fatal(http.ListenAndServe(":"+*port, nil))
}
' > srvcache.go
sudo go run srvcache.go -d ~/repo

Configure the APT repository and install skywire to other nodes

configure the other boardsto use your local package repository

The process is the same as outlined here:

https://fiber.magnetosphere.net/

substituite the IP address:port of the board on your local network in place of deb.magnetosphere.net

FUTURE CLUSTER UPDATES

This may seem like a lot, and I can think of at least one bug affecting this process at current. But the next time you want to update the process becomes much shorter because most of that was just initial configuration steps.

Refer to the steps already defined above

  • enter the chroot; change to the user
  • use yay to build and install the updated version of skywire
  • uninstall that from the chroot
  • cd ~/.cache/yay/skywire-bin
  • makepkg -p deb.PKGBUILD
  • you can exit the chroot here
  • install the built skywire .deb package (on the hypervisor)
  • add the built .deb package to a local repository
  • update skywire on the nodes

The only thing I omitting here is that you can, if you desire, manually edit the PKGBUILD to update the vesion or whatever you like if it is not working, but I can fix it immediately in the AUR when I learn of any issue.

Technical Overview - using ArchARM and Armbian

It’s highly convenient if the board which is your hypervisor is supported by archlinuxARM. The skywire .deb package can be created and provided to the other boards; using an amalgam of the methods for each operating system. The board running archlinuxARM can create the .deb package of skywire and host a repository with this package. The information above should be sufficiently informative of this process.

Optimization / repurposing skyimager

A lot of this can be automated. Creating and updating the apt repo can be scripted. But the main automation should be a local repository that is set by skyimager to the ip address of the hypervisor on the local network. Skyimager should not be generating the config files for skywire. I would also argue that the skybian image be done away with entirely and the modifications to the image should simply be packaged, but the images are necessary in a certain context, if the location where they are being installed does not otherwise have an internet connection.

Get Started with Skycoin