The goal of the project was to set up an environment for running automatic Linux kernel boot tests on embedded devices. A wiki of the project can be found [here](https://wiki.k-space.ee/index.php?title=Automated_Embedded_Testing).
In order to run Linux kernel boot tests automatically, it should be possible to send the compiled kernel image to the device remotely. I had access to two devices with Allwinner SoCs:
* [Xunlong Orange Pi Zero](http://linux-sunxi.org/Orange_Pi_Zero)
* [Xunlong Orange Pi Zero Plus 2](http://linux-sunxi.org/Orange_Pi_Zero_Plus)
Allwinner SOCs have a [ROM chip](http://linux-sunxi.org/BROM) containing the primary program loader, which can put the device being booted into [FEL mode](http://linux-sunxi.org/FEL). FEL mode is usually used for initial programming and recovery of Allwinner devices. In FEL mode, the device acts as a USB slave and it's possible to load artifacts into the device's memory and make the device execute from a specified location. Sunxi has developed a command-line tool sunxi-fel that implements the FEL protocol over USB.
*NB! I wasn't eventually able to boot the Xunlong Orange Pi Zero Plus 2 device, so the instructions for this device should be taken with a grain of salt, especially in step 1.2.*
## 1.1. Compiling sunxi-fel
Since the sunxi-tools debian package available to my OS (Ubuntu 17.10) contained a version of **sunxi-fel** that wasn't able to boot the device, I compiled the sunxi-fel tool from source.
U-Boot needs to be cross compiled for the architecture of the device, it will be running on. Xunlong Orange Pi Zero contained an H2+ **arm** SoC while the Xunlong Orange Pi Zero Plus 2 contained an H5 **arm64** SoC.
## 1.4. Creating an SD card to force the device into FEL mode
There are multiple options to force the device into FEL mode, one of them is to use a special SD card image. Make sure to replace /dev/sdX with the correct device.
This file will set up the environment variables of U-Boot so that it can automatically boot the kernel without needing further instructions in the u-boot prompt.
The file can be named anything, but is assumed later that this file was named **env.txt**. The first line of the file (#=uEnv) will be detected by the **sunxi-fel** tool and as a result the file will be fed to U-Boot for execution when it first loads.
The value of bootargs will be used as boot arguments for the kernel that will be booted. Kernel arguments are documented at [kernel.org](https://www.kernel.org/doc/Documentation/admin-guide/kernel-parameters.txt).
The value of bootcmd will be executed by U-Boot when a booting timeout ends, it usually boots the system.
bootz is a boot command that is used for compressed kernels. The first argument (0x46000000) indicates the location in memory of the kernel, the second argument (-) indicates the location of initrd and the third argument (0x49000000) indicates the location of the device tree binary.
Run `dmesg -w` in one terminal session and connect the device's serial to your computer's USB port. This way you'll see where it is mounted. Once you know the mount location, you can connect to the device's serial:
```bash
picocom -b115200 /dev/ttyUSB3
```
You'll be seeing the U-Boot and kernel booting messages in this session once the booting is initiated.
Make sure the SD card created in step 1.4. is in the device and connect the device's OTG microUSB to your computer's USB to power the device. The FEL protocol will also work over this cable. To make sure the device entered FEL mode and the connection is established, run:
```bash
sudo ./sunxi-tools/sunxi-fel version
```
If everything is correct, you should see the device's bootloader version printed.
To initiate booting for Xunlong Orange Pi Zero, run:
In my case, booting the Zero Plus 2 resulted in the following error:
```
SPL: Unsupported Boot Device!
SPL: failed to boot from all boot devices
### ERROR ### Please RESET the board ###
```
The problem is with U-Boot, the secondary program loader is not able to boot it. There must be something I did wrong when compiling U-Boot for this device.
# 2. Booting the devices through another device
The objective of this step was to set up a device (let's call it master) that can force the device under test (DUT) to reboot by controlling a relay which switches the power to the DUT. The master could act as a slave device in the context of a LAVA environment set up in step 4.
I was not able to work on this due to time constraints.
# 3. Booting the devices over ethernet
I was not able to work on this due to time constraints.
# 4. Setting up the LAVA environment
From [LAVA v2 documentation](https://validation.linaro.org/static/docs/v2/index.html)
>LAVA is a continuous integration system for deploying operating systems onto physical and virtual hardware for running tests.
LAVA has a web interface through which users can submit test jobs to devices connected to the system. The test jobs are defined in YAML format. A test log is attached to each test job submission and it is made accessible through the web interface. The log contains boot messages and interactions on the device's tty prompt along with test result messages inserted by LAVA.
I set up a LAVA environment at 172.20.8.240 (lava-debian.k-space.lan), which is accessible inside k-space VPN. Along with setting up the system I created ansible scripts that automate the following adminstrative tasks:
* Installing the necessary packages to set up LAVA and configuring apache to make it accessible on port 80
* Uninstalling LAVA
* Creating users with superuser access
The ansible scripts are available here: https://git.k-space.ee/madislutter/ansible-lava.
The LAVA instance is currently set up as a single-node as opposed to multi-node setup. No physical devices have been connected to the machine running LAVA, though a orangepi-zero-h2-plus-01 device has been created. The test jobs and test definitions which I successfully ran on a qemu device are available in this repository. There is also a test job in this repository named orangepi-zero.yml, but I wasn't able to run it successfully. The error message is the following ([also visible here](http://172.20.8.240/scheduler/job/17)):
```
ConfigurationError: The LAVA instance is not configured correctly. Please report this error to LAVA admins.
case: job
result: fail
definition: lava
error_type: Configuration
error_msg: commands not specified in boot parameters
```
The problem seems to lie with the device dictionary of the device I ran this job on. The orange-pi-zero-plus-h2-01 device dictionary is currently set to simply:
While working on setting up LAVA, it became apparent to me that LAVA does not support compiling the operating systems that are deployed onto devices under test. This means a separate tool needs to be found or developed that compiles a new kernel whenever a commit is made to the kernel and submits a new job to LAVA, detailing inside the job description YAML from where the newly compiled kernel can be downloaded.
# 5. Integrating the LAVA environment with KernelCI
The final objective was to connect the k-space test lab to the central [KernelCI lab](https://kernelci.org/) so that whenever a commit is made to the linux kernel, it would be tested on all devices in k-space and the developers could have feedback if anything went wrong.
I was not able to work on this because the prerequirement of this step is completing all previous steps.
* ["How Linux Works"](https://nostarch.com/howlinuxworks2) by Brian Ward gives a nice high-level overview of how a Linux system boots. Understanding that was a prerequisite to understanding what I was doing.
* [Linux sunxi wiki](http://linux-sunxi.org/Main_Page) - information about Allwinner SoC based devices