# Report for the software project course ITX8521

The goal of the project was to set up an environment for running automatic Linux kernel boot tests on embedded devices. A wiki of the project can be found [here](https://wiki.k-space.ee/index.php?title=Automated_Embedded_Testing).

The subgoals were defined as following:

1. Booting devices over USB
2. Booting devices through another device
3. Booting devices over ethernet
4. Setting up the LAVA environment
5. Integrating the LAVA environment with KernelCI

## 1. Booting devices over USB
In order to run Linux kernel boot tests automatically, it should be possible to send the compiled kernel image to the device remotely. I had access to two devices with Allwinner SoCs:

* [Xunlong Orange Pi Zero](http://linux-sunxi.org/Orange_Pi_Zero)
* [Xunlong Orange Pi Zero Plus 2](http://linux-sunxi.org/Orange_Pi_Zero_Plus)

Allwinner SOCs have a [ROM chip](http://linux-sunxi.org/BROM) containing the primary program loader, which can put the device being booted into [FEL mode](http://linux-sunxi.org/FEL). FEL mode is usually used for initial programming and recovery of Allwinner devices. In FEL mode, the device acts as a USB slave and it's possible to load artifacts into the device's memory and make the device execute from a specified location. Sunxi has developed a command-line tool sunxi-fel that implements the FEL protocol over USB.

*NB! I wasn't eventually able to boot the Xunlong Orange Pi Zero Plus 2 device, so the instructions for this device should be taken with a grain of salt, especially in step 1.2.*

## 1.1. Compiling sunxi-fel
Since the sunxi-tools debian package available to my OS (Ubuntu 17.10) contained a version of **sunxi-fel** that wasn't able to boot the device, I compiled the sunxi-fel tool from source.
```bash
git clone https://github.com/linux-sunxi/sunxi-tools --depth 1
cd sunxi-tools
make clean
make -j5
cd ..
```

## 1.2. Compiling Das U-Boot
First get the source code:
```bash
git clone git://git.denx.de/u-boot.git
cd u-boot
git tag
git checkout v2018.05
```
U-Boot needs to be cross compiled for the architecture of the device, it will be running on. Xunlong Orange Pi Zero contained an H2+ **arm** SoC while the Xunlong Orange Pi Zero Plus 2 contained an H5 **arm64** SoC.

For Xunlong Orange Pi Zero:
```bash
# Make sure you have the necessary cross-compiler
sudo apt update && sudo apt install gcc-arm-linux-gnueabihf
export CROSS_COMPILE=arm-linux-gnueabihf-
make orangepi_zero_defconfig
make -j5
cd ..
```
For Xunlong Orange Pi Zero Plus 2, we also need to build BL31:
```bash
# Make sure you have the necessary cross-compiler
sudo apt update && sudo apt install gcc-aarch64-linux-gnu
export CROSS_COMPILE=aarch64-linux-gnu-
# Build BL31, a dependency of 64-bit U-Boot
cd ..
git clone https://github.com/apritzel/arm-trusted-firmware.git
cd arm-trusted-firmware
make clean
make PLAT=sun50iw1p1 DEBUG=1 bl31
cd ..
# Build U-Boot
cd u-boot
export BL31=../arm-trusted-firmware/build/sun50iw1p1/debug/bl31.bin
make orangepi_zero_plus2_defconfig
make -j5
cd ..
```

## 1.3. Compiling Linux kernel and device tree binary
First get the source code:
```bash
git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git --depth 1
cd linux
git tag
git checkout v4.17
make mrproper
```
For Xunlong Orange Pi Zero:
```bash
export ARCH=arm
export CROSS_COMPILE=arm-linux-gnueabihf-
make sunxi_defconfig
make -j5 zImage dtbs
cd ..
```
For Xunlong Orange Pi Zero Plus 2 I couldn't find a suitable defconfig in arch/arm64/configs. So I used a defconfig which I found on github:
```bash
$ cd arch/arm64/configs
$ wget https://raw.githubusercontent.com/OrangePiLibra/OrangePiH5_kernel/master/arch/arm64/configs/OrangePiH5_Zero_Plus2_defconfig
cd ../../..
$ export ARCH=arm64
$ export CROSS_COMPILE=aarch64-linux-gnu-
make OrangePiH5_Zero_Plus2_defconfig
make -j5 zImage dtbs
cd ..
```

## 1.4. Creating an SD card to force the device into FEL mode
There are multiple options to force the device into FEL mode, one of them is to use a special SD card image. Make sure to replace /dev/sdX with the correct device.
```bash
wget https://github.com/linux-sunxi/sunxi-tools/raw/master/bin/fel-sdboot.sunxi
dmesg -w
# Insert the SD and see where it gets mounted
dd if=fel-sdboot.sunxi of=/dev/sdX bs=1024 seek=8
```

http://linux-sunxi.org/FEL#Through_a_special_SD_card_image

## 1.4. Creating an environment file for U-Boot
Create a file with the following contents:
```
#=uEnv
bootargs=console=ttyS0,115200 earlyprintk root=/dev/mmcblk0p1 rw panic=2 clk_ignore_unused loglevel=15
boot=bootz 0x46000000 - 0x49000000
bootcmd=run boot
```
This file will set up the environment variables of U-Boot so that it can automatically boot the kernel without needing further instructions in the u-boot prompt.
http://linux-sunxi.org/FEL/USBBoot#Overriding_environment_variables_with_uEnv-style_data

The file can be named anything, but is assumed later that this file was named **env.txt**. The first line of the file (#=uEnv) will be detected by the **sunxi-fel** tool and as a result the file will be fed to U-Boot for execution when it first loads.

The value of bootargs will be used as boot arguments for the kernel that will be booted. Kernel arguments are documented at [kernel.org](https://www.kernel.org/doc/Documentation/admin-guide/kernel-parameters.txt).

The value of bootcmd will be executed by U-Boot when a booting timeout ends, it usually boots the system. 

bootz is a boot command that is used for compressed kernels. The first argument (0x46000000) indicates the location in memory of the kernel, the second argument (-) indicates the location of initrd and the third argument (0x49000000) indicates the location of the device tree binary.

https://www.denx.de/wiki/view/DULG/UBootEnvVariables
## 1.5. Booting the device
Run `dmesg -w` in one terminal session and connect the device's serial to your computer's USB port. This way you'll see where it is mounted. Once you know the mount location, you can connect to the device's serial:
```bash
picocom -b115200 /dev/ttyUSB3
```
You'll be seeing the U-Boot and kernel booting messages in this session once the booting is initiated.

Make sure the SD card created in step 1.4. is in the device and connect the device's OTG microUSB to your computer's USB to power the device. The FEL protocol will also work over this cable. To make sure the device entered FEL mode and the connection is established, run:

```bash
sudo ./sunxi-tools/sunxi-fel version
```
If everything is correct, you should see the device's bootloader version printed.

To initiate booting for Xunlong Orange Pi Zero, run:
```bash
sudo ./sunxi-tools/sunxi-fel --progress --verbose \
  uboot u-boot/u-boot-sunxi-with-spl.bin \
  write 0x46000000 linux/arch/arm/boot/zImage \
  write 0x49000000 linux/arch/arm/boot/dts/sun8i-h2-plus-orangepi-zero.dtb \
  write 0x49100000 env.txt
```
To initiate booting for Xunlong Orange Pi Zero Plus 2, run:
```bash
sudo ./sunxi-tools/sunxi-fel --progress --verbose \
  uboot u-boot/u-boot-sunxi-with-spl.bin \
  write 0x46000000 linux/arch/arm/boot/zImage \
  write 0x49000000 linux/arch/arm64/boot/dts/allwinner/sun50i-h5-orangepi-zero-plus2.dtb \
  write 0x49100000 env.txt
```
In my case, booting the Zero Plus 2 resulted in the following error:
```
SPL: Unsupported Boot Device!
SPL: failed to boot from all boot devices
### ERROR ### Please RESET the board ###
```
The problem is with U-Boot, the secondary program loader is not able to boot it. There must be something I did wrong when compiling U-Boot for this device.

# 2. Booting the devices through another device
The objective of this step was to set up a device (let's call it master) that can force the device under test (DUT) to reboot by controlling a relay which switches the power to the DUT. The master could act as a slave device in the context of a LAVA environment set up in step 4. 

I was not able to work on this due to time constraints.
# 3. Booting the devices over ethernet
I was not able to work on this due to time constraints.
# 4. Setting up the LAVA environment
From [LAVA v2 documentation](https://validation.linaro.org/static/docs/v2/index.html)
>LAVA is a continuous integration system for deploying operating systems onto physical and virtual hardware for running tests.

LAVA has a web interface through which users can submit test jobs to devices connected to the system. The test jobs are defined in YAML format. A test log is attached to each test job submission and it is made accessible through the web interface. The log contains boot messages and interactions on the device's tty prompt along with test result messages inserted by LAVA.

I set up a LAVA environment at 172.20.8.240 (lava-debian.k-space.lan), which is accessible inside k-space VPN. Along with setting up the system I created ansible scripts that automate the following adminstrative tasks:
* Installing the necessary packages to set up LAVA and configuring apache to make it accessible on port 80
* Uninstalling LAVA
* Creating users with superuser access

The ansible scripts are available here: https://git.k-space.ee/madislutter/ansible-lava.

The LAVA instance is currently set up as a single-node as opposed to multi-node setup. No physical devices have been connected to the machine running LAVA, though a orangepi-zero-h2-plus-01 device has been created. The test jobs and test definitions which I successfully ran on a qemu device are available in this repository. There is also a test job in this repository named orangepi-zero.yml, but I wasn't able to run it successfully. The error message is the following ([also visible here](http://172.20.8.240/scheduler/job/17)):
```
ConfigurationError: The LAVA instance is not configured correctly. Please report this error to LAVA admins.
case: job
result: fail
definition: lava
error_type: Configuration
error_msg: commands not specified in boot parameters
```

The problem seems to lie with the device dictionary of the device I ran this job on. The orange-pi-zero-plus-h2-01 device dictionary is currently set to simply:
```jinja2
{% extends 'sun8i-h2-plus-orangepi-zero.jinja2' %}
```
---
While working on setting up LAVA, it became apparent to me that LAVA does not support compiling the operating systems that are deployed onto devices under test. This means a separate tool needs to be found or developed that compiles a new kernel whenever a commit is made to the kernel and submits a new job to LAVA, detailing inside the job description YAML from where the newly compiled kernel can be downloaded.

# 5. Integrating the LAVA environment with KernelCI
The final objective was to connect the k-space test lab to the central [KernelCI lab](https://kernelci.org/) so that whenever a commit is made to the linux kernel, it would be tested on all devices in k-space and the developers could have feedback if anything went wrong.

I was not able to work on this because the prerequirement of this step is completing all previous steps.

# Sources of information
* ["How Linux Works"](https://nostarch.com/howlinuxworks2) by Brian Ward gives a nice high-level overview of how a Linux system boots. Understanding that was a prerequisite to understanding what I was doing.
* [Linux sunxi wiki](http://linux-sunxi.org/Main_Page) - information about Allwinner SoC based devices
* [Das U-Boot manual](https://www.denx.de/wiki/DULG/Manual)
* [LAVA documentation](https://validation.linaro.org/static/docs/v2/contents.html)
* [Ansible documentation](https://docs.ansible.com/ansible/latest/index.html)
* [Arch wiki](https://wiki.archlinux.org/) - helpful with all the rest