User Manual

This manual is an early work in progress, not least because Liminix is not yet really ready for users who are not also developers. Your feedback to improve it is very welcome.

Installation

The Liminix installation process is not quite like installing NixOS on a real computer, but some NixOS experience will nevertheless be helpful in understanding it. The steps are as follows:

  • Decide whether you want the device to be updatable in-place (there are advantages and disadvantages), or if you are happy to generate and flash a new image whenever changes are required.

  • Create a configuration.nix describing the system you want

  • Build an image

  • Flash it to the device

Supported devices

For a list of devices that Liminix (present or previous versions) has run on, refer to devices/ in the source repo. For devices that _currently_ build, cross-reference it with the CI status. Everything that builds is (usually) expected to run, so if you end up with an image that builds but doesn’t boot, please report it as a bug.

As of June 2023 the device list is a little thin. Adding devices based on the Atheros or Mediatek (Ralink) platform should be quite straightforward if you have some C/Linux kernel experience and are prepared to open it up and attach serial wires: please refer to the Developer Manual.

Choosing a flavour (read-only or updatable)

Liminix installations come in two “flavours”- read-only or in-place updatable:

  • a read-only installation can’t be updated once it is flashed to your device, and so must be reinstalled in its entirety every time you want to change it. It uses the squashfs filesystem which has very good compression ratios and so you can pack quite a lot of useful stuff onto your device. This is good if you don’t expect to change it often.

  • an updatable installation has a writable filesystem so that you can update configuration, upgrade packages and install new packages over the network after installation. This uses the jffs2 filesystem: although it does compress the data, the need to support writes means that it can’t pack quite as small as squashfs, so you will not have as much space to play with.

Updatability caveats

At the time of writing this manual the read-only squashfs support is much more mature. Consider also that it may not be possible to perform “larger” updates in-place even if you do opt for updatability. If you have (for example) an 11MB system on a 16MB device, you won’t be able to do an in-place update of something fundamental like the C library (libc), as this will temporarily require 22MB to install all the packages needing the new library before the packages using the old library can be removed. A writable system will be more useful for smaller updates such as installing a new package (perhaps you temporarily need tcpdump to diagnose a network problem) or for changing configuration files.

Note also that the kernel is not part of the filesystem so cannot be updated this way. Kernel changes require a full reflash.

Creating configuration.nix

You need to create a configuration.nix that describes your device and the services that you want to run on it. The best way to get started is by reading one of the examples such as examples/rotuer.nix and modifying it to your needs.

configuration.nix conventionally describes the packages, services, user accounts etc of the device. It does not describe the hardware itself, which is specified separately in the build command (as you will see below).

Most of the functionality of a Liminix system is driven by services which are declared by modules: thus, to add for example an NTP service you first add modules/ntp to your imports list, then you create a service by calling config.system.service.ntp.build { .... } with the appropriate service-dependent configuration parameters.

let svc = config.system.service;
in {
  # ...
  imports = [
    ./modules/ntp
    # ....
  ];
  config.services.ntp = svc.ntp.build {
    pools = { "pool.ntp.org" = ["iburst"]; };
    makestep = { threshold = 1.0; limit = 3; };
  };

A full list of module options is provided later in this manual.

You most likely want to include the standard module unless you have a quite unusual use case for a very minimal system, in which case you will understand what it does and what happens if you leave it out.

imports = [
  ./modules/standard.nix
]
configuration.rootfsType = "jffs2"; # or "squashfs"

Building

Build Liminix using the default.nix in the project toplevel directory, passing it arguments for configuration and hardware. For example:

nix-build -I liminix-config=./tests/smoke/configuration.nix \
 --arg device "import ./devices/qemu" -A outputs.default

In this command <liminix-config> points to your configuration.nix, device is the file for your hardware device definition, and outputs.default will generate some kind of Liminix image output appropriate to that device.

For the qemu device in this example, outputs.default is an alias for outputs.vmbuild, which creates a directory containing a squashfs root image and a kernel. You can use the mips-vm command to run this.

For the currently supported hardware devices, outputs.default creates a directory containing a file called firmware.bin. This is a raw image file that can be written directly to the firmware flash partition.

Flashing

Flashing from the boot monitor

If you are prepared to open the device and have a TTL serial adaptor of some kind to connect it to, you can probably flash it using U-Boot. This is quite hardware-specific, and sometimes involves soldering: please refer to the Developer Manual.

Flashing from an existing Liminix system with flashcp

The flash procedure from an existing Liminix-system is two-step. First we reboot the device (using “kexec”) into an “ephemeral” RAM-based version of the new configuration, then when we’re happy it works we can flash the image - and if it doesn’t work we can reboot the device again and it will boot from the old image.

Building the RAM-based image

To create the ephemeral image, build outputs.kexecboot instead of outputs.default. This generates a directory containing the root filesystem image and kernel, along with an executable called kexec and a boot.sh script that runs it with appropriate arguments.

For example

nix-build --show-trace -I liminix-config=./examples/arhcive.nix \
  --arg device "import ./devices/gl-ar750"
  -A outputs.kexecboot && \
  (tar chf - result | ssh root@the-device tar -C /run -xvf -)

and then login to the device and run

cd /run/result
sh ./boot.sh .

This will load the new kernel and map the root filesystem into a RAM disk, then start executing the new kernel. This is effectively a reboot - be sure to close all open files and finish anything else you were doing first.

If the new system crashes or is rebooted, then the device will revert to the old configuration it finds in flash.

Building the second (permanent) image

While running in the kexecboot system, you can copy the permanent image to the device with ssh

build-machine$ tar chf - result/firmware.bin | \
 ssh root@the-device tar -C /run -xvf -

Next you need to connect to the device and locate the “firmware” partition, which you can do with a combination of dmesg output and the contents of /proc/mtd

<5>[    0.469841] Creating 4 MTD partitions on "spi0.0":
<5>[    0.474837] 0x000000000000-0x000000040000 : "u-boot"
<5>[    0.480796] 0x000000040000-0x000000050000 : "u-boot-env"
<5>[    0.487056] 0x000000050000-0x000000060000 : "art"
<5>[    0.492753] 0x000000060000-0x000001000000 : "firmware"

# cat /proc/mtd
dev:    size   erasesize  name
mtd0: 00040000 00001000 "u-boot"
mtd1: 00010000 00001000 "u-boot-env"
mtd2: 00010000 00001000 "art"
mtd3: 00fa0000 00001000 "firmware"
mtd4: 002a0000 00001000 "kernel"
mtd5: 00d00000 00001000 "rootfs"

Now run (in this example)

flashcp -v firmware.bin /dev/mtd3
“I know my new image is good, can I skip the intemediate step?”

In addition to giving you a chance to see if the new image works, this two-step process ensures that you’re not copying the new image over the top of the active root filesystem. It might work, or it might crash in surprising ways.

Flashing from OpenWrt (not currently advised!)

Caution

At your own risk! This will (at least in some circumstances) lead to bricking the device: we think this flash method is currently incompatible with use of a writeable (jffs2) filesystem.

If your device is running OpenWrt then it probably has the mtd command installed. After transferring the image onto the device using e.g. ssh, you can run it as follows:

mtd -r write /tmp/firmware.bin firmware

For more information, please see the OpenWrt manual which may also contain (hardware-dependent) instructions on how to flash an image using the vendor firmware - perhaps even from a web interface.

Updating an installed system (JFFS2)

Adding packages

If your device is running a JFFS2 root filesystem, you can build extra packages for it on your build system and copy them to the device: any package in Nixpkgs or in the Liminix overlay is available with the pkgs prefix:

nix-build -I liminix-config=./my-configuration.nix \
 --arg device "import ./devices/mydevice" -A pkgs.tcpdump

nix-shell -p min-copy-closure root@the-device result/

Note that this only copies the package to the device: it doesn’t update any profile to add it to $PATH

Rebuilding the system

liminix-rebuild is the Liminix analogue of nixos-rebuild, although its operation is a bit different because it expects to run on a build machine and then copy to the host device. Run it with the same liminix-config and device parameters as you would run nix-build, and it will build any new/changed packages and then copy them to the device using SSH. For example:

liminix-rebuild root@the-device  -I liminix-config=./examples/rotuer.nix --arg device "import ./devices/gl-ar750"

This will

  • build anything that needs building

  • copy new or changed packages to the device

  • reboot the device

It doesn’t delete old packages automatically: to do that run min-collect-garbage, which will delete any packages not in the current system closure. Note that Liminix does not have the NixOS concept of environments or generations, and there is no way back from this except for building the previous configuration again.

Caveats

  • it needs there to be enough free space on the device for all the new packages in addition to all the packages already on it - which may be a problem if a lot of things have changed (e.g. a new version of nixpkgs).

  • it cannot upgrade the kernel, only userland

Configuration options

Base options

  • option boot.commandLine

    Kernel command line

    type list of non-empty string

    default

    [ ]
  • option boot.tftp.ipaddr

    Our IP address to use when creating scripts to boot or flash from U-Boot. Not relevant in normal operation

    type string

  • option boot.tftp.loadAddress

    RAM address at which to load data when transferring via TFTP. This is not the address of the flash storage, nor the kernel load address: it should be set to some part of RAM that’s not used for anything else and suitable for temporary storage.

    type string

  • option boot.tftp.serverip

    IP address of the TFTP server. Not relevant in normal operation

    type string

  • option defaultProfile.packages

    List of packages which are available in a login shell. (This is analogous to systemPackages in NixOS, but we don’t symlink into /run/current-system, we just add the paths in /etc/profile

    type list of package

  • option filesystem

    Skeleton filesystem, represented as nested attrset. Consult the source code if you need to add to this

    type anything

  • option rootfsType

    type one of “squashfs”, “jffs2”

    default

    "squashfs"

  • option services

    type attribute set of s6-rc service

Busybox

Busybox provides stripped-down versions of many usual Linux/Unix tools, and may be configured to include only the commands (termed “applets”) required by the user or by other included modules.

  • option programs.busybox.applets

    Applets required

    type list of string

    example

    [
  "sh"
  "getty"
  "login"
]

    default

    [ ]
  • option programs.busybox.options

    Other busybox config flags that do not map directly to applet names (often prefixed FEATURE_)

    type attribute set of non-empty string

    example

    {
  FEATURE_DD_IBS_OBS = "y";
}

    default

    { }

Hardware-dependent options

These are attributes of the hardware not of the application you want to run on it, and would usually be set in the “device” file: devices/manuf-model/default.nix

  • option hardware.defaultOutput

    “Default” output: what gets built for this device when outputs.default is requested. Typically this is “flashimage” or “vmroot”

    type non-empty string

  • option hardware.dts.includes

    List of directories to search for DTS includes (.dtsi files)

    type list of path

    default

    [ ]
  • option hardware.dts.src

    Path to the device tree source (usually from OpenWrt)

    type path

  • option hardware.entryPoint

    type unspecified value

  • option hardware.flash.address

    Start address of whichever partition (often called “firmware”) we’re going to overwrite with our kernel uimage and root fs. Usually not the entire flash, as we don’t want to clobber the bootloader, calibration data etc

    type string

  • option hardware.flash.eraseBlockSize

    Flash erase block size in bytes

    type string

  • option hardware.flash.size

    Size in bytes of the firmware partition

    type string

  • option hardware.loadAddress

    type unspecified value

    default

    null

  • option hardware.networkInterfaces

    type attribute set of anything

  • option hardware.radios

    Kernel modules (from mac80211 package) required for the wireless devices on this board

    type list of string

    example

    [
  "ath9k"
  "ath10k"
]

    default

    [ ]

  • option hardware.rootDevice

    type unspecified value

hostname

  • option hostname

    System hostname of the device, as returned by gethostname(2). May or may not correspond to any name it’s reachable at on any network.

    type non-empty string

    default

    "liminix"

initramfs

  • option boot.initramfs.enable

    Whether to enable initramfs.

    type boolean

    example

    true

    default

    false

ramdisk

  • option boot.ramdisk.enable

    Whether to enable reserving part of memory as an MTD-based RAM disk. Needed for TFTP booting or for kexec-based revertable upgrade .

    type boolean

    example

    true

    default

    false

tftpboot

  • option boot.tftp.freeSpaceBytes

    type signed integer

    default

    0

Users

User- and group-related configuration.

Changes made here are reflected in files such as :file:/etc/shadow, :file:/etc/passwd, :file:/etc/group etc. If you are familiar with user configuration in NixOS, please note that Liminix does not have the concept of “mutable users” - files in /etc/ are symlinks to the immutable store, so you can’t e.g change a password with passwd

  • option groups

    type attribute set of (submodule)

  • option groups.<name>.gid

    type signed integer

  • option groups.<name>.usernames

    type list of string

    default

    [ ]

  • option users

    type attribute set of (submodule)

  • option users.<name>.dir

    type string

    default

    "/run"

  • option users.<name>.gecos

    type string

    example

    "Jo Q User"

    default

    ""

  • option users.<name>.gid

    type signed integer

  • option users.<name>.openssh.authorizedKeys.keys

    type list of string

    default

    [ ]
  • option users.<name>.passwd

    encrypted password, as generated by mkpasswd -m sha512crypt

    type string

    example

    "$6$RIYL.EgWOrtoJ0/7$Z53a8sc0o6AU/kuFOGiLJKhwVavTG/deoM7JTs6luNczYSUsh4UYmhvT8sVzm.l8F/LZXhhhkC7IHQs5UGAIM/"

    default

    "!!"

  • option users.<name>.shell

    type string

    default

    "/bin/sh"

  • option users.<name>.uid

    type signed integer

Bridge module

Allows creation of Layer 2 software “bridge” network devices. A common use case is to merge together a hardware Ethernet device with one or more WLANs so that several local devices appear to be on the same network.

path modules/bridge/default.nix

  • service system.service.bridge.members

    Service parameters

    • option members

      interfaces to add to the bridge

      type list of s6-rc service

    • option primary

      primary bridge interface

      type s6-rc service

  • service system.service.bridge.primary

    Service parameters

    • option ifname

      bridge interface name to create

      type string

Dnsmasq

This module includes a service to provide DNS, DHCP, and IPv6 router advertisement for the local network.

path modules/dnsmasq/default.nix

  • service system.service.dnsmasq

    Service parameters

    • option domain

      Domain name for DHCP service: causes the DHCP server to return the domain to any hosts which request it, and sets the domain which it is legal for DHCP-configured hosts to claim

      type string

    • option group

      Specifies the unix group which dnsmasq will run as

      type string

      default

      dnsmasq

    • option hosts

      type attribute set of (submodule)

    • option interface

      type s6-rc service

    • option ranges

      type list of string

    • option resolvconf

      type null or s6-rc service

    • option upstreams

      type list of string

    • option user

      Specifies the unix user which dnsmasq will run as

      type string

      default

      dnsmasq

Firewall

Provides a service to create an nftables ruleset based on configuration supplied to it.

path modules/firewall/default.nix

  • service system.service.firewall

    Service parameters

    • option ruleset

      firewall ruleset

      type attribute set of (attribute set)

Hostapd

Hostapd (host access point daemon) enables a wireless network interface to act as an access point and authentication server, providing IEEE 802.11 access point management, and IEEE 802.1X/WPA/WPA2/EAP Authenticators. In less technical terms, this service is what you need for your Liminix device to provide a wireless network that clients can connect to.

If you have more than one wireless network interface (e.g. wlan0, wlan1) you can run an instance of hostapd on each of them.

path modules/hostapd/default.nix

  • service system.service.hostapd

    Service parameters

    • option interface

      type s6-rc service

    • option params

      type attribute set

Mount

Mount filesystems

path modules/mount/default.nix

  • service system.service.mount

    Service parameters

    • option device

      type string

    • option fstype

      type string

      default

      auto

    • option mountpoint

      type string

    • option options

      type list of string

Network

Basic network services for creating hardware ethernet devices and adding addresses

path modules/network/default.nix

  • service system.service.network.address

    network interface address

    Service parameters

    • option address

      type string

    • option family

      type one of “inet”, “inet6”

    • option interface

      type s6-rc service

    • option prefixLength

      type integer between 0 and 128 (both inclusive)

  • service system.service.network.dhcp.client

    DHCP v4 client

    Service parameters

    • option interface

      type s6-rc service

  • service system.service.network.forward

    Service parameters

    • option enableIPv4

      type boolean

    • option enableIPv6

      type boolean

  • service system.service.network.link

    hardware network interface

    Service parameters

    • option ifname

      type string

    • option mtu

      type null or signed integer

  • service system.service.network.route

    Service parameters

    • option interface

      Interface to route through. May be omitted if it can be inferred from “via”

      type null or s6-rc service

    • option metric

      route metric

      type signed integer

    • option target

      host or network to add route to

      type string

    • option via

      address of next hop

      type string

NTP

A network time protocol implementation so that your Liminix device may synchronize its clock with an accurate time source, and optionally also provide time service to its peers. The implementation used in Liminix is Chrony

path modules/ntp/default.nix

  • service system.service.ntp

    Service parameters

    • option allow

      subnets from which NTP clients are allowed to access the server

      type list of string

    • option bindaddress

      type null or string

    • option binddevice

      type null or string

    • option dumpdir

      type path

      default

      /run/chrony

    • option extraConfig

      type strings concatenated with “\n”

      default

    • option makestep

      type null or (submodule)

    • option peers

      type attribute set of list of string

    • option pools

      type attribute set of list of string

    • option servers

      type attribute set of list of string

    • option user

      type string

      default

      ntp

PPP

A PPPoE (PPP over Ethernet) configuration to address the case where your Liminix device is connected to an upstream network using PPPoE. This is typical for UK broadband connections where the physical connection is made by OpenReach (“Fibre To The X”) and common in some other localities as well: ask your ISP if this is you.

path modules/ppp/default.nix

  • service system.service.pppoe

    Service parameters

    • option interface

      ethernet interface to run PPPoE over

      type s6-rc service

    • option ppp-options

      options supplied on ppp command line

      type list of string

Secure Shell

Provide SSH service using Dropbear

path modules/ssh/default.nix

  • service system.service.ssh

    Service parameters

    • option address

      Listen on specified address

      type null or string

    • option allowLocalPortForward

      Enable local port forwarding

      type boolean

    • option allowPasswordLogin

      Allow login using password (disable for public key auth only)

      type boolean

    • option allowPasswordLoginForRoot

      Allow root to login using password (disable for public key auth only)

      type boolean

    • option allowRemoteConnectionToForwardedPorts

      Allow remote hosts to connect to local forwarded ports (by default they are bound to loopback)

      type boolean

    • option allowRemotePortForward

      Enable remote port forwarding

      type boolean

    • option allowRoot

      Allow root to login

      type boolean

    • option extraConfig

      type strings concatenated with “ “

      default
    • option port

      Listen on specified TCP port

      type 16 bit unsigned integer; between 0 and 65535 (both inclusive)

VLAN

Virtual LANs give you the ability to sub-divide a LAN. Linux can accept VLAN tagged traffic and presents each VLAN ID as a different network interface (eg: eth0.100 for VLAN ID 100)

Some Liminix devices with multiple ethernet ports are implemented using a network switch connecting the physical ports to the CPU, and require using VLAN in order to send different traffic to different ports (e.g. LAN vs WAN)

path modules/vlan/default.nix

  • service system.service.vlan

    Service parameters

    • option ifname

      interface name to create

      type string

    • option primary

      existing physical interface

      type s6-rc service

    • option vid

      VLAN identifier (VID) in range 1-4094

      type string

Watchdog

Enable hardware watchdog (for devices that support one) and feed it by checking the health of specified critical services. If the watchdog feeder stops, the device will reboot.

path modules/watchdog/default.nix

  • service system.service.watchdog

    Service parameters

    • option headStart

      delay in seconds before watchdog starts checking service health

      type signed integer

    • option watched

      services to watch

      type list of s6-rc service