Introduction
What is Liminix?
Liminix is a Nix-based collection of software tailored for domestic wifi router or IoT devices, of the kind that OpenWrt or DD-WRT or Gargoyle or Tomato run on. It’s not NixOS on your router: we target devices that are underpowered for the full NixOS experience. It uses busybox tools, musl instead of GNU libc, and s6-rc instead of systemd.
The Liminix name comes from Liminis, in Latin the genitive declension of "limen", or "of the threshold". Your router stands at the threshold of your (online) home and everything you send to/receive from the outside word goes across it.
Where to find out more
The Manual
You are reading it now, and it is available from wherever you are reading it :-) but its canonical location is https://www.liminix.org/doc/
Source code
Liminix source code is held in git, and hosted at https://gti.telent.net/dan/liminix, with a mirror at https://github.com/telent/liminix. You can clone from either of those repos. For more on this, see Contributing.
IRC
Mailing lists
Three Liminix mailing lists are available: all are quite low volume. To subscribe to any of these lists, send an email to listname+subscribe@liminix.org. You can write anything you want in the subject and message body: only the destination address is important.
-
announce@liminix.org
for infrequent announcements from Liminix maintainers -
devel@liminix.org
for development-related discussion, patches, suggestions etc -
users@liminix.org
for help requests and general discussion
The mailing lists are managed with Mlmmj and archived with MHonArc.
Standards of behaviour
Liminix is dedicated to providing a harassment-free experience for everyone. We do not tolerate harassment of participants in any form.
The Liminix Code of Conduct applies to all Liminix spaces, including the IRC channel, mailing lists, and any other forums, both online and off. Anyone who violates the code of conduct may be sanctioned or expelled from these spaces at the discretion of the project leadership.
Tutorial
Liminix is very configurable, which can make it initially quite daunting - especially if you’re learning Nix or Linux or networking concepts at the same time. In this section we build some "worked example" Liminix images to introduce the concepts. If you follow the examples exactly, they should work. If you change things as you go along, they may work differently or not at all, but the experience should be educational either way.
Requirements
You will need a reasonably powerful computer running Nix. Target devices for Liminix are unlikely to have the CPU power and disk space to be able to build it in situ, so the build process is based around "cross-compilation" from another computer. The build machine can be any reasonably powerful desktop/laptop/server PC running NixOS. Standalone Nixpkgs installations on other Linux distributions - or on MacOS, or even in a Docker container - also ought to work but are untested.
Running in Qemu
You can try out Liminix without even having a router to play with. Clone the Liminix git repository and change into its directory
git clone https://gti.telent.net/dan/liminix
cd liminix
Now build Liminix
nix-build -I liminix-config=./examples/hello-from-qemu.nix \
--arg device "import ./devices/qemu" -A outputs.default
In this command liminix-config
points to the desired software
configuration (e.g. services, users, filesystem, secrets) and device
describes the hardware (or emulated hardware) to run it on.
outputs.default
tells Liminix that we want the default image output
for flashing to the device: for the Qemu "hardware" it’s an alias for
outputs.vmbuild
, which creates a directory containing a root
filesystem image and a kernel.
Tip
|
The first time you run this it may take several hours, because it builds all of the dependencies including a full MIPS gcc and library toolchain. Once those intermediate build products are in the nix store, subsequent builds will be much faster - practically instant, if nothing has changed. |
Now you can try it:
./result/run.sh
This starts the Qemu emulator with a bunch of useful options, to run the Liminix configuration you just built. It connects the emulated device’s serial console and the QEMU monitor to stdin/stdout.
You should now see Linux boot messages and after a few seconds be
presented with a root shell prompt. You can run commands to look at the
filesystem, see what processes are running, view log messages (in
:file:/run/log/current), etc. To kill the emulator, press ^P (Control P)
then c to enter the "QEMU Monitor", then type quit
at the (qemu)
prompt.
To see that it’s running network services we need to connect to its emulated network. Start the machine again, if you had stopped it, and open up a second terminal on your build machine. We’re going to run another virtual machine attached to the virtual network, which will request an IP address from our Liminix system and give you a shell you can run ssh from.
We use System Rescue in tty mode (no graphical output) for this example, but if you have some other favourite Linux Live CD ISO - or, for that matter, any other OS image that QEMU can boot - adjust the command to suit.
Download the System Rescue ISO:
curl https://fastly-cdn.system-rescue.org/releases/10.01/systemrescue-10.01-amd64.iso -O
and run it
nix-shell -p qemu --run " \
qemu-system-x86_64 \
-echr 16 \
-m 1024 \
-cdrom systemrescue-10.01-amd64.iso \
-netdev socket,mcast=230.0.0.1:1235,localaddr=127.0.0.1,id=lan \
-device virtio-net,disable-legacy=on,disable-modern=off,netdev=lan,mac=ba:ad:3d:ea:21:01 \
-display none -serial mon:stdio"
System Rescue displays a boot menu at which you should select the "serial console" option, then after a few moments it boots to a root prompt. You can now try things out:
-
run
ip a
and see that it’s been allocated an IP address in the range 10.3.0.0/16. -
run
ping 10.3.0.1
to see that the Liminix VM responds -
run
ssh root@10.3.0.1
to try logging into it.
Congratulations! You have installed your first Liminix system - albeit it has no practical use and it’s not even real. The next step is to try running it on hardware.
Installing on hardware
For the next example, we’re going to install onto an actual hardware device. These steps have been tested using a GL.iNet GL-MT300A, which has been chosen for the purpose because it’s cheap and easy to unbrick if necessary.
Warning
|
There is always a risk of rendering your device unbootable by flashing it with an image that doesn’t work. The GL-MT300A has a builtin "debrick" procedure in the boot monitor and is also comparatively simple to attach serial cables to (soldering not required), so it is lower-risk than some devices. Using some other Liminix-supported MIPS hardware device also ought to work here, but you accept the slightly greater bricking risk if it doesn’t. See Supported hardware for device support status. |
You may want to read and inwardly digest the section on Serial connections when you start working with Liminix on real hardware. You won’t need serial access for this example, assuming it works, but it allows you to see the boot monitor and kernel messages, and to login directly to the device if for some reason it doesn’t bring its network up.
Now we can build Liminix. Although we could use the same example
configuration as we did for Qemu, you might not want to plug a DHCP
server into your working LAN because it will compete with the real DHCP
service. So we’re going to use a different configuration with a DHCP
client: this is examples/hello-from-mt300.nix
It’s instructive to compare the two configurations:
diff -u examples/hello-from-qemu.nix examples/hello-from-mt300.nix
You’ll see a new boot.tftp
stanza which you can ignore,
services.dns
has been removed, and the static IP address allocation
has been replaced by a dhcp.client
service.
nix-build -I liminix-config=./examples/hello-from-mt300.nix \
--arg device "import ./devices/gl-mt300a" -A outputs.default
Tip
|
The first time you run this it may take several hours. Again? Yes, even if you ran the previous example. Qemu is set up as a big-endian system whereas the MediaTek SoC on this device is little-endian - so it requires building all of the dependencies including an entirely different MIPS gcc and library toolchain to the other one. |
This time in result/
you will see a bunch of files. Most of them you
can ignore for the moment, but result/firmware.bin
is the firmware
image you can flash.
Flashing
Again, there are a number of different ways you could do this: using TFTP with a serial cable, through the stock firmware’s web UI, or using the vendor’s "debrick" process. The last of these options has a lot to recommend it for a first attempt:
-
it works no matter what firmware is currently installed
-
it doesn’t require plugging a router into the same network as your build system and potentially messing up your actual upstream
-
no need to open the device and add cables
You can read detailed instructions on the vendor site, but the short version is:
-
turn the device off
-
connect it by ethernet cable to a computer
-
configure the computer to have static ip address 192.168.1.10
-
while holding down the Reset button, turn the device on
-
after about five seconds you can release the Reset button
-
visit http://192.168.1.1/ using a web browser on the connected computer
-
click on "Browse" and choose
result/firmware.bin
-
click on "Update firmware"
-
wait a minute or so while it updates.
There’s no feedback from the web interface when the flashing is
finished, but what should happen is that the router reboots and starts
running Liminix. Now you need to figure out what address it got from
DHCP - e.g. by checking the DHCP server logs, or maybe by pinging
hello.lan
or something. Once you’ve found it on the network you can
ping it and ssh to it just like you did the Qemu example, but this time
for real.
Warning
|
Do not leave the default root password in place on any device exposed to the internet! Although it has no writable storage and no default route, a motivated attacker with some imagination could probably still do something awful using it. |
Congratulations Part II! You have installed your first Liminix system on actual hardware - albeit that it still has no practical use.
Exercise for the reader: change the default password by editing
examples/hello-from-mt300.nix
, and then create and upload a new
image that has it set to something less hopeless.
Routing
The third example examples/demo.nix
is a fully-functional home "WiFi
router" - although you will have to edit it a bit before it will
actually work for you. Copy examples/demo.nix
to my-router.nix
(or other name of your choice) and open it in your favourite text
editor. Everywhere that the text EDIT
appears is either a place you
probably want to change or a place you almost certainly need to change.
There’s a lot going on in this configuration:
-
it provides a wireless access point using the
hostapd
service: in this stanza you can change the ssid, the channel, the passphrase etc. -
the wireless lan and wired lan are bridged together with the
bridge
service, so that your wired and wireless clients appear to be on the same network.
Tip
|
If you were using a hardware device that provides both 2.4GHz and 5GHz
wifi, you’d probably find that it has two wireless devices (often called
wlan0 and wlan1). In Liminix we handle this by running two |
-
we use the combination DNS and DHCP daemon provided by the
dnsmasq
service, which you can configure -
the upstream network is "PPP over Ethernet", provided by the
pppoe
service. Assuming that your ISP uses this standard, they will have provided you with a PPP username and password (sometimes this will be listed as "PAP" or "CHAP") which you can edit into the configuration -
this example supports the new[1] Internet Protocol v6 as well as traditional IPv4. Configuring IPv6 seems to vary from one ISP to the next: this example expects them to be providing IP address allocation and "prefix delegation" using DHCP6.
Build it using the same method as the previous example
nix-build -I liminix-config=./my-router.nix \
--arg device "import ./devices/gl-mt300a" -A outputs.default
and then you can flash it to the device.
Bonus: in-place updates
This configuration uses a writable filesystem (see the line
rootfsType = "jffs2"
), which means that once you’ve flashed it for
the first time, you can make further updates over SSH onto the running
router. To try this, make a small change (I’d suggest changing the
hostname) and then run
nix-build -I liminix-config=./my-router.nix \
--arg device "import ./devices/gl-ar750" \
-A outputs.systemConfiguration && \
result/install.sh root@address-of-the-device
(This requires the device to be network-accessible from your build machine, which for a test/demo system might involve a second network device in your build system - USB ethernet adapters are cheap - or a bit of messing around unplugging cables.)
For more information about in-place-updates, see the manual section
Rebuilding the system
.
Final thoughts
-
These are demonstration configs for pedagogical purposes. If you’d like to see some more realistic uses of Liminix,
examples/rotuer,arhcive,extneder.nix
are based on some actual real hosts in my home network. -
The technique used here for flashing was chosen mostly because it doesn’t need much infrastructure/tooling, but it is a bit of a faff (requires physical access, vendor specific). There are slicker ways to do it that need a bit more setup - we’ll talk about that later as well.
Footnotes
Installation
Hardware devices vary wildly in their affordances for installing new operating systems, so it should be no surprise that the Liminix installation procedure is hardware-dependent. This section contains generic instructions, but please refer to the documentation for your device to find whether and how well they apply.
Most of the supported devices fall into one of two broad categories:
-
devices we install by preparing a raw flash image and copying it directly onto (some part of) the flash. This is analogous to (though not quite the same as) using https://www.man7.org/linux/man-pages/man1/dd.1.html:dd(1) on a "grown up" computer to copy a raw disk image. Devices in this category are usually smaller, older, and/or less powerful.
-
devices where the vendor provides a "higher level" storage abstraction, such as http://linux-mtd.infradead.org/doc/ubi.html:UBI over raw flash, or a consumer flash such as MMC, or another storage technology entirely. Installation on these devices is less uniform because it depends on exactly what kind of storage abstraction.
Building a firmware image
Liminix uses the Nix language to provide congruent configuration
management. This means that to change anything about the way in which a
Liminix system works, you make that change in your configuration.nix
(or one of the other files it references), and rerun nix-build
to
action the change. It is not possible (at least, without shenanigans) to
make changes by logging into the device and running imperative commands
whose effects may later be overridden: configuration.nix
always
describes the entire system and can be used to recreate that system at
any time. You can usefully keep it under version control.
If you are familiar with NixOS, you will notice some similarities between NixOS and Liminix configuration, and also some differences. Sometimes the differences are due to the resource-constrained devices we deploy onto, sometimes due to differences in the uses these devices are put to.
For a more full description of how to configure Liminix, see
Configuration. Assuming for the moment that you want a typical home
wireless gateway/router, the best way to get started is to copy
examples/rotuer.nix
and edit it for your requirements.
$ cp examples/rotuer.nix configuration.nix
$ vi configuration.nix # other editors are available
$ # adjust this next command for your hardware device
$ nix-build -I liminix-config=./configuration.nix \
--arg device "import ./devices/gl-mt300a" -A outputs.default
For raw flash devices, this will leave you with a file
result/firmware.bin
which you now need to write to the flash.
For other devices, please check the device documentation
Flashing from the boot monitor (TFTP install)
You will need
-
to open the device and attach a TTL serial adaptor of some kind
-
a TFTP server on the network that the device is plugged into (or can be plugged into for installation)
Installing via serial connection is quite hardware-specific and depending on the device may even involve soldering. However, it is in some ways the most "reliable" option: if you can see what’s happening (or not happening) in early boot, the risk of "bricking" is substantially reduced and you have options for recovering if you misstep or flash a bad image.
Serial connections
To speak to U-Boot on your device you’ll usually need a serial connection to it. This typically involves opening the box, locating the serial header pins (TX, RX and GND) and connecting a USB TTL converter to them.
The Rolls Royce of USB/UART cables is the FTDI cable, but there are cheaper alternatives based on the PL2303 and CP2102 chipsets - or you could even get creative and use the UART GPIO pins on a Raspberry Pi. Whatever you do, make sure that the voltages are compatible: if your device is 3.3V (this is typical but not universal), you don’t want to be sending it 5v or (even worse) 12v.
Run a terminal emulator such as Minicom on the computer at other end of the link. 115200 8N1 is the typical speed.
Note
|
TTL serial connections often have no flow control and so don’t always like having massive chunks of text pasted into them - and U-Boot may drop characters while it’s busy. So don’t do that. If using Minicom, you may find it helps to bring up the "Termimal settings" dialog (C^A T), then configure "Newline tx delay" to some small but non-zero value. |
When you turn the router on you should be greeted with some messages from U-Boot, followed by the instruction to hit some key to stop autoboot. Do this and you will get to the prompt. If you didn’t see anything, the strong likelihood is that TX and RX are the wrong way around, or your computer is expecting flow control which the 3 wire connection does not provide. If you see garbage, try a different speed.
Interesting commands to try first in U-Boot are help
and
printenv
.
TFTP
You will also need to configure a TFTP server on a network that’s accessible to the device: how you do that will vary according to which TFTP server you’re using and so is out of scope for this document.
HINT: [tufted], a rudimentary TFTP server, is supplied with Liminix for development purposes. It may or may not fit your needs here.
Building and installing the image
Follow the device-specific instructions for "TFTP install": usually, the steps are
-
build the outputs.mtdimage output
-
copy
result/firmware.bin
to wherever your TFTP server serves files from -
execute the commands listed in
result/flash.scr
at the U-Boot command line -
reset the device
You should now see messages from U-Boot, then from the Linux kernel and eventually a shell prompt.
Note
|
Before you reboot, check which networks the device is plugged into, and disconnect as necessary. If you’ve just installed a DHCP server or anything else that responds to broadcasts, you may not want it to do that on the network that you temporarily connected it to for installing it. |
Flashing from OpenWrt
Caution
|
Untested! A previous version of these instructions (without the -e flag) led to soft-bricking the device when flashing a JFFS2 image. The current version should work better but if you are reading this message then nobody has yet confirmed it |
If your device is running OpenWrt then it probably has the mtd
command installed. Transfer result/firmware.bin
onto the running
device using e.g. scp
. Now flash as follows:
mtd -e -r write /tmp/firmware.bin firmware
The options to this command are for "erase before writing" and "reboot after writing".
For more information, please see the OpenWrt manual which may also contain (hardware-dependent) instructions on how to flash an image using the vendor firmware - perhaps even from a web interface.
Flashing from Liminix
If the device is already running Liminix then in general you cannot safely copy a new image over the top of the running system while it is running.
If the running system was configured with Levitate you can use
that to safely flash your new image. Otherwise you may attempt to use
flashcp
directly, but bust out the serial leads in preparation for
it going horribly wrong.
For Administrators
Configuration
There are many things you can specify in a configuration, but most commonly you need to change:
-
which services (processes) to run
-
what packages to install
-
permitted users and groups
-
Linux kernel configuration options
-
Busybox applets
-
filesystem layout
Modules
Modules are a means of abstraction which allow "bundling" of
configuration options related to a common purpose or theme. For example,
the dnsmasq
module defines a template for a dnsmasq service, ensures
that the dnsmasq package is installed, and provides a dnsmasq user and
group for the service to run as. The ppp
module defines a service
template and also enables various PPP-related kernel configuration.
Not all modules are included in the configuration by default, because
that would mean that the kernel (and the Busybox binary providing common
CLI tools) was compiled with many unnecessary bells and whistles and
therefore be bigger than needed. (This is not purely an academic concern
if your device has little flash storage). Therefore, specifying a
service is usually a two-step process. For example, to add an NTP
service you first add modules/ntp
to your imports
list, then you
create a service by calling config.system.service.ntp.build { .... }
with the appropriate service-dependent configuration parameters.
let svc = config.system.service;
in {
# ...
imports = [
./modules/ntp
# ....
];
config.services.ntp = svc.ntp.build {
pools = { "pool.ntp.org" = ["iburst"]; };
makestep = { threshold = 1.0; limit = 3; };
};
Merely including the module won’t define the service on its own: it only
creates the template in config.system.service.foo
and you have to
create an actual service using the template. This is an intentional
choice to allow the creation of multiple differently-configured services
based on the same template - perhaps e.g. when you have multiple
networks (VPNs etc) in different trust domains, or you want to run two
SSH daemons on different ports. (For the background to this, please
refer to the architecture decision record <adr/module-system>
)
Tip
|
Liminix modules should be quite familiar (but also different) if you already know how to use NixOS modules. We use the NixOS module infrastructure code, meaning that you should recognise the syntax, the type system, the rules for combining configuration values from different sources. We don’t use the NixOS modules themselves, because the underlying system is not similar enough for them to work. |
Services
In Liminix a service is any kind of long-running task or process on the system, that is managed (started, stopped, and monitored) by a service supervisor. A typical SOHO router might have services to
-
answer DHCP and DNS requests from the LAN
-
provide a wireless access point
-
connect using PPPoE or L2TP to an upstream network
-
start/stop the firewall
-
enable/disable IP packet forwarding
-
mount filesystems
(Some of these might not be considered services using other definitions of the term: for example, this L2TP process would be a "client" in the client/server classification; and enabling packet forwarding doesn’t require any long-lived process - just a setting to be toggled. However, there is value in being able to use the same abstractions for all the things to manage them and specify their dependency relationships - so in Liminix "everything is a service")
The service supervision system enables service health monitoring, restart of unhealthy services, and failover to "backup" services when a primary service fails or its dependencies are unavailable. The intention is that you have a framework in which you can specify policy requirements like "ethernet wan dhcp-client should be restarted if it crashes, but if it can’t start because the hardware link is down, then 4G ppp service should be started instead".
Any attribute in config.services will become part of the default set of services that s6-rc will try to bring up. Services are usually started at boot time, but controlled services are those that are required only in particular contexts. For example, a service to mount a USB backup drive should run only when the drive is attached to the system. Liminix currently implements three kinds of controlled service:
-
"uevent-rule" service controllers use sysfs/uevent to identify when particular hardware devices are present, and start/stop a controlled service appropriately.
-
the "round-robin" service controller is used for service failover: it allows you to specify a list of services and runs each of them in turn until it exits, then runs the next.
-
the "health-check" service wraps another service, and runs a "health check" command at regular intervals. When the health check fails, indicating that the wrapped service is not working, it is terminated and allowed to restart.
Runtime secrets (external vault)
Secrets (such as wifi passphrases, PPP username/password, SSH keys, etc)
that you provide as literal values in configuration.nix
are
processed into into config files and scripts at build time, and
eventually end up in various files in the (world-readable)
/nix/store
before being baked into a flashable image. To change a
secret - whether due to a compromise, or just as part of to a routine
key rotation - you need to rebuild the configuration and potentially
reflash the affected devices.
To avoid this, you may instead use a "secrets service", which is a mechanism for your device to fetch secrets from a source external to the Nix store, and create at runtime the configuration files and scripts that start the services which require them.
Not every possible parameter to every possible service is configurable
using a secrets service. Parameters which can be configured this way are
those with the type liminix.lib.types.replacable
. At the time this
document was written, these include:
-
ppp (pppoe and l2tp):
username
,password
-
ssh:
authorizedKeys
-
hostapd: all parameters (most likely to be useful for
wpa_passphrase
)
To use a runtime secret for any of these parameters:
-
create a secrets service to specify the source of truth for secrets
-
use the
outputRef
function in the service parameter to specify the secrets service and path
For example, given you had an HTTPS server hosting a JSON file with the structure
"ssh": {
"authorizedKeys": {
"root": [ "ssh-rsa ....", "ssh-rsa ....", ... ]
"guest": [ "ssh-rsa ....", "ssh-rsa ....", ... ]
}
}
you could use a configuration.nix
fragment something like this to
make those keys visible to ssh:
services.secrets = svc.secrets.outboard.build {
name = "secret-service";
url = "http://10.0.0.1/secrets.json";
username = "secrets";
password = "liminix";
interval = 30; # minutes
dependencies = [ config.services.lan ];
};
services.sshd = svc.ssh.build {
authorizedKeys = outputRef config.services.secrets "ssh/authorizedKeys";
};
There are presently two implementations of a secrets service:
Outboard secrets (HTTPS)
This service expects a URL to a JSON file containing all the secrets.
You may specify a username and password along with the URL, which are used if the file is password-protected (HTTP Basic authentication). Note that this is not a protection against a malicious local user: the username and password are normal build-time parameters so will be readable in the Nix store. This is a mitigation against the URL being accidentally discovered due to e.g. a log file or error message on the server leaking.
Tang secrets (encrypted local file)
Aternatively, secrets may be stored locally on the device, in a file that has been encrypted using Tang.
Tang is a server for binding data to network presence.
This sounds fancy, but the concept is simple. You have some data, but you only want it to be available when the system containing the data is on a certain, usually secure, network.
services.secrets = svc.secrets.tang.build {
name = "secret-service";
path = "/run/mnt/usbstick/secrets.json.jwe";
interval = 30; # minutes
dependencies = [ config.services.mount-usbstick ];
};
The encryption uses the same scheme/algorithm as
Clevis : you may use the
Clevis
instructions to encrypt the file on another host and then copy it to
your Liminix device, or you can use tangc encrypt
to encrypt
directly on the device. (That latter approach may pose a chicken/egg
problem if the device needs secrets to boot up and run the services you
are relying on in order to login).
System Administration
Services on a running system
Liminix services are built on s6-rc, which is itself layered on s6. Services are defined at build time in your configuration (see Services for information) and can’t be added to/changed at runtime, but to monitor events or diagnose problems you may need to inspect them on the running system. Here are some of the most commonly used s6,-rc commands:
What | How |
---|---|
List all running services |
|
List all services that are not running |
|
List services that |
|
… transitively |
|
List services that depend on service |
|
… transitively |
|
Stop service |
|
Start service |
|
Start service |
|
s6-rc-up-tree
brings up a service and all services that depend on
it, except for any services that depend on a "controlled" service that
is not currently running. Controlled services are not started at boot
time but in response to external events (e.g. plugging in a particular
piece of hardware) so you probably don’t want to be starting them by
hand if the conditions aren’t there.
A service may be up or down (there are no intermediate states like
"started" or "stopping" or "dying" or "cogitating"). Some (but not all)
services have "readiness" notifications: the dependents of a service
with a readiness notification won’t be started until the service signals
(by writing to a nominated file descriptor) that it’s prepared to start
work. Most services defined by Liminix also have a timeout-up
parameter, which means that if a service has readiness notifications and
doesn’t become ready in the allotted time (defaults 20 seconds) it will
be terminated and its state set to down.
If the process providing a service dies, it will be restarted automatically. Liminix does not automatically set it to down.
(If the process providing a service dies without ever notifying readiness, Liminix will restart it as many times as it has to until the timeout period elapses, and then stop it and mark it down.)
Controlled services
Controlled services are those which are started/stopped on demand by a controller (another service) instead of being started at boot time. For example:
-
svc.uevent-rule.build
creates a controlled service which is active when a particular hardware device (identified by uevent/sysfs directory) is present. -
svc.round-robin.build
creates a service controller that invokes two or more services in turn, running the next one when the process providing the previous one exits. We use this for failover from one network connection to a backup connection, for example. -
svc.health-check.build
creates a service controller that runs a controlled service and periodically tests whether it is healthy by running an external health check command or script. If the check command repeatedly fails, the controlled service is restarted.The Configuration section of the manual describes controlled services in more detail. Some operational considerations
-
round-robin
detects a service status by looking at itsoutputs
directory, so it won’t work unless the service creates some outputs. This is considered a bug and will be fixed in a future release -
health-check
works for longruns but not for oneshots, as it internally relies ons6-svc
to restart the process
Logs
Logs for all services are collated into /run/log/current
. The log
file is rotated when it reaches a threshold size, into another file in
the same directory whose name contains a TAI64 timestamp.
Each log line is prefixed with a TAI64 timestamp and the name of the
service, if it is a longrun. If it is a oneshot, a timestamp and the
name of some other service. To convert the timestamp into a
human-readable format, use s6-tai64nlocal
.
# ls -l /run/log/
-rw-r--r-- 1 0 lock
-rw-r--r-- 1 0 state
-rwxr--r-- 1 98059 @4000000000025cb629c311ac.s
-rwxr--r-- 1 98061 @40000000000260f7309c7fb4.s
-rwxr--r-- 1 98041 @40000000000265233a6cc0b6.s
-rwxr--r-- 1 98019 @400000000002695d10c06929.s
-rwxr--r-- 1 98064 @4000000000026d84189559e0.s
-rwxr--r-- 1 98055 @40000000000271ce1e031d91.s
-rwxr--r-- 1 98054 @400000000002760229733626.s
-rwxr--r-- 1 98104 @4000000000027a2e3b6f4e12.s
-rwxr--r-- 1 98023 @4000000000027e6f0ed24a6c.s
-rw-r--r-- 1 42374 current
# tail -2 /run/log/current
@40000000000284f130747343 wan.link.pppoe Connect: ppp0 <--> /dev/pts/0
@40000000000284f230acc669 wan.link.pppoe sent [LCP ConfReq id=0x1 <asyncmap 0x0> <magic 0x667a9594> <pcomp> <accomp>]
# tail -2 /run/log/current | s6-tai64nlocal
1970-01-02 21:51:45.828598156 wan.link.pppoe sent [LCP ConfReq id=0x1 <asyncmap 0x0> <magic 0x667a9594> <pcomp> <accom
p>]
1970-01-02 21:51:48.832588765 wan.link.pppoe sent [LCP ConfReq id=0x1 <asyncmap 0x0> <magic 0x667a9594> <pcomp> <accom
p>]
Log persistence
Logs written to /run/log/
will not survive a reboot or crash, as it
is an ephemeral filesystem.
On supported hardware you can enable logging to
pstore
which means the most recent log messages will be preserved on reboot.
Set the config option logging.persistent.enable = true
to log
messages to /dev/pmsg0
as well as to the regular log. This is a
circular buffer, so when it fills up newer messages will overwrite the
oldest messages.
Logs found in pstore after a reboot will be moved at startup to
/run/log/previous-boot
Updating an installed system
If your system has a writable root filesystem (JFFS2, btrfs etc
-anything but squashfs), we have mechanisms for in-places updates
analogous to nixos-rebuild
, but the operation is a bit different
because it expects to run on a build machine and then copy to the host
device using ssh
.
To use this, build the outputs.updater
target and then run the
update.sh
script it generates.
nix-build -I liminix-config=./my-configuration.nix \
--arg device "import ./devices/mydevice" \
-A outputs.updater
./result/bin/update.sh root@the-device
The update script uses min-copy-closure to copy new or changed packages to the device, then (perhaps) reboots it. The reboot behaviour can be affected by flags:
-
--no-reboot will cause it not to reboot at all, if you would rather do that yourself. Note that none of the newly-installed or updated services will be running until you do.
-
--fast causes it tn not do a full reboot, but instead to restart only the services that have been changed. This will restart all of the services that have updated store paths (and anything that depends on them), but will not affect services that haven’t changed.
It doesn’t delete old packages automatically: to do that run
min-collect-garbage
, which will delete any packages not in the
current system closure. Note that Liminix does not have the NixOS
concept of environments or generations, and there is no way back from
this except for building the previous configuration again.
Caveats
-
it needs there to be enough free space on the device for all the new packages in addition to all the packages already on it - which may be a problem if there is little flash storage or if a lot of things have changed (e.g. a new version of nixpkgs).
-
it may not be able to upgrade the kernel: this is device-dependent. If your device boots from a kernel image on a raw MTD partition or or UBI volume, update.sh is unable to alter the kernel partition. If your device boots from a kernel inside the filesystem (e.g. using bootloader.extlinux or bootloder.fit) then the kernel will be upgraded along with the userland
Recovery/downgrades
The update.sh
script also creates a timestamped symlink on the
device which points to the system configuration it installs. If you
install a configuration that doesn’t work, you can revert to any other
installed configuration by
-
booting to some kind of rescue or recovery system (which may be some vendor-provided rescue option, or your own recovery system perhaps based on
examples/recovery.nix
) and mounting your Liminix filesystem on /mnt -
picking another previously-installed configuration that _did work, and switching back to it:
# ls -ld /mnt/*configuration
lrwxrwxrwx 1 90 /mnt/20252102T182104.configuration -> nix/store/v1w0h4zw65ah4c2r0k7nyy125qrxhq78-system-configuration-aarch64-unknown-linux-musl
lrwxrwxrwx 1 90 /mnt/20251802T181822.configuration -> nix/store/wqjl9s9xljl2wg8257292zghws9ssidk-system-configuration-aarch64-unknown-linux-musl
# : 20251802T181822 is the working system, so reinstall it
# /mnt/20251802T181822.configuration/bin/install /mnt
# umount /mnt
# reboot
This will install the previous configuration’s activation binary into /bin, and copy its kernel and initramfs into /boot. Note that it depends on the previous system not having been garbage-collected.
Adding packages
If you simply wish to add a package without any change to services, you
can call min-copy-closure
directly to install any package in Nixpkgs
or in the Liminix overlay
nix-build -I liminix-config=./my-configuration.nix \
--arg device "import ./devices/mydevice" -A pkgs.tcpdump
nix-shell -p min-copy-closure root@the-device result/
Note that this only copies the package and its dependencies to the
device: it doesn’t update any profile to add it to $PATH
Levitate: Reinstalling on a running system
Liminix is initially installed from a monolithic firmware.bin
- and
unless you’re running a writable filesystem, the only way to update it
is to build and install a whole new firmware.bin
. However, you
probably would prefer not to have to remove it from its installation
site, unplug it from the network and stick serial cables in it all over
again.
It is not (generally) safe to install a new firmware onto the flash
partitions that the active system is running on. To address this we have
levitate
, which a way for a running Liminix system to "soft restart"
into a ramdisk running only a limited set of services, so that the main
partitions can then be safely flashed.
Configuration
Levitate needs to be configured when you create the initial system to specify which services/packages/etc to run in maintenance mode. Most likely you want to configure a network interface and an ssh for example so that you can login to reflash it.
defaultProfile.packages = with pkgs; [
...
(levitate.override {
config = {
services = {
inherit (config.services) dhcpc sshd watchdog;
};
defaultProfile.packages = [ mtdutils ];
users.root = config.users.root;
};
})
];
Use
Connect (with ssh, probably) to the running Liminix system that you wish to upgrade.
bash$ ssh root@the-device
Run levitate
. This takes a little while (perhaps a few tens of
seconds) to execute, and copies all config required for maintenance mode
to /run/maintenance
.
# levitate
Reboot into maintenance mode. You will be logged out
# reboot
Connect to the device again - note that the ssh host key will have changed.
# ssh -o UserKnownHostsFile=/dev/null root@the-device
Check we’re in maintenance mode
# cat /etc/banner
LADIES AND GENTLEMEN WE ARE FLOATING IN SPACE
Most services are disabled. The system is operating
with a ram-based root filesystem, making it safe to
overwrite the flash devices in order to perform
upgrades and maintenance.
Don't forget to reboot when you have finished.
Perform the upgrade, using flashcp. This is an example, your device will differ
# cat /proc/mtd
dev: size erasesize name
mtd0: 00030000 00010000 "u-boot"
mtd1: 00010000 00010000 "u-boot-env"
mtd2: 00010000 00010000 "factory"
mtd3: 00f80000 00010000 "firmware"
mtd4: 00220000 00010000 "kernel"
mtd5: 00d60000 00010000 "rootfs"
mtd6: 00010000 00010000 "art"
# flashcp -v firmware.bin mtd:firmware
All done
# reboot
For Developers
In any Nix-based system the line between "configuration" and "development" is less of a line and more of a continuum. This section covers some topics further towards the latter end.
Writing modules and services
It helps here to know NixOS! Liminix uses the NixOS module infrastructure code, meaning that everything that has been written for NixOS about the syntax, the type system, and the rules for combining configuration values from different sources is just as applicable here.
Services
For the most part, for common use cases, we hope that Liminix modules
provide service templates for all the services you will need, and you
will only have to pass the right parameters to build
.
But if you’re reading this then our hopes are in vain. To create a custom service of your own devising, use the oneshot or longrun functions:
-
a "longrun" service is the "normal" service concept: it has a
run
action which describes the process to start, and it watches that process to restart it if it exits. The process should not attempt to daemonize or "background" itself, otherwise s6-rc will think it died. Whatever it prints to standard output/standard error will be logged.
config.services.cowsayd = pkgs.liminix.services.longrun {
name = "cowsayd";
run = "${pkgs.cowsayd}/bin/cowsayd --port 3001 --breed hereford";
# don't start this until the lan interface is ready
dependencies = [ config.services.lan ];
}
-
a "oneshot" service doesn’t have a process attached. It consists of
up
anddown
actions which are bits of shell script that are run at the appropriate points in the service lifecycle
config.services.greenled = pkgs.liminix.services.oneshot {
name = "greenled";
up = ''
echo 17 > /sys/class/gpio/export
echo out > /sys/class/gpio/gpio17/direction
echo 0 > /sys/class/gpio/gpio17/value
'';
down = ''
echo 0 > /sys/class/gpio/gpio17/value
'';
}
Services may have dependencies: as you see above in the cowsayd
example, it depends on some service called config.services.lan
,
meaning that it won’t be started until that other service is up.
Service outputs
Outputs are a mechanism by which a service can provide data which may be required by other services. For example:
-
the DHCP client service can expect to receive nameserver address information as one of the fields in the response from the DHCP server: we provide that as an output which a dependent service for a stub name resolver can use to configure its upstream servers.
-
a service that creates a new network interface (e.g. ppp) will provide the name of the interface (
ppp0
, orppp1
orppp7
) as an output so that a dependent service can reference it to set up a route, or to configure firewall rules.
A service myservice
should write its outputs as files in
/run/services/outputs/myservice
: you can look around this directory
on a running Liminix system to see how it’s used currently. Usually we
use the in_outputs
shell function in the up
or run
attributes of the service:
(in_outputs ${name}
for i in lease mask ip router siaddr dns serverid subnet opt53 interface ; do
(printenv $i || true) > $i
done)
The outputs are just files, so technically you can read them using anything that can read a file. Liminix has two "preferred" mechanisms, though:
One-off lookups
In any context that ends up being evaluated by the shell, use output
to print the value of an output
services.defaultroute4 = svc.network.route.build {
via = "$(output ${services.wan} address)";
target = "default";
dependencies = [ services.wan ];
};
Continuous updates
The downside of using shell functions in downstream service startup scripts is that they only run when the service starts up: if a service output changes, the downstream service would have to be restarted to notice the change. Sometimes this is OK but other times the downstream has no other need to restart, if it can only get its new data.
For this case, there is the anoia.svc
Fennel library, which allows
you to write a simple loop which is iterated over whenever a service’s
outputs change. This code is from
modules/dhcp6c/acquire-wan-address.fnl
(fn update-addresses [wan-device addresses new-addresses exec]
;; run some appropriate "ip address [add|remove]" commands
)
(fn run []
(let [[state-directory wan-device] arg
dir (svc.open state-directory)]
(accumulate [addresses []
v (dir:events)]
(update-addresses wan-device addresses
(or (v:output "address") []) system))))
The output
method seen here accepts a filename (relative to the
service’s output directory), or a directory name. It returns the first
line of that file, or for directories it returns a table (Lua’s
key/value datastructure, similar to a hash/dictionary) of the outputs in
that directory.
Design considerations for outputs
For preference, outputs should be short and simple, and not require downstream services to do complicated parsing in order to use them. Shell commands in Liminix are run using the Busybox shell which doesn’t have the niceties evel of an advanced shell like Bash, let alone those of a real programming language.
Note also that the Lua svc
library only reads the first line of each
output.
Modules
Modules in Liminix conventionally live in
modules/somename/default.nix
. If you want or need to write your own,
you may wish to refer to the examples there in conjunction with reading
this section.
A module is a function that accepts {lib, pkgs, config, ... }
and
returns an attrset with keys imports, options, config
.
-
imports
is a list of paths to the other modules required by this one -
options
is a nested set of option declarations -
config
is a nested set of option definitions
The NixOS manual section Writing NixOS Modules is a quite comprehensive reference to writing NixOS modules, which is also mostly applicable to Liminix except that it doesn’t cover service templates.
Service templates
Although you can define services "ad hoc" using longrun
or oneshot
as above, this approach has limitations if
you’re writing code intended for wider use. Services in the
modules bundled with Liminix are implemented following a pattern we
call "service templates": functions that accept a type-checked
attrset and return an appropriately configured service that can be
assigned by the caller to a key in config.services
.
To expose a service template in a module, it needs the following:
-
an option declaration for
system.service.myservicename
with the type ofliminix.lib.types.serviceDefn
options = {
system.service.cowsay = mkOption {
type = liminix.lib.types.serviceDefn;
};
};
-
an option definition for the same key, which specifies where to import the service template from (often
./service.nix
) and the types of its parameters.
config.system.service.cowsay = config.system.callService ./service.nix {
address = mkOption {
type = types.str;
default = "0.0.0.0";
description = "Listen on specified address";
example = "127.0.0.1";
};
port = mkOption {
type = types.port;
default = 22;
description = "Listen on specified TCP port";
};
breed = mkOption {
type = types.str;
default = "British Friesian"
description = "Breed of the cow";
};
};
Then you need to provide the service template itself, probably in
./service.nix
:
{
# any nixpkgs package can be named here
liminix
, cowsayd
, serviceFns
, lib
}:
# these are the parameters declared in the callService invocation
{ address, port, breed} :
let
inherit (liminix.services) longrun;
inherit (lib.strings) escapeShellArg;
in longrun {
name = "cowsayd";
run = "${cowsayd}/bin/cowsayd --address ${address} --port ${builtins.toString port} --breed ${escapeShellArg breed}";
}
Tip
|
Not relevant to module-based services specifically, but a common gotcha
when specifiying services is forgetting to transform "rich" parameter
values into text when composing a command for the shell to execute. Note
here that the port number, an integer, is stringified with toString ,
and the name of the breed, which may contain spaces, is escaped with
escapeShellArg
|
Types
All of the NixOS module types are available in Liminix. These
Liminix-specific types also exist in pkgs.liminix.lib.types
:
-
service
: an s6-rc service -
interface
: an s6-rc service which specifies a network interface -
serviceDefn
: a service "template" definition
In the future it is likely that we will extend this to include other useful types in the networking domain: for example; IP address, network prefix or netmask, protocol family and others as we find them.
Emulated devices
Unless your changes depend on particular hardware devices, you may want to test your new/changed module with one of the emulated "devices" which runn on your build machine using the free QEMU machine emulator. They are
-
qemu
(MIPS) -
qemu-armv7l
(32 bit ARM) -
qemu-aarch64
(64 bit ARM)
This means you don’t need to keep flashing or messing with U-Boot: it also enables testing against emulated network peers using QEMU socket networking, which may be preferable to letting Liminix loose on your actual LAN. To build,
nix-build -I liminix-config=path/to/your/configuration.nix --arg device "import ./devices/qemu" -A outputs.default
This creates a result/
directory containing a vmlinux
and a
rootfs
, and a shell script run.sh
which invokes QEMU to run
that kernel with that filesystem. It connects the Liminix serial console
and the QEMU
monitor to stdin/stdout. Use ^P
(not ^A
) to switch to the monitor.
If you run with --background /path/to/some/directory
as the first
parameter, it will fork into the background and open Unix sockets in
that directory for console and monitor. Use nix-shell --run
connect-vm
to connect to either of these sockets, and ^O to
disconnect.
Networking
VMs can network with each other using QEMU socket networking. We observe these conventions, so that we can run multiple emulated instances and have them wired up to each other in the right way:
-
multicast 230.0.0.1:1234 : access (interconnect between router and "isp")
-
multicast 230.0.0.1:1235 : lan
-
multicast 230.0.0.1:1236 : world (the internet)
Any VM started by a run.sh
script is connected to "lan" and
"access". The emulated upstream (see below) runs PPPoE and is
connected to "access" and "world".
Upstream connection
In pkgs/routeros there is a derivation to install and configure
Mikrotik RouterOS as a PPPoE access
concentrator connected to the access
and world
networks, so that
Liminix PPPoE client support can be tested without actual hardware.
This is made available as the routeros
command in buildEnv
, so
you can do something like:
mkdir ros-sockets nix-shell nix-shell$ routeros ros-sockets nix-shell$ connect-vm ./ros-sockets/console
to start it and connect to it. Note that by default it runs in the background. It is connected to "access" and "world" virtual networks and runs a PPPoE service on "access" - so a Liminix VM with a PPPOE client can connect to it and thus reach the virtual internet. [ check, but pretty sure this is not the actual internet ]
Liminix does not provide RouterOS licences and it is your own responsibility if you use this to ensure you’re compliant with the terms of Mikrotik’s licencing. It may be supplemented or replaced in time with configurations for RP-PPPoE and/or Accel PPP.
Hardware hacking/porting to new device
The steps to port to a new hardware device are largely undocumented at present (although this hasn’t stopped people from figuring it out already). As an outline I would recommend
-
choose hardware that OpenWrt already supports, otherwise you will probably spend a lot of time writing kernel code. The OpenWrt kernel supports many network interfaces and other hardware for a lot of hardware boards that might only just about be able to boot Linux on a serial port if you stick to mainline Linux
-
work out how to get a serial console on it. You are unlikely to get working networking on your first go at boulding a kernel
-
find the most similar device in Liminiux and copy
devices/existing-similar-device
todevices/cool-new-device
as a starting point -
use the kernel configuration (
/proc/config.gz
) from OpenWrt as a reference for the kernel config you’ll need to specify indevices/cool-new-device/default.nix
-
break it down into achieveable goals. Your first goal should be something that can TFTP boot the kernel as far as a running userland. Networking is harder, Wifi often much harder - it sometimes also depends on having working flash even if you’re TFTP booting because the driver expects to load wifi firmware or calibration data from the flash
-
ask on IRC!
TFTP
How you get your image onto hardware will vary according to the device, but is likely to involve taking it apart to add wires to serial console pads/headers, then using U-Boot to fetch images over TFTP. The OpenWrt documentation has a good explanation of what you may expect to find on the device.
tufted
is a rudimentary TFTP server which runs from the command
line, has an allowlist for client connections, and follows symlinks,
so you can have your device download images direct from the
./result
directory without exposing /nix/store/
to the
internet or mucking about copying files to /tftproot
. If the
permitted device is to be given the IP address 192.168.8.251 you might
do something like this:
nix-shell --run "tufted -a 192.168.8.251 result"
Now add the device and server IP addresses to your configuration:
boot.tftp = {
serverip = "192.168.8.111";
ipaddr = "192.168.8.251";
};
and then build the derivation for outputs.default
or
outputs.mtdimage
(for which it will be an alias on any device where
this is applicable). You should find it has created
-
result/firmware.bin
which is the file you are going to flash -
result/flash.scr
which is a set of instructions to U-Boot to download the image and write it to flash after erasing the appropriate flash partition.
Note
|
TTL serial connections typically have no form of flow control and so
don’t always like having massive chunks of text pasted into them - and
U-Boot may drop characters while it’s busy. So don’t necessarily expect
to copy-paste the whole of boot.scr into a terminal emulator and
have it work just like that. You may need to paste each line one at a
time, or even retype it.
|
Running from RAM
For a faster edit-compile-test cycle, you can build a TFTP-bootable image which boots directly from RAM (using phram) instead of needing to be flashed first. In your device configuration add
imports = [
./modules/tftpboot.nix
];
and then build outputs.tftpboot
. This creates a file result/boot.scr
, which you can copy and paste into U-Boot to
transfer the kernel and filesystem over TFTP and boot the kernel from
RAM.
Networking
You probably don’t want to be testing a device that might serve DHCP, DNS and routing protocols on the same LAN as you (or your colleagues, employees, or family) are using for anything else, because it will interfere. You also might want to test the device against an "upstream" connection without having to unplug your regular home router from the internet so you can borrow the cable/fibre/DSL.
bordervm
is included for this purpose. You will need
-
a Linux machine with a spare (PCI or USB) ethernet device which you can dedicate to Liminix
-
an L2TP service such as https://www.aa.net.uk/broadband/l2tp-service/
You need to "hide" the Ethernet device from the host so that QEMU has
exclusive use of it. For PCI this means configuring it for VFIO
passthru; for USB you need to unload the module(s) it uses. I have
this segment in my build machine’s configuration.nix
which you may
be able to adapt:
boot = {
kernelParams = [ "intel_iommu=on" ];
kernelModules = [
"kvm-intel" "vfio_virqfd" "vfio_pci" "vfio_iommu_type1" "vfio"
];
postBootCommands = ''
# modprobe -i vfio-pci
# echo vfio-pci > /sys/bus/pci/devices/0000:01:00.0/driver_override
'';
blacklistedKernelModules = [
"r8153_ecm" "cdc_ether"
];
};
services.udev.extraRules = ''
SUBSYSTEM=="usb", ATTRS{idVendor}=="0bda", ATTRS{idProduct}=="8153", OWNER="dan"
'';
Then you can execute run-border-vm
in a buildEnv
shell, which
starts up QEMU using the NixOS configuration in
bordervm-configuration.nix
.
Inside the VM
-
your Liminix checkout is mounted under
/home/liminix/liminix
-
TFTP is listening on the ethernet device and serving
/home/liminix/liminix
. The server IP address is 10.0.0.1 -
a PPPOE-L2TP relay is running on the same ethernet card. When the connected Liminix device makes PPPoE requests, the relay spawns L2TPv2 Access Concentrator sessions to your specified L2TP LNS. Note that authentication is expected at the PPP layer not the L2TP layer, so the PAP/CHAP credentials provided by your L2TP service can be configured into your test device - bordervm doesn’t need to know about them.
To configure bordervm, you need a file called bordervm.conf.nix
which you can create by copying and appropriately editing
bordervm.conf-example.nix
Note
|
If you make changes to the bordervm configuration after executing
run-border-vm , you need to remove the border.qcow2 disk image
file otherwise the changes won’t get picked up.
|
Contributing
Patches welcome! Also bug reports, documentation improvements, experience reports/case studies etc etc all equally as welcome.
-
if you have an obvious bug fix, new package, documentation improvement or other uncontroversial small patch, send it straight in.
-
if you have a large new feature or design change in mind, please please get in touch to talk about it before you commit time to implementing it. Perhaps it isn’t what we were expecting, almost certainly we will have ideas or advice on what it should do or how it should be done.
Liminix development is not tied to Github or any other particular forge. How to send changes:
-
Push your Liminix repo with your changes to a git repository somewhere on the Internet that I can clone from. It can be on Codeberg or Gitlab or Sourcehut or Forgejo or Gitea or Github or a bare repo in your own personal web space or any kind of hosting you like.
-
Email devel@liminix.org with the URL of the repo and the branch name, and we will take a look.
If that’s not an option, I’m also happy for you to send your changes direct to the list itself, as an incremental git bundle or using git format-patch. We’ll work it out somehow.
The main development repo for Liminix is hosted at https://gti.telent.net/dan/liminix, with a read-only mirror at https://github.com/telent/liminix. If you’re happy to use Github then you can fork from the latter to make your changes, but please use the mailing list one of the approved routes to tell me about your changes because I don’t regularly go there to check PRs.
Remember that the Code of Conduct applies to all Liminix spaces, and anyone who violates it may be sanctioned or expelled from these spaces at the discretion of the project leadership.
Nix language style
This section describes some Nix language style points that we attempt to adhere to in this repo. Some are more aspirational than actual.
-
indentation and style is according to
nixfmt-rfc-style
-
favour
callPackage
over rawimport
for calling derivations or any function that may generate one - any code that might needpkgs
or parts of it. -
prefer
let inherit (quark) up down strange charm
overwith quark
, in any context where the scope is more than a single expression or there is more than one reference toup
,down
etc.with pkgs; [ foo bar baz]
is OK,with lib; stdenv.mkDerivation { ... }
is usually not. -
<liminix>
is defined only when running tests, so don’t refer to it in "application" code -
the parameters to a derivation are sorted alphabetically, except for
lib
,stdenv
and maybe other non-package "special cases" -
where a
let
form defines multiple names, put a newline after the tokenlet
, and indent each name two characters -
to decide whether some code should be a package or a module? Packages are self-contained - they live in
/nix/store/eeeeeee-name
and don’t directly change system behaviour by their presence or absense. modules can add to/etc
or/bin
or other global state, create services, all that side-effecty stuff. Generally it should be a package unless it can’t be.
Copyright
The Nix code in Liminix is MIT-licenced (same as Nixpkgs), but the code it combines from other places (e.g. Linux, OpenWrt) may have a variety of licences. Copyright assignment is not expected: just like when submitting to the Linux kernel you retain the copyright on the code you contribute.
Automated builds
Automated builds are run on each push to the main branch. This tests that (among other things)
-
every device image builds
-
the build for the “qemu” target is executed with a fake network upstream to test
-
PPPoE and DHCP service
-
hostap (wireless gateway)
You can view the build output at https://build.liminix.org . The tests are defined in ci.nix.
Unfortunately there’s no (easy) way I can make my CI infrastructure run your code, other than merging it. But see Running tests for how to exercise the same code locally on your machine.
Running tests
You can run all of the tests by evaluating ci.nix
, which is the
input I use in Hydra.
nix-build -I liminix=`pwd` ci.nix -A pppoe # run one job
nix-build -I liminix=`pwd` ci.nix -A all # run all jobs
Troubleshooting
Diagnosing unexpectedly large images
Sometimes you can add a package and it causes the image size to balloon
because it has dependencies on other things you didn’t know about. Build
the outputs.manifest
attribute, which is a JSON representation of
the filesystem, and you can run nix-store --query
on it.
nix-build -I liminix-config=path/to/your/configuration.nix \
--arg device "import ./devices/qemu" -A outputs.manifest \
-o manifest
nix-store -q --tree manifest
Module options
Code of Conduct
Note
|
As of Feb 2023, “RESPONSE TEAM” and “LEADERSHIP TEAM” in the text that follows both refer to me, Daniel Barlow, as the project leader. |
Liminix is dedicated to providing a harassment-free experience for everyone. We do not tolerate harassment of participants in any form.
This code of conduct applies to all Liminix spaces, including the IRC channel, mailing lists, and other forums, both online and off. Anyone who violates this code of conduct may be sanctioned or expelled from these spaces at the discretion of the RESPONSE TEAM.
Some Liminix spaces may have additional rules in place, which will be made clearly available to participants. Participants are responsible for knowing and abiding by these rules.
Harassment includes:
-
Offensive comments related to gender, gender identity and expression, sexual orientation, disability, mental illness, neuro(a)typicality, physical appearance, body size, age, race, or religion.
-
Unwelcome comments regarding a person’s lifestyle choices and practices, including those related to food, health, parenting, drugs, and employment.
-
Deliberate misgendering or use of ‘dead’ or rejected names.
-
Gratuitous or off-topic sexual images or behaviour in spaces where they’re not appropriate.
-
Physical contact and simulated physical contact (eg, textual descriptions like “hug” or “backrub”) without consent or after a request to stop.
-
Threats of violence.
-
Incitement of violence towards any individual, including encouraging a person to commit suicide or to engage in self-harm.
-
Deliberate intimidation.
-
Stalking or following.
-
Harassing photography or recording, including logging online activity for harassment purposes.
-
Sustained disruption of discussion.
-
Unwelcome sexual attention.
-
Pattern of inappropriate social contact, such as requesting/assuming inappropriate levels of intimacy with others
-
Continued one-on-one communication after requests to cease.
-
Deliberate “outing” of any aspect of a person’s identity without their consent except as necessary to protect vulnerable people from intentional abuse.
-
Publication of non-harassing private communication.
Liminix prioritizes marginalized people’s safety over privileged people’s comfort. RESPONSE TEAM reserves the right not to act on complaints regarding:
-
‘Reverse’ -isms, including ‘reverse racism,’ ‘reverse sexism,’ and ‘cisphobia’
-
Reasonable communication of boundaries, such as “leave me alone,” “go away,” or “I’m not discussing this with you.”
-
Communicating in a ‘tone’ you don’t find congenial
-
Criticizing racist, sexist, cissexist, or otherwise oppressive behavior or assumptions
Reporting
If you are being harassed by a member of Liminix, notice that someone else is being harassed, or have any other concerns, please contact the RESPONSE TEAM at [email address or other contact point]. If the person who is harassing you is on the team, they will recuse themselves from handling your incident. We will respond as promptly as we can.
This code of conduct applies to Liminix spaces, but if you are being harassed by a member of Liminix outside our spaces, we still want to know about it. We will take all good-faith reports of harassment by Liminix members, especially LEADERSHIP TEAM, seriously. This includes harassment outside our spaces and harassment that took place at any point in time. The abuse team reserves the right to exclude people from Liminix based on their past behavior, including behavior outside Liminix spaces and behavior towards people who are not in Liminix.
In order to protect volunteers from abuse and burnout, we reserve the right to reject any report we believe to have been made in bad faith. Reports intended to silence legitimate criticism may be deleted without response.
We will respect confidentiality requests for the purpose of protecting victims of abuse. At our discretion, we may publicly name a person about whom we’ve received harassment complaints, or privately warn third parties about them, if we believe that doing so will increase the safety of Liminix members or the general public. We will not name harassment victims without their affirmative consent.
Consequences
Participants asked to stop any harassing behavior are expected to comply immediately.
If a participant engages in harassing behavior, RESPONSE TEAM may take any action they deem appropriate, up to and including expulsion from all Liminix spaces and identification of the participant as a harasser to other Liminix members or the general public.
License and attribution
The policy is based on the Geek Feminism Community anti-harassment/Policy and is the work of Annalee Flower Horne with assistance from Valerie Aurora, Alex Skud Bayley, Tim Chevalier, and Mary Gardiner.
Appendix A: Supported hardware
Recommended devices
For development, the supported GL.iNet devices are all good choices if you can find them, as they have a builtin "debrick" procedure in the boot monitor and are also comparatively simple to attach serial cables to (soldering not required), so are lower-risk than some other devices.
For a more powerful device, something with an ath10k wireless would be the safe bet, or the Linksys E8450 which seems popular in the OpenWrt community.
Belkin RT-3200 / Linksys E8450
This device is based on a 64 bit Mediatek MT7622 ARM platform, and is mostly feature-complete in Liminix but as of Dec 2024 has seen very little actual use.
Hardware summary
-
MediaTek MT7622BV (1350MHz)
-
128MB NAND flash
-
512MB RAM
-
b/g/n wireless using MediaTek MT7622BV (MT7615E driver)
-
a/n/ac/ax wireless using MediaTek MT7915E
Installation
Liminix on this device uses the UBI volume management system to perform wear leveling on the flash. This is not set up from the factory, so a one-time step is needed to prepare it before Liminix can be installed.
Preparation
To prepare the device for Liminix you first need to use the OpenWrt UBI Installer image to rewrite the flash layout. As of Jan 2025 there are two versions of the installer available: the release version 1.0.2 and the pre-release 1.1.3 and for Liminix you nee the pre-relese. The release version of the installer creates UBI volumes according to an older layout that is not compatible with the Linux 6.6.67 kernel used in Liminix.
You can run the installer in one of two ways: either follow the instructions to do it through the vendor web interface, or you can drop to U-Boot and use TFTP
MT7622> setenv ipaddr 10.0.0.6
MT7622> setenv serverip 10.0.0.1
MT7622> tftpboot 0x42000000 openwrt-mediatek-mt7622-linksys_e8450-ubi-initramfs-recovery-installer.itb
MT7622> bootm 0x42000000
This will write the new flash layout and then boot into a "recovery" OpenWrt installation.
Building/installing Liminix ----------------
The default target for this device is outputs.ubimage
which makes a
ubifs image suitable for use with ubiupdatevol
. To write this to the
device we use the OpenWrt recovery system installed in the previous
step. In this configuration the device assigns itself the IP address
192.168.1.1/24 on its LAN ports and expects the connected computer to
have 192.168.1.254
Warning
|
The ubi0_7 device in these instructions is correct as of
Dec 2024 (dangowrt/owrt-ubi-installer commit d79e7928). If you are
installing some time later, it is important to check the output from
|
$ nix-build -I liminix-config=./my-configuration.nix --arg device "import ./devices/belkin-rt3200" -A outputs.default
$ cat result/rootfs | ssh root@192.168.1.1 "cat > /tmp/rootfs"
$ ssh root@192.168.1.1
root@OpenWrt:~# ubimkvol /dev/ubi0 --name=liminix --maxavsize
root@OpenWrt:~# ubinfo -a
[...]
Volume ID: 7 (on ubi0)
Type: dynamic
Alignment: 1
Size: 851 LEBs (108056576 bytes, 103.0 MiB)
State: OK
Name: liminix
Character device major/minor: 250:8
root@OpenWrt:~# ubiupdatevol /dev/ubi0_7 /tmp/rootfs
To make the new system bootable we also need to change some U-Boot
variables. boot_production
needs to mount the filesystem and boot
the FIT image found there, and bootcmd
needs to be told _not
to boot the rescue image if there are records in pstore, because that
interferes with config.log.persistent
root@OpenWrt:~# fw_setenv orig_boot_production $(fw_printenv -n boot_production)
root@OpenWrt:~# fw_setenv orig_bootcmd $(fw_printenv -n bootcmd)
root@OpenWrt:~# fw_setenv boot_production 'led $bootled_pwr on ; ubifsmount ubi0:liminix && ubifsload ${loadaddr} boot/fit && bootm ${loadaddr}'
root@OpenWrt:~# fw_setenv bootcmd 'run boot_ubi'
For subsequent Liminix reinstalls, just run the ubiupdatevol
command
again. You don’t need to repeat the "Preparation" step and in fact
should seek to avoid it if possible, as it will reset the erase counters
used for write levelling. Using UBI-aware tools is therefore preferred
over any kind of "factory" wipe which will reset them.
GL.iNet GL-AR750
Hardware summary
The GL-AR750 "Creta" travel router features:
QCA9531 @650Mhz SoC
dual band wireless: IEEE 802.11a/b/g/n/ac
two 10/100Mbps LAN ports and one WAN
128MB DDR2 RAM
16MB NOR Flash
supported in OpenWrt by the "ath79" SoC family
The GL-AR750 has two distinct sets of wifi hardware. The 2.4GHz radio is part of the QCA9531 SoC, i.e. it’s on the same silicon as the CPU, the Ethernet, the USB etc. The device is connected to the host via AHB and it is supported in Linux using the ath9k driver. 5GHz wifi is provided by a QCA9887 PCIe (PCI embedded) WLAN chip, supported by the ath10k driver.
Installation
As with many GL.iNet devices, the stock vendor firmware is a fork of
OpenWrt, meaning that the binary created by system-outputs-mtdimage
can be flashed using the vendor web UI or the U-Boot emergency "unbrick"
routine.
Flashing over an existing Liminix system is not possible while that
system is running, otherwise you’ll be overwriting flash partitions
while they’re in use - and that might not end well. Configure the system
with levitate
if you need to make it upgradable.
Vendor web page: https://www.gl-inet.com/products/gl-ar750/
OpenWrt web page: https://openwrt.org/toh/gl.inet/gl-ar750
GL.iNet GL-MT300A
The GL-MT300A is based on a MT7620 chipset.
For flashing from U-Boot, the firmware partition is from 0xbc050000 to 0xbcfd0000.
WiFi on this device is provided by the rt2800soc module. It expects firmware to be present in the "factory" MTD partition, so - assuming we want to use the wireless - we need to build MTD support into the kernel even if we’re using TFTP root.
Installation
The stock vendor firmware is a fork of OpenWrt, meaning that the binary
created by system-outputs-mtdimage
can be flashed using the vendor
web UI or the U-Boot emergency "unbrick" routine.
Flashing over an existing Liminix system is not possible while that
system is running, otherwise you’ll be overwriting flash partitions
while they’re in use - and that might not end well. Configure the system
with levitate
if you need to make it upgradable.
Vendor web page: https://www.gl-inet.com/products/gl-mt300a/
OpenWrt web page: https://openwrt.org/toh/gl.inet/gl-mt300a
GL.iNet GL-MT300N-v2
The GL-MT300N-v2 "Mango" is is very similar to the gl-mt300a
, but is
based on the MT7628 chipset instead of MT7620. It’s also marginally
cheaper and comes in a yellow case not a blue one. Be sure your device
is v2 not v1, which is a different animal and has only half as much RAM.
Installation
The stock vendor firmware is a fork of OpenWrt, meaning that the binary
created by system-outputs-mtdimage
can be flashed using the vendor
web UI or the U-Boot emergency "unbrick" routine.
Flashing over an existing Liminix system is not possible while that
system is running, otherwise you’ll be overwriting flash partitions
while they’re in use - and that might not end well. Configure the system
with levitate
if you need to make it upgradable.
Vendor web page: https://www.gl-inet.com/products/gl-mt300n-v2/
OpenWrt web page: https://openwrt.org/toh/gl.inet/gl-mt300n_v2
OpenWrt One
Hardware summary
-
MediaTek MT7981B (1300MHz)
-
256MB NAND Flash
-
1024MB RAM
-
WLan hardware: Mediatek MT7976C
Status
-
Only tested over TFTP so far.
-
WiFi (2.4G and 5G) works.
-
2.5G ethernet port works.
Limitations
-
adding he_bss_color="128" causes Invalid argument for hostap
-
nvme support untested
-
I don’t think the front LEDs work yet
Installation
TODO: add instructions on how to boot directly from TFTP to memory and how to install from TFTP to flash without going through OpenWrt.
The instructions below assume you can boot and SSH into OpenWrt:
Boot into OpenWrt and create a 'liminix' UBI partition:
root@OpenWrt:~# ubimkvol /dev/ubi0 --name=liminix --maxavsize
Remember the 'Volume ID' that was created for this new partition
Build the UBI image and write it to this new partition:
$ nix-build -I liminix-config=./my-configuration.nix --arg device "import ./devices/openwrt-one" -A outputs.default $ cat result/rootfs | ssh root@192.168.1.1 "cat > /tmp/rootfs" $ ssh root@192.168.1.1 root@OpenWrt:~# ubiupdatevol /dev/ubi0_X /tmp/rootfs # replace X with the volume id, if needed check with ubinfo
Reboot into the U-Boot prompt and boot with:
OpenWrt One> ubifsmount ubi0:liminix && ubifsload ${loadaddr} boot/fit && bootm ${loadaddr}'
If this works, reboot into OpenWrt and configure U-Boot to boot ubifs by default:
root@OpenWrt:~# fw_setenv orig_boot_production $(fw_printenv -n boot_production) root@OpenWrt:~# fw_setenv boot_production 'led white on ; ubifsmount ubi0:liminix && ubifsload ${loadaddr} boot/fit && bootm ${loadaddr}'
Troubleshooting
The instructions above assume you can boot and SSH into the (recovery) OpenWrt installation. If you have broken your device to the point where that is no longer possible, you could re-install OpenWrt, but probably you could also install directly from U-Boot:
QEMU MIPS
This target produces an image for QEMU, the "generic and open source machine emulator and virtualizer".
MIPS QEMU emulates a "Malta" board, which was an ATX form factor evaluation board made by MIPS Technologies, but mostly in Liminix we use paravirtualized devices (Virtio) instead of emulating hardware.
Building an image for QEMU results in a result/
directory containing
run.sh
vmlinux
, and rootfs
files. To invoke the emulator,
run run.sh
.
The configuration includes two emulated "hardware" ethernet devices and
the kernel mac80211_hwsim
module to provide an emulated wlan device.
To read more about how to connect to this network, refer to
qemu-networking
in the Development manual.
QEMU Aarch64
This target produces an image for the QEMU "virt" platform using a 64 bit CPU type.
ARM targets differ from MIPS in that the kernel format expected by QEMU
is an "Image" (raw binary file) rather than an ELF file, but this is
taken care of by run.sh
. Check the documentation for the qemu
target for more information.
QEMU ARM v7
This target produces an image for the QEMU "virt" platform using a 32 bit CPU type.
ARM targets differ from MIPS in that the kernel format expected by QEMU
is an "Image" (raw binary file) rather than an ELF file, but this is
taken care of by run.sh
. Check the documentation for the QEMU
(MIPS) target for more information.
TP-Link Archer AX23 / AX1800 Dual Band Wi-Fi 6 Router
Hardware summary
-
MediaTek MT7621 (880MHz)
-
16MB Flash
-
128MB RAM
-
WLan hardware: Mediatek MT7905, MT7975
Limitations
Status LEDs do not work yet.
Uploading an image via tftp doesn’t work yet, because the Archer uboot version is so old it doesn’t support overriding the DTB from the mboot command. The tftpboot module doesn’t support this yet, see https://gti.telent.net/dan/liminix/pulls/5 for the WiP.
Turris Omnia
This is a 32 bit ARMv7 MVEBU device, which is usually shipped with TurrisOS, an OpenWrt-based system. Rather than reformatting the builtin storage, we install Liminix on to the existing btrfs filesystem so that the vendor snapshot/recovery system continues to work (and provides you an easy rollback if you decide you don’t like Liminix after all).
The install process has two stages, and is intended that you should not need to open the device and add a serial console (although it may be handy for visibility, and in case anything goes wrong). First we build a minimal installation/recovery system, then we reboot into that recovery image to prepare the device for the full target install.
Installation using a USB stick
First, build the image for the USB stick. Review
examples/recovery.nix
in order to change the default root password
(which is secret
) and/or the SSH keys, then build it with
$ nix-build -I liminix-config=./examples/recovery.nix \
= --arg device "import ./devices/turris-omnia" \
= -A outputs.mbrimage -o mbrimage
$ file -L mbrimage
mbrimage: DOS/MBR boot sector; partition 1 : ID=0x83, active, start-CHS (0x0,0,5), end-CHS (0x6,130,26), startsector 4, 104602 sectors
Next, copy the image from your build machine to a USB storage medium
using dd
or your other most favoured file copying tool, which might
be a comand something like this:
$ dd if=mbrimage of=/dev/path/to/the/usb/stick \
= bs=1M conv=fdatasync status=progress
The Omnia’s default boot order only checks USB after it has failed to boot from eMMC, which is not ideal for our purpose. Unless you have a serial cable, the easiest way to change this is by booting to TurrisOS and logging in with ssh:
root@turris:/# fw_printenv boot_targets
boot_targets=mmc0 nvme0 scsi0 usb0 pxe dhcp
root@turris:/# fw_setenv boot_targets usb0 mmc0
root@turris:/# fw_printenv boot_targets
boot_targets=usb0 mmc0
root@turris:/# reboot -f
It should now boot into the recovery image. It expects a network cable
to be plugged into LAN2 with something on the other end of it that
serves DHCP requests. Check your DHCP server logs for a request from a
liminix-recovery
host and figure out what IP address was assigned.
$ ssh liminix-recovery.lan
You should get a "Busybox" banner and a root prompt. Now you can start preparing the device to install Liminix on it. First we’ll mount the root filesystem and take a snapshot:
# mkdir /dest && mount /dev/mmcblk0p1 /dest
# schnapps -d /dest create "pre liminix"
# schnapps -d /dest list
ERROR: not a valid btrfs filesystem: /
= # | Type | Size | Date | Description
------+-----------+-------------+---------------------------+------------------------------------
= 1 | single | 16.00KiB | 1970-01-01 00:11:49 +0000 | pre liminix
(not a valid btrfs filesystem: /
is not a real error)
then we can remove all the files
# rm -r /dest/@/*
and then it’s ready to install the real Liminix system onto. On your
build system, create the Liminix configuration you wish to install: here
we’ll use the rotuer
example.
build$ nix-build -I liminix-config=./examples/rotuer.nix \
= --arg device "import ./devices/turris-omnia" \
= -A outputs.systemConfiguration
and then use min-copy-closure
to copy it to the device.
build$ nix-shell --run \
= "min-copy-closure -r /dest/@ root@liminix-recovery.lan result"
and activate it
build$ ssh root@liminix-recovery.lan \
= "/dest/@/$(readlink result)/bin/install /dest/@"
The final steps are performed directly on the device again: add a
symlink so U-Boot can find /boot
, then restore the default boot
order and reboot into the new configuration.
# cd /dest && ln -s @/boot .
# fw_setenv boot_targets "mmc0 nvme0 scsi0 usb0 pxe dhcp"
# cd / ; umount /dest
# reboot
Installation using a TFTP server and serial console
If you have a serial
console connection and a TFTP server, and would
rather use them than fiddling with USB sticks, the
examples/recovery.nix
configuration also works using the
tftpboot
output. So you can do
build$ nix-build -I liminix-config=./examples/recovery.nix \
= --arg device "import ./devices/turris-omnia" \
= -A outputs.tftpboot
and then paste the generated result/boot.scr
into U-Boot, and you
will end up with the same system as you would have had after booting
from USB. If you don’t have a serial console connection you could
probably even get clever with elaborate use of fw_setenv
, but that
is left as an exercise for the reader.
Zyxel NWA50AX
Zyxel NWA50AX is quite close to the GL-MT300N-v2 "Mango" device, but it is based on the MT7621 chipset instead of the MT7628.
Installation
This device is pretty, but, due to its A/B capabilities, can be a bit hard to use completely.
The stock vendor firmware is a downstream fork of U-Boot: https://github.com/RaitoBezarius/uboot-nwa50ax with restricted boot commands. Fortunately, OpenWrt folks figured out trivial command injections, so you can use most of the OpenWrt commands without trouble by just command injecting atns, atna or atnf, e.g. atns "; $real_command".
From factory web UI, you can upload the result of the zyxel-nwa-fit output. From another operating system, you need to dumpimage -T flat_dt -p 0 $zyxel-nwa-fit -o firmware.bin, flash_erase $(mtd partition of the target partition firmware or zy_firmware) 0 0, then you complete by nandwrite -p $(mtd partition of the target partition firmware or zy_firmware) firmware.bin.
How to put the firmware.bin on the machine is left to you as an exercise, e.g. SSH, TFTP, whatever.
From serial, you have two choices:
-
Flash this system via U-Boot: same reasoning as from an existing Linux system, two choices:
-
ymodem the binary, perform the write manually, you can inspire yourself from the script contained in the vendor firmware, those are just a FIT containing a script.
-
prepare a FIT containing a script executing your commands, tftpboot this.
-
-
-
boot from an existing Liminix system, e.g. TFTPBOOT image.
-
boot from an OpenWrt system, i.e. follow OpenWrt steps.
Once you are in a Linux system, understand that this device has A/B boot.
OpenWrt provides you with zyxel-bootconfig to set/unset the image status and choice.
The kernel is booted with bootImage=<number> which tells you which slot are you on.
You should find yourself with 10ish MTD partitions, the most interesting ones are two:
-
firmware: 40MB
-
firmware_1: 40MB
In the current setup, they are split further into kernel (8MB) and ubi (32MB).
Once you are done with first installation, note that if you want to use the A/B feature, you need to write a _secondary image on the slot B. There is no proper flashing code that will set the being-updated slot to new and boot on it to verify if it’s working. This is a WIP.
Upgrading your system can be achieved via:
-
liminix-rebuild for the userspace.
-
flash_erase + nandwrite for the kernelspace to the other slot than the one you are booted on, note that you can just nandwrite the mtd partition corresponding to the kernel and not the whole firmware.
If you soft-bricked your AP, i.e. you cannot boot anything in U-Boot, no worries, just plug the serial console, prepare a TFTP server (via tufted for example), download vendor firmware, set up atns, atnf, etc. and run atnz.
This will reflash everything back to normal via TFTP.
If you hard-bricked your AP, i.e. U-Boot is telling you to transfer a valid bootloader via ymodem, just extract a U-Boot from the vendor OS, send it via ymodem and use the previous operations to perform a full flash this time of all partitions.
Note that if you erased your MRD partition, you lost your serial and MAC address. There’s no way to recover the original one except by reading the physical label on your… device!
If you super-hard-bricked your AP, i.e. no output on serial console, congratulations, you reached one of the rare state of this device. You need an external NAND flasher to repair it and write the first stage from Mediatek to continue the previous recovery operations.
Development TODO list:
-
Better support for upgrade automation w.r.t. to A/B, e.g. automagic scripts.
-
Mount the logs partition, mount / as overlayfs of firmware ? rootfs and rootfs_data for extended data.
-
Jitter-based entropy injection? Device can be slow to initialize its CRNG and hostapd will reject few clients at the start because of that.
-
Defaults for hostapd based on MT7915 capabilities? See the example for one possible list.
-
Remove primary/secondary hack and put it in preinit.
-
Offer ways to reflash the bootloader itself to support direct boot via UBI and kernel upgrades via filesystem rewrite.
Vendor web page: https://www.zyxel.com/fr/fr/products/wireless/ax1800-wifi-6-dual-radio-nebulaflex-access-point-nwa50ax
OpenWrt web page: https://openwrt.org/inbox/toh/zyxel/nwa50ax OpenWrt tech data: https://openwrt.org/toh/hwdata/zyxel/zyxel_nwa50ax
Appendix B: Module and service options
Base options
- option
boot.commandLine
= Kernel command line =
= type list of non-empty string =
= default =[ ]
- option
boot.commandLineDtbNode
= Kernel command line’s devicetree node =
= type one of "bootargs", "bootargs-override" =
= default ="bootargs"
option
boot.imageFormat
type one of "fit", "uimage"
default
"uimage"
option
boot.imageType
type one of "primary", "secondary"
default
"primary"
- option
boot.tftp.ipaddr
= Our IP address to use when creating scripts to boot or flash from = U-Boot. Not relevant in normal operation =
= type string
- option
boot.tftp.loadAddress
= RAM address at which to load data when transferring via TFTP. This is = not the address of the flash storage, nor the kernel load address: it = should be set to some part of RAM that’s not used for anything else = and suitable for temporary storage. =
= type unsigned integer, meaning >=0
- option
boot.tftp.serverip
= IP address of the TFTP server. Not relevant in normal operation =
= type string
- option
defaultProfile.packages
= List of packages which are available in a login shell. (This is = analogous to systemPackages in NixOS, but we don’t symlink into = /run/current-system, we just add the paths in /etc/profile =
= type list of package
- option
filesystem
= Skeleton filesystem, represented as nested attrset. Consult the source = code if you need to add to this =
= type anythingoption
rootOptions
type null or string
default
null
option
rootfsType
type one of "btrfs", "ext4", "jffs2", "squashfs", "ubifs"
default
"squashfs"
option
services
type attribute set of s6-rc service
option
system.callService
type function that evaluates to a(n) function that evaluates to a(n) anything
Busybox
Busybox provides stripped-down versions of many usual Linux/Unix tools, and may be configured to include only the commands (termed "applets") required by the user or by other included modules.
- option
programs.busybox.applets
= Applets required =
= type list of string =
= example =[ = "sh" = "getty" = "login" ]=
= default =[ ]
- option
programs.busybox.options
= Other busybox config flags that do not map directly to applet names = (often prefixed FEATURE) =
= type attribute set of non-empty string =
= example ={ = FEATURE_DD_IBS_OBS = "y"; }=
= default ={ }
Hardware-dependent options
These are attributes of the hardware not of the application you want to
run on it, and would usually be set in the "device" file:
devices/manuf-model/default.nix
- option
hardware.alignment
= Alignment passed to mkimage for FIT =
= type null or (unsigned integer, meaning >=0) =
= default =null
- option
hardware.defaultOutput
= "Default" output: what gets built for this device when outputs.default = is requested. Typically this is "mtdimage" or "vmroot" =
= type non-empty string
- option
hardware.dts.includePaths
= List of directories to search for DTS includes (.dtsi files) =
= type list of path =
= default =[ ]
- option
hardware.dts.includes
= "dtsi" fragments to include in the generated device tree =
= type list of path =
= default =[ ]
- option
hardware.dts.src
= If the device requires an external device tree to be loaded alongside = the kernel, this is the path to the device tree source (we usually get = these from OpenWrt). This value may be null if the platform creates = the device tree - currently this is the case only for QEMU. =
= type null or pathoption
hardware.entryPoint
type unsigned integer, meaning >=0
- option
hardware.flash.address
= Start address of whichever partition (often called "firmware") we’re = going to overwrite with our kernel uimage and root fs. Usually not the = entire flash, as we don’t want to clobber the bootloader, calibration = data etc =
= type unsigned integer, meaning >=0
- option
hardware.flash.eraseBlockSize
= Flash erase block size in bytes =
= type unsigned integer, meaning >=0
- option
hardware.flash.size
= Size in bytes of the firmware partition =
= type unsigned integer, meaning >=0option
hardware.loadAddress
type unsigned integer, meaning >=0
default
null
option
hardware.networkInterfaces
type attribute set of anything
- option
hardware.radios
= Kernel modules (from mac80211 package) required for the wireless = devices on this board =
= type list of string =
= example =[ = "ath9k" = "ath10k" ]=
= default =[ ]
option
hardware.ram.startAddress
type signed integer
- option
hardware.rootDevice
= Full path to preferred root device =
= type string =
= example ="/dev/mtdblock3"
option
hardware.ubi.logicalEraseBlockSize
type string
option
hardware.ubi.maxLEBcount
type string
option
hardware.ubi.minIOSize
type string
option
hardware.ubi.physicalEraseBlockSize
type string
hostname
- option
hostname
= System hostname of the device, as returned by gethostname(2). May or = may not correspond to any name it’s reachable at on any network. =
= type non-empty string =
= default ="liminix"
Kernel-related options
- option
kernel.conditionalConfig
= Kernel config options that should only be applied when some other = option is present. =
= type attribute set of attribute set of non-empty string =
= example ={ = USB = { = USB_XHCI_HCD = "y"; = USB_XHCI_MVEBU = "y"; = }; }=
= default ={ }
- option
kernel.config
= Kernel config options, as listed in Kconfig* files in the kernel = source tree. Do not include the leading "CONFIG" prefix when = defining these. Most values are "y", "n" or "m", but sometimes other = strings are also used. =
= type attribute set of non-empty string =
= example ={ = BRIDGE = "y"; = TMPFS = "y"; = FW_LOADER_USER_HELPER = "n"; };
option
kernel.extraPatchPhase
type strings concatenated with "\n"
default
"true"
option
kernel.makeTargets
type list of string
- option
kernel.modular
= support loadable kernel modules =
= type boolean =
= default =true
option
kernel.src
type path
default
<derivation linux.tar.gz>
option
kernel.version
type string
default
"5.15.137"
logging
- option
logging.persistent.enable
= Whether to enable store logs across reboots. =
= type boolean =
= example =true=
= default =false
boot-extlinux
- option
boot.loader.extlinux.enable
= Whether to enable extlinux. =
= type boolean =
= example =true=
= default =false
boot-fit
- option
boot.loader.fit.enable
= Whether to enable FIT in /boot. =
= type boolean =
= example =true=
= default =false
initramfs
- option
boot.initramfs.enable
= Whether to enable initramfs. =
= type boolean =
= example =true=
= default =false
tftpboot
option
boot.tftp.appendDTB
type boolean
default
false
option
boot.tftp.compressRoot
type boolean
default
false
option
boot.tftp.freeSpaceBytes
type signed integer
default
0
option
boot.tftp.kernelFormat
type one of "zimage", "uimage"
default
"uimage"
ramdisk
- option
boot.ramdisk.enable
= Whether to enable reserving part of memory as an MTD-based RAM disk. = Needed for TFTP booting . =
= type boolean =
= example =true=
= default =false
s6
- option
logging.directory
= default log directory =
= type path =
= default ="/run/log"
- option
logging.script
= "log script" used by fallback s6-log process =
= type string =
= default ="pliminix t"
- option
logging.shipping.enable
= Whether to enable unix socket for log shipping. =
= type boolean =
= example =true=
= default =false
- option
logging.shipping.service
= log shipper service =
= type s6-rc service
- option
logging.shipping.socket
= socket pathname =
= type path =
= default ="/run/.log-shipping.sock"
Users
User- and group-related configuration.
Changes made here are reflected in files such as :file:/etc/shadow,
:file:/etc/passwd, :file:/etc/group etc. If you are familiar with user
configuration in NixOS, please note that Liminix does not have the
concept of "mutable users" - files in /etc/ are symlinks to the
immutable store, so you can’t e.g change a password with passwd
option
groups
type attribute set of (submodule)
option
groups.<name>.gid
type signed integer
option
groups.<name>.usernames
type list of string
default
[ ]
option
users
type attribute set of (submodule)
option
users.<name>.dir
type string
default
"/run"
option
users.<name>.gecos
type string
example
"Jo Q User"
default
""
option
users.<name>.gid
type signed integer
option
users.<name>.openssh.authorizedKeys.keys
type list of string
default
[ ]
- option
users.<name>.passwd
= encrypted password, as generated by mkpasswd -m sha512crypt =
= type string =
= example ="$6$RIYL.EgWOrtoJ0/7$Z53a8sc0o6AU/kuFOGiLJKhwVavTG/deoM7JTs6luNczYSUsh4UYmhvT8sVzm.l8F/LZXhhhkC7IHQs5UGAIM/"=
= default ="!!"
option
users.<name>.shell
type string
default
"/bin/sh"
option
users.<name>.uid
type signed integer
Bridge module
Allows creation of Layer 2 software "bridge" network devices. A common use case is to merge together a hardware Ethernet device with one or more WLANs so that several local devices appear to be on the same network.
path modules/bridge/default.nix
service
system.service.bridge.members
Service parameters
- option
members
= interfaces to add to the bridge =
= type list of s6-rc service
- option
primary
= primary bridge interface =
= type s6-rc service
service
system.service.bridge.primary
Service parameters
- option
ifname
= bridge interface name to create =
= type string
DHCP6 client module
This is for use if you have an IPv6-capable upstream that provides address information and/or prefix delegation using DHCP6. It provides a service to request address information in the form of a DHCP lease, and two dependent services that listen for updates to the DHCP address information and can be used to update addresses of network interfaces that you want to assign those prefixes to
path modules/dhcp6c/default.nix
service
system.service.dhcp6c.address
Service parameters
option
client
type anything
- option
interface
= interface to assign the address to =
= type s6-rc service
service
system.service.dhcp6c.client
Service parameters
- option
interface
= interface (usually WAN) to query for DHCP6 =
= type s6-rc service
service
system.service.dhcp6c.prefix
Service parameters
option
client
type anything
- option
interface
= interface to assign <prefix>::1 to =
= type s6-rc service
Dnsmasq
This module includes a service to provide DNS, DHCP, and IPv6 router advertisement for the local network.
path modules/dnsmasq/default.nix
service
system.service.dnsmasq
Service parameters
- option
domain
= Domain name for DHCP service: causes the DHCP server to return the = domain to any hosts which request it, and sets the domain which it is = legal for DHCP-configured hosts to claim =
= type string
- option
group
= Specifies the unix group which dnsmasq will run as =
= type string =
= default =dnsmasq
option
hosts
type attribute set of (submodule)
option
interface
type s6-rc service
option
ranges
type list of string
option
resolvconf
type null or s6-rc service
option
upstreams
type list of string
- option
user
= Specifies the unix user which dnsmasq will run as =
= type string =
= default =dnsmasq
Firewall
Provides a service to create an nftables ruleset based on configuration supplied to it.
path modules/firewall/default.nix
service
system.service.firewall
Service parameters
- option
extraRules
= firewall ruleset =
= type attribute set of (attribute set)
- option
rules
= firewall ruleset =
= type attribute set of (attribute set)option
zones
type attribute set of list of s6-rc service
Hostapd
Hostapd (host access point daemon) enables a wireless network interface to act as an access point and authentication server, providing IEEE 802.11 access point management, and IEEE 802.1X/WPA/WPA2/EAP Authenticators. In less technical terms, this service is what you need for your Liminix device to provide a wireless network that clients can connect to.
If you have more than one wireless network interface (e.g. wlan0, wlan1) you can run an instance of hostapd on each of them.
path modules/hostapd/default.nix
service
system.service.hostapd
Service parameters
option
interface
type s6-rc service
option
params
type attribute set
ifwait
path modules/ifwait/default.nix
service
system.service.ifwait
Service parameters
option
interface
type s6-rc service
option
service
type s6-rc service
option
state
type string
Mount
Mount filesystems
path modules/mount/default.nix
service
system.service.mount
Service parameters
option
fstype
type string
default
auto
option
mountpoint
type string
option
options
type list of string
option
partlabel
type string
Network
Basic network services for creating hardware ethernet devices and adding addresses
path modules/network/default.nix
- service
system.service.network.address
= network interface address =
= Service parameters =
= _ = optionaddress
=
= = type string = = optionfamily
=
= = type one of "inet", "inet6" = = optioninterface
=
= = type s6-rc service = = optionprefixLength
=
= = type integer between 0 and 128 (both inclusive) = = _
- service
system.service.network.dhcp.client
= DHCP v4 client =
= Service parameters =
= _ = * optioninterface
=
= = *type s6-rc service = = _service
system.service.network.forward
Service parameters
option
enableIPv4
type boolean
option
enableIPv6
type boolean
- service
system.service.network.link
= hardware network interface =
= Service parameters =
= __ = ** =
- = option
devpath
= Path to the sysfs node of the device. If you provide this and the = ifname option, the device will be renamed to the name given by = ifname. =
= type null or string = ** =- = option
ifname
= Device name as used by the kernel (as seen in "ip link" or = "ifconfig" output). If devpath is also specified, the device will be = renamed to the name provided. =
= type string = * optionmtu
=
= _ = *type null or signed integer = = _service
system.service.network.route
Service parameters
- option
interface
= Interface to route through. May be omitted if it can be inferred from = "via" =
= type null or s6-rc service
- option
metric
= route metric =
= type signed integer
- option
target
= host or network to add route to =
= type string
- option
via
= address of next hop =
= type string
NTP
A network time protocol implementation so that your Liminix device may synchronize its clock with an accurate time source, and optionally also provide time service to its peers. The implementation used in Liminix is Chrony
path modules/ntp/default.nix
service
system.service.ntp
Service parameters
- option
allow
= subnets from which NTP clients are allowed to access the server =
= type list of stringoption
bindaddress
type null or string
option
binddevice
type null or string
option
dumpdir
type path
default
/run/chrony
option
extraConfig
type strings concatenated with "\n"
default
option
makestep
type null or (submodule)
option
peers
type attribute set of list of string
option
pools
type attribute set of list of string
option
servers
type attribute set of list of string
option
user
type string
default
ntp
PPP
ppoe
(PPP over Ethernet) provides a service to address the case
where your Liminix device is connected to an upstream network using
PPPoE. This is typical for UK broadband connections where the physical
connection is made by OpenReach ("Fibre To The X") and common in some
other localities as well: check with your ISP if this is you.
l2tp
(Layer 2 Tunelling Protocol) provides a service that tunnels
PPP over the Internet. This may be used by some ISPs in conjunction with
a DHCP uplink, or other more creative forms of network connection
path modules/ppp/default.nix
service
system.service.l2tp
Service parameters
- option
debug
= log the contents of all control packets sent or received =
= type booleanoption
lcpEcho
type unspecified
- option
lns
= hostname or address of the L2TP network server =
= type string
- option
password
= password =
= type null or string or function that evaluates to a(n) anything
- option
ppp-options
= options supplied on ppp command line =
= type list of string
- option
username
= username =
= type null or string or function that evaluates to a(n) anything
service
system.service.pppoe
Service parameters
- option
bandwidth
= approximate bandwidth in bytes/second. Used to calculate rate limits = for ICMP =
= type null or signed integer
- option
debug
= log the contents of all control packets sent or received =
= type boolean
- option
interface
= ethernet interface to run PPPoE over =
= type s6-rc serviceoption
lcpEcho
type unspecified
- option
password
= password =
= type null or string or function that evaluates to a(n) anything
- option
ppp-options
= options supplied on ppp command line =
= type list of string
- option
username
= username =
= type null or string or function that evaluates to a(n) anything
Secrets
path modules/secrets/default.nix
- service
system.service.secrets.outboard
= fetch secrets from external vault with https =
= Service parameters =
= __ = ** =
- = option
interval
= how often to check the source, in minutes =
= type signed integer = ** =- = option
name
= service name =
= type string = ** =- = option
password
= password for HTTP basic auth =
= type null or string = ** =- = option
url
= source url =
= type string matching the pattern https?://.* = ** =- = option
username
= username for HTTP basic auth =
= type null or string = __
- service
system.service.secrets.subscriber
= wrapper around a service that needs notifying (e.g. restarting) when = secrets change =
= Service parameters =
= __ = ** =
- = option
action
= how do we notify the service to regenerate its config =
= type one of "restart", "restart-all", "hup", "int", "quit", = "kill", "term", "winch", "usr1", "usr2" =
= default =restart-all= =
= optionservice
;; = subscribing service that will receive notification =
= type s6-rc service = =
= optionwatch
;; = secrets paths to subscribe to =
= type list of function that evaluates to a(n) anything = __ *+ service
system.service.secrets.tang
:: = fetch secrets from encrypted local pathname, using tang =
= Service parameters =
= _ = =
= optioninterval
;; = how often to check the source, in minutes =
= type signed integer = =
= optionname
;; = service name =
= type string = * =
= optionpath
;; = encrypted source pathname =
= *type path = _
Secure Shell
Provide SSH service using Dropbear
path modules/ssh/default.nix
service
system.service.ssh
Service parameters
- option
address
= Listen on specified address =
= type null or string
- option
allowLocalPortForward
= Enable local port forwarding =
= type boolean
- option
allowPasswordLogin
= Allow login using password (disable for public key auth only) =
= type boolean
- option
allowPasswordLoginForRoot
= Allow root to login using password (disable for public key auth only) =
= type boolean
- option
allowRemoteConnectionToForwardedPorts
= Allow remote hosts to connect to local forwarded ports (by default = they are bound to loopback) =
= type boolean
- option
allowRemotePortForward
= Enable remote port forwarding =
= type boolean
- option
allowRoot
= Allow root to login =
= type boolean
- option
authorizedKeys
= Authorized SSH public keys for each username. If this optin is = provided it overrides any keys found in /home/{username}/.ssh =
= type null or (attribute set of list of non-empty string) or function = that evaluates to a(n) anythingoption
extraConfig
type strings concatenated with " "
default
- option
port
= Listen on specified TCP port =
= type 16 bit unsigned integer; between 0 and 65535 (both inclusive)
uevent-rule
path modules/uevent-rule/default.nix
- service
system.service.uevent-rule
= a service which starts other services based on device state (sysfs) =
= Service parameters =
= __ = ** =
- = option
serviceName
= name of the service to run when the rule matches =
= type string = ** =- = option
symlink
= create symlink targeted on devpath =
= type null or string = * optionterms
=
= _ = *type attribute set = = _
VLAN
Virtual LANs give you the ability to sub-divide a LAN. Linux can accept VLAN tagged traffic and presents each VLAN ID as a different network interface (eg: eth0.100 for VLAN ID 100)
Some Liminix devices with multiple ethernet ports are implemented using a network switch connecting the physical ports to the CPU, and require using VLAN in order to send different traffic to different ports (e.g. LAN vs WAN)
path modules/vlan/default.nix
service
system.service.vlan
Service parameters
- option
ifname
= interface name to create =
= type string
- option
primary
= existing physical interface =
= type s6-rc service
- option
vid
= VLAN identifier (VID) in range 1-4094 =
= type string
Watchdog
Enable hardware watchdog (for devices that support one) and feed it by checking the health of specified critical services. If the watchdog feeder stops, the device will reboot.
path modules/watchdog/default.nix
service
system.service.watchdog
Service parameters
- option
headStart
= delay in seconds before watchdog starts checking service health =
= type signed integer
- option
watched
= services to watch =
= type list of s6-rc service
Appendix C: Outputs
Outputs are artefacts that can be installed somehow on a target device, or "installers" which run on the target device to perform the installation.
There are different outputs because different target devices need different artefacts, or have different ways to get that artefact installed. The options available for a particular device are described in the section for that device.
mtdimage
This creates an image called firmware.bin
suitable for squashfs or
jffs2 systems. It can be flashed from U-Boot (if you have a serial
console connection), or on some devices from the vendor firmware, or
from Liminix when using levitate
If you are flashing from U-Boot, the file flash.scr
is a sequence of
commands which you can paste at the U-Boot prompt to to transfer the
firmware file from a TFTP server and write it to flash. Please read the
script before running it: flash operations carry the potential to brick
your device
Note
|
TTL serial connections typically have no form of flow control and so
don’t always like having massive chunks of text pasted into them - and
U-Boot may drop characters while it’s busy. So don’t necessarily expect
to copy-paste the whole of |
tftpboot
This output is intended for developing on a new device. It assumes you have a serial connection and a network connection to the device and that your build machine is running a TFTP server.
The output is a directory containing kernel and root filesystem image,
and a script boot.scr
of U-Boot commands that will load the images
into memory and run them directly, instead of first writing them to
flash. This saves time and erase cycles.
It uses the Linux phram driver to emulate a flash device using a segment of physical RAM.
table: 0x8d8ba0
table: 0x8d8ba0
ubimage
This output provides a UBIFS filesystem image and a small U-Boot script to make the manual installation process very slightly simpler. You will need a serial connection and a network connection to a TFTP server containing the filesystem image it creates.
Warning
|
These steps were tested on a Belkin RT3200 (also known as Linksys E8450). Other devices may be set up differently, so use them as inspiration and don’t just paste them blindly. |
-
determine which MTD device is being used for UBI, and the partition name:
uboot> ubi part
Device 0: ubi0, MTD partition ubi
In this case the important value is ubi0
-
list the available volumes and create a new one on which to install Liminix
uboot> ubi info l
[ copious output scrolls past ]
Expect there to be existing volumes and for some or all of them to be important. Unless you know what you’re doing, don’t remove anything whose name suggests it’s related to uboot, or any kind of backup or recovery partition. To see how much space is free:
uboot> ubi info
[ ... ]
UBI: available PEBs: 823
Now we can make our new root volume
uboot> ubi create liminix -
3) transfer the root filesystem from the build system and write
it to the new volume. Paste the contents of result/flash.scr
one
line at a time into U-Boot:
uboot> setenv serverip 10.0.0.1
uboot> setenv ipaddr 10.0.0.8
uboot> setenv loadaddr 4007FF28
uboot> tftpboot $loadaddr result/rootfs
uboot> ubi write $loadaddr liminix $filesize
Now we have the root filesystem installed on the device. You can even
mount it and poke around using ubifsmount ubi0:liminix; ubifsls /
4) optional: before you configure the device to boot into Liminix automatically, you can try booting it by hand to see if it works:
uboot> ubifsmount ubi0:liminix
uboot> ubifsload ${loadaddr} boot/fit
uboot> bootm ${loadaddr}
Once you’ve done this and you’re happy with it, reset the device to return to U-Boot.
5) Instructions for configuring autoboot are likely to be very
device-dependent and you should consult the Liminix documentation for
your device. (If you’re bringing up a new device, some detective work
may be needed. Try running printenv and trace through the
flow of execution from (probably) $bootcmd
and look for a suitable
variable to change)
-
Now you can reboot the device into Liminix
uboot> reset
updater
For configurations with a writable filesystem, create a shell script that runs on the build system and updates the device over the network to the new configuration
vmroot
This target is for use with the qemu, qemu-aarch64, qemu-armv7l devices.
It generates an executable run.sh
which invokes QEMU. It connects
the Liminix serial console and the
QEMU monitor to
stdin/stdout. Use ^P (not ^A) to switch between monitor and stdio.
If you call run.sh
with --background /path/to/some/directory
as
the first parameter, it will fork into the background and open Unix
sockets in that directory for console and monitor. Use nix-shell
--run connect-vm
to connect to either of these sockets, and ^O to
disconnect.
Liminix VMs are networked using QEMU socket networking. The default behaviour is to connect
-
multicast 230.0.0.1:1234 ("access") to eth0
-
multicast 230.0.0.1:1235 ("lan") to eth1
Refer to border-network-gateway
for details of how to start an
emulated upstream on the "access" network that your Liminix device can
talk to.