| Betreuer | Niklas Gollenstede |
| Projekt | ATLAS |
| IBR Gruppe | VSS (Prof. Dietrich) |
| Art | Bachelorarbeit |
| Status | offen |
AbstractStoring and especially fetching of industry-standard container images is inefficient, especially when lower layers are changed. As a result, base images and libraries are often not updated sufficiently frequently. In prior work, we showed that the Nix build and deployment tool can be used to remove redundancies in stored images and transfer them differentially and/or with fine granularity. The starting point for the bachelor or project thesis offered here is to make an industry-standard OCI runtime (Docker/Podman) use the extracted images that have been pushed to a host via Nix, and to investigate advantages (storage/bandwidth savings, performance gains) and limitations (partially defunct tooling, missing support for file ownership) of doing so. Background / Problem DescriptionContainers are lightweight, isolated environments for running applications. The Open Container Initiative (OCI) defines standards for containers that are well supported by commonly used tooling. OCI images are built in consecutive steps. A common workflow is to start with a base image like Ubuntu or Alpine Linux (for containers), then install specific packages, copy in custom sources, compile those, and finally set configuration. From each step to the next, all modified/new files are captured in a layer that is identified by a hash over its contents. When a OCI runtime needs an image, it fetches all the layers it doesn't have yet, and stacks them over each other, commonly with an overlayfs. When the build instructions for a layer change, it and all following layers will be rebuilt and, unless the build and archiving is bitwise-reproducible, re-transmitted. Using the Nix "package manager", wich is also a general-purpose build framework and deployment tool, we can take an externally built / existing OCI image, collapse its layers into a single file tree, removing any redundancies, and move that single build artifact to the target device, either explicitly or as part of a system update. As part our LCTES '23 paper, we created a differential transfer mechanism for Nix packages: if a package of presumably similar contents can be found on the target, then only files or even pats of files that are different between the current and previous version need to be transferred. This removes much of the overhead introduced by transferring whole stacks of image layers (or trees of Nix dependencies) after minuscule changes. Task / ApproachSo far, we have only done preliminary tests to actually run the applications in the container images. The images are transferred to the target device, and they contain all the application files they should, but they are not installed for an OCI runtime yet -- doing that is your job. One approach would be to reverse-engineer at which specific paths accompanied by which metadata files the runtime requires the extracted images to be. Another would be to implement a completely new, but hopefully rather simple, storage backend. You are to choose and implement an appropriate approach and evaluate its resulting properties. One limitation of our approach is that all files in the Nix-store, and thus our extracted OCI images, are read-only, root-owned and world-readable. The only difference in terms of file permissions that the Nix-store allows for is wither files are executable or not. Paired with an overlayfs for a writable upper layer, or more targeted mounts of tmpfses, and volume mounts for persistent mutable data, this should be sufficient for almost all applications. Some applications, however, may refuse to run (or break) when file ownership is not as they expect. An advantage, past the storage and bandwidth savings, could be that at runtime, we don't need to stack any layers. If we don't need/use a writable layer over the whole container file system, we can run containers without overlayfs. This may provide a filesystem performance benifit. Further Readings / ListeningsYouTube:
| |