Skip to content

systemd

About systemd in Hyperscale

We try to follow the systemd build from Fedora rawhide as closely as possible.

CI/CD

Links can be found on the CI/CD page.

Contributing to the RPM spec / New Version Updates in the SIG

Make sure you're onboarded onto the SIG before following these steps.

For the following examples, most/all of the links will go to the c10s-sig-hyperscale branch when there are mentions of the RPM sources repo. Substitute as needed with other branches (e.g. c9s-sig-hyperscale).

When we're building for 2 releases (e.g. c10s and c9s), you can do most/all of the work on one branch (e.g. c10s) and merge the changes to the other branch (e.g. c9s) using the power of Git. Unless something happens, like a dependency on a newer release not being available on an older release, rarely will changes on the newer branch fail on the older one.

Steps:

  • Clone the CentOS RPM sources repo
    • Checkout the c10s-sig-hyperscale branch
  • Add the Fedora RPM sources repo as a git remote
    • git remote add fedora https://src.fedoraproject.org/rpms/systemd && git fetch fedora
  • Merge changes from Fedora Rawhide.
    • git merge fedora/rawhide
    • Solve any merge conflicts
    • Carefully go through each commit from Rawhide. Take special care for commits that introduce changes in default behavior (e.g. https://src.fedoraproject.org/rpms/systemd/c/ba02e904964116b848080ca72243174f4ef3eced?branch=ba02e904964116b848080ca72243174f4ef3eced) as these can have a drastic impact on our users. Whenever finding such a commit, bring it up for discussion with the other members of the SIG to determine how we should handle the change.
    • Another thing to look out for is newly added Obsoletes for the systemd package itself, for example, Obsoletes: systemd < 246.6-2. These prevent the old version from being installed anymore when the package with the Obsoletes is added to the rpm repository and should be removed until we can be reasonably sure that a newer version of systemd than is being obsoleted is installed everywhere.
  • Next, you can check the remaining differences between the Fedora rawhide spec and the CentOS Hyperscale spec by running git diff fedora/rawhide..HEAD while on the c10s-sig-hyperscale branch. Try to remove any differences between the two that aren't strictly necessary anymore.
  • Include patches that haven't been merged upstream yet or which haven't been backported to a stable branch
    • Note that we're very hesitant on including downstream patches and there needs to be a very good reason to include downstream patches. Every downstream patch should have a comment to an upstream pull request where the functionality is proposed for inclusion in systemd itself.
    • If the patch has been upstreamed but isn't in a stable release yet, you can ask upstream for a new stable release or prepare one yourself using the instructions found here. Urgent fixes or newer feature backports that aren't appropriate for stable releases can be added as patches to the rpm repository itself. To apply a patch to the rpm repository, you can do the following:
    • Add something like PatchXXXX: https://github.com/systemd/systemd/pull/$PR.patch to the rpm spec
    • If a PR doesn't apply cleanly to the stable branch, cherry-pick the commits into a subbranch of the corresponding stable branch in the systemd repository and open a PR on Github which you can then list in the rpm spec. If the patch is a feature backport, you can immediately close the PR as it won't be merged anyway.
  • Update the version and release.
    • If we're building rpms for the same systemd stable release (from https://github.com/systemd/systemd/releases) as is shipped in Fedora Rawhide, no changes are required to the spec.
    • If we're building a different release than the one shipped in Fedora Rawhide, update the default value for the Version field to match the version of the stable release that we want to build (e.g. 256.2).
    • Update the Release value. This should match Fedora, except that we also append a dot number value to indicate which iteration of the Hyperscale release we're on (e.g. if Fedora has 256.2-1, Hyperscale should have 256.2-1.1).
  • Note that generally, aside from changing the version, release and including patches, all other changes should be implemented in the Fedora spec and backported to the Hyperscale spec to keep the diff between the two as small as possible.
    • Exceptions can be made when Fedora makes a change that we want to explicitly want to deviate from.
    • The upstream systemd repository builds rpms for CentOS Stream using the Fedora Rawhide spec, so the more our spec matches the Fedora Rawhide one, the better our test coverage.
  • Download the source tarball and patches from the systemd github repository:
    • spectool --define "_sourcedir $PWD" -g systemd.spec
  • Update the sources file with the new tarball name and SHA512 hash.
    • Use sha512sum to calculate the SHA512 hash of the tarball.
  • Upload the new version tarball to the CentOS lookaside cache.
    • Upload the tarball to the CentOS lookaside cache by calling the lookaside_upload_sig script found here.
    • For example, the command could look as follows:
    • centos-git-common/lookaside_upload_sig -f systemd-256.2.tar.gz -n systemd
  • Build locally with mock
  • mock -r centos-stream-hyperscale-10-x86_64 -D "%dist .hs.el10" --sources . --spec systemd.spec
  • The resulting rpms can be found in /var/lib/mock/centos-stream-10-x86_64/result
  • Test that the new rpms work as expected (See Testing section)
  • All updates are in and working as expected? Push your changes to a fork of the rpm sources repository, create a pull request and ask the other Hyperscale maintainers to take a look!

Testing

Submit MR to GitLab repo (https://gitlab.com/CentOS/Hyperscale/rpms/systemd). The CI system will automatiicaly build and run systemd integration tests.

Note: Information in sections below maybe outdated.

Testing SELinux with the systemd-releng repository To test use cases not covered by the systemd integration tests, we have a separate set of mkosi configuration files in the [systemd-releng](https://gitlab.com/CentOS/Hyperscale/releng/systemd-releng) repository. Due to SELinux limitations, all mkosi commands in the selinux-releng repository have to be run as the root user as we cannot do arbitrary SELinux relabeling unless we're running with root privileges. First, clone the repository (or use an existing checkout): ```shell git clone https://gitlab.com/CentOS/Hyperscale/releng/systemd-releng.git cd systemd-releng ``` Next, build a Hyperscale image, again including the downloaded rpms: ```shell sudo mkosi --profile hyperscale -r --volatile-package-directory -f ``` We can now try to boot into the image: ```shell sudo mkosi qemu ``` If the boot succeeds and we end up in a root shell, great! Otherwise, we'll need to debug. If the boot failed this is either due to SELinux denials or because dracut hasn't been updated to accomodate changes in newer systemd releases. We can start by booting with SELinux in permissive mode: ```shell sudo mkosi --kernel-command-line-extra=enforcing=0 qemu ``` If this still doesn't get you into a root shell, the problem might lie with dracut, see the next section for more details. Otherwise, regardless of whether we're in enforcing or permissive mode, the first thing to check is whether there's any SELinux denials: ```shell journalctl -t audit -g AVC ``` If there are denials, we can compare them with the denials on a Fedora Rawhide system by running the following: ```shell sudo mkosi -d fedora -f qemu journalctl -t audit -g AVC ``` If the denials don't pop up on Fedora Rawhide, that's a good indication that we should backport a newer version of [selinux-policy](https://git.centos.org/rpms/selinux-policy). To backport a newer version, you can run the following: ```shell git clone https://git.centos.org/rpms/selinux-policy.git cd selinux-policy git remote add fedora https://src.fedoraproject.org/rpms/selinux-policy git fetch rawhide git checkout rawhide fedpkg sources git checkout cs-sig-hyperscale git merge fedora/rawhide ``` To double check that the newer version fixes the denials, first use `mock` to build the rpms: ```shell mock -r centos-stream-hyperscale--x86_64 --sources . --spec selinux-policy.spec ``` Then build and boot the image with the new selinux-policy rpms and check if the denials are fixed: ```shell cd sudo mkosi \ --profile=hyperscale \ --release= \ --volatile-package-directory= \ --volatile-package-directory=/var/lib/mock/centos-stream-hyperscale--x86_64/result \ --force \ qemu journalctl -t audit -g AVC ``` If the denials that were shown previously are gone, great! The new version of selinux-policy includes fixes for the denials you were seeing. In this case you can go ahead and build the new selinux-policy rpm in Hyperscale: ```shell cd /lookaside_upload_sig -f macro-expander -n selinux-policy /lookaside_upload_sig -f container-selinux.tgz -n selinux-policy /lookaside_upload_sig -f selinux-policy-*.tar.gz -n selinux-policy git checkout cs-sig-hyperscale git push cbs build hyperscales-packages-main-els git+https://git.centos.org/rpms/selinux-policy.git#(git rev-parse HEAD) cbs tag-build hyperscales-packages-main-release selinux-policy-.hs.el ``` If the denials also occur on Fedora Rawhide, file an issue for the [selinux-policy](https://github.com/fedora-selinux/selinux-policy/issues) repository on Github. Include the lines from the `journalctl` output showing the denials. Also include a reproducer to allow the maintainers to reproduce the issue. A reproducer will generally look as follows (you can copy paste this and modify it): ```shell git clone https://github.com/systemd/mkosi.git ln -s /usr/local/bin/mkosi $(pwd)/mkosi/bin/mkosi git clone https://gitlab.com/CentOS/Hyperscale/releng/systemd-releng sudo mkosi -d fedora -f qemu ``` You can extend the above with any extra commands required to reproduce the SELinux denials in the virtual machine. Now it's a matter of waiting and working with the policy maintainers to fix the denials, and backport the newer version of selinux-policy once it is released in Fedora Rawhide (see above steps).
Testing the dracut backport with the systemd-releng repository Because new systemd releases often require changes in dracut as well, we also maintain a backport of dracut from Fedora Rawhide. Every time a change in a new release of systemd requires a corresponding change in dracut, a newer release of dracut should be backported first before doing a new release of systemd in the Hyperscale SIG. To backport a new version of dracut from Fedora Rawhide, you can run the following steps: ```shell git clone https://git.centos.org/rpms/dracut.git cd dracut git remote add fedora https://src.fedoraproject.org/rpms/dracut git fetch rawhide git checkout rawhide fedpkg sources git checkout cs-sig-hyperscale git merge fedora/rawhide ``` We can then build the new dracut rpms with mock: ```shell mock -r centos-stream-hyperscale--x86_64 --sources . --spec dracut.spec ``` And finally test whether the image built with mkosi boots with the new version of dracut as follows: ```shell cd sudo mkosi \ --profile=hyperscale \ --release= \ --volatile-package-directory= \ --volatile-package-directory=/var/lib/mock/centos-stream-hyperscale--x86_64/result \ --force \ qemu ``` If everything works as expected, we can build and tag the new dracut version in CBS: ```shell cd /lookaside_upload_sig -f -n dracut git checkout cs-sig-hyperscale git push cbs build hyperscales-packages-main-els git+https://git.centos.org/rpms/dracut.git#(git rev-parse HEAD) cbs tag-build hyperscales-packages-main-release dracut-.hs.el ``` If there are still issues with booting, you'll need to debug the issue and resolve it together with upstream. Be sure to also extend this section with any useful debugging information to make the process easier for the next contributor.
Testing upgrades from stock CentOS Stream with the systemd-releng repository To test upgrades from stock CentOS Stream to CentOS Stream + Hyperscale, you can build and boot without the Hyperscale profile and then upgrade from within the virtual machine: ```shell sudo mkosi -f qemu dnf upgrade ``` The packages enabling the CentOS Hyperscale repositories are automatically installed when building the image, so running `dnf upgrade` after booting is sufficient to upgrade the system to CentOS Hyperscale. After upgrading, run `journalctl -p warning` to see if anything went wrong during the upgrade.
CBS Build Commands For CBS you will need to be [onboarded to the SIG](onboarding.md). From inside the fork of the [systemd Hyperscale RPM sources](https://git.centos.org/rpms/systemd/tree/c9s-sig-hyperscale) repo: Once the specfile changes are pushed you can do a real build: ```shell git checkout cs-sig-hyperscale # Scratch build cbs build --scratch hyperscales-packages-main-els "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)" # Official build cbs build hyperscales-packages-main-els "git+https://git.centos.org/rpms/systemd.git#$(git rev-parse HEAD)" # Tag it for testing cbs tag-build hyperscales-packages-main-testing systemd-.hs.el # Tag for release (this will let it go to the mirrors) cbs tag-build hyperscales-packages-main-release systemd-.hs.el ``` To do builds in the facebook tag, use the above commands but replace `main` by `facebook`.