Packaging guidelines (draft)

ermo

@Snekwranglers

Please organise your knowledge here. All of you should have edit access to all post tagged Documentation.

Documentation

As the state of the tooling is still in flux, the best thing to do to get an idea for how things work is to look at the recipes in the main snekpit recipe repo.

The other thing to do is to look at the macro %actions and %(definitions) in the boulder code repo.

The naming of concepts has been tweaked compared to what some of you might be used to, with the primary purpose being more sensible and clear semantics in terms of natural speech and its supporting mental models.

Additionally, it might be worth pointing out that this is not $otheros!

You are encouraged to forget everything you thought you knew about how things are supposed to work and be open to the different goals of Serpent OS which spawned the tooling you're about to use. The fully fleshed out workflow is also not yet ready for you to use, so things will feel rather manual initially.

How to get started writing a recipe

To get a basic recipe, use boulder new -h to get started.

Thanks to work done by our contributors, git upstreams can now be used in recipes .

Packaging Guidelines

Follow these guidelines and build consensus by talking to your peers if a situation arise that they don't cover. If the BDFL chips in, listen carefully.

Naming

Package names and categories (=dirs) are all-lowercase: lang/python, console/nano

Package splitting

No standalone -devel packages. If a package is all headers, they should be in the %(name) package

The recipe `pkg/` dir and `%(pkgdir)`

Place extra files such as patches, vendor configs etc. in the pkg/ dir inside the recipe dir. Files inside pkg/ will automatically be copied over to %(pkgdir) during boulder builds, from where each can be referenced as %(pkgdir)/path/filename in the recipe. The directory name pkg/ was chosen because it's short and stands out in tree invocations.

What to commit

git add manifest.x86_64.bin manifest.x86_64.jsonc monitoring.yml stone.yml

manifest.x86_64.bin Binary moss readable file used to create moss collection stone.index files for dep resolution etc.
manifest.x86_64.jsonc Human readable version of manifest.bin used for git diff introspection purposes
monitoring.yml See the security monitoring discussion
stone.yml The actual build recipe

Commit messages

The goal here is to make the git shortlog of the monorepos easy to read at a glance.

Use a category/pkg: verb <foo> format with one commit per package change.

Examples:

shells/fish: Add vX.Y
shells/fish: Update to vX.Y
shells/fish: Fix (...)

Bump release for functional recipe changes only; do not bump release for no functional change commits (documentation typos, formatting).

No functional change commits use the [NFC] tag (policy in clang, worth adopting):

shells/fish: [NFC] Tweak comments.

Recipe licensing

We only accept ~~Zlib~~ MPL-2.0 licensed code and recipe contributions due to the desirable nature of its patent protections. boulder new has been updated to use a REUSE compliant license header, but existing recipes will need to be updated with the following REUSE-compliant verbiage at the top:

#
# SPDX-FileCopyrightText: © 2020-2023 Serpent OS Developers
#
# SPDX-License-Identifier: MPL-2.0
#

davidjharder

Going to dump my work-in-progres doc here.

It is organized as bullet points for the stone.yml keys, and includes a a lot of basic info that should exist somewhere.
Some is stolen from the linting discussion here https://forums.serpentos.com/d/37-boulder-moss-format-lint-suggestions-for-future-reference.
"Linting rules" are suggestions at this point

`stone.yml` Keys

`name`

The name should match the upstream name, Except:
- When grouping packages by language. Example: python-markupsafe
- Fonts add -font to the name. Example: terminus-font (not a rule yet)
- Icon themes add -icon-theme to the name. Example: papirus-icon-theme (not a rule yet)
Linting rules: only: lowercase, letters, dashes, numbers, underscores?

`version`

The version should match upstream
Linting rules: enclosed in single quotes, no whitespace?

`release`

Release is an integer value that increases by +1 with each build, even when no other keys are changed.
Linting rule: increasing integer

`summary`

Not required (should it be?), but strongly preferred
A sentence explaining the package. Often taken directly from the upstream homepage.
Omit trailing periods
Linting rule: single line

`license`

A list of one or more SPDX identifiers https://spdx.org/licenses/
boulder new scans the given upstream source and generates a list, but the packager should still check the generated list for correctness and completeness. Remember, boulder new can only scan one upstream source.

`homepage`

Required
Linting rule: URL

`description`

Not required, but strongly preferred
If the upstream project has a longer explanation of the package put it here. Otherwise, copy summary (common practice, but should this be changed?, required??)
Limit the description to one paragraph
Use line breaks. Or don't; idc
Do not use the description for user instructions

Upstream sources: `upstreams`

One or more tarball or git sources
Tarballs are preferred over git sources. If possible, use official source tarballs provided by the upstream project though automatically generated tarballs are fine as well.
Git sources should use the official git upstream for a project and not a mirror. Use the exact commit a particular tag points to, instead of the tag itself, and then put the tag in a comment. Example git|https://github.com/KhronosGroup/SPIRV-Tools.git : 63de608daeb7e91fbea6d7477a50debe7cac57ce # sdk-1.3.239.0. This helps in reproducibility since you're guaranteed to get the same source every time (tags can be changed).
The first source is extracted to ???. Subsequent sources are extracted to ??

`networking`

Allows network access when the package is being built
Can be omitted when using the default value networking : no

Dependencies: `buildeps`, `rundeps`, `checkdeps`

List components required in the for each stage. Omit if empty.
Use pkgconfig() whenever possible for buildeps
Individual binaries should be listed as binary(name) Example: autoconf
Sort components alphabetically, starting with binary(), then pkgconfig(), then packages. OR straight alphabetically??
Do not include components provided by the base tooling
Do not include redundant components
Linting rule: order

Build stages: `setup`, `build`, `install`, `workload`, `check`

Omit keys that are not required. Many packages use only build and install
use macros provided in boulder. No documentation, refer to source https://github.com/serpent-os/boulder/tree/main/data/macros/actions
keys like name and version can be accessed with %(name) and %(version)
Aliases for common directories are here https://github.com/serpent-os/boulder/blob/main/data/macros/arch/base.yml
Examples of common packaging, possibly with their own guidelines
- golang
- patching
- python
- rust

`toolchain`

Omit when using defaults. By default, Serpent uses the clang toolchain.
If a package requires a different toolchain use this key to change it. Example: toolchain: gnu
Refer to boulder source for defaults
Where should this key go? Sometimes found after install, sometimes after license

`tuning`

Omit when using defaults
Refer to boulder source for defaults, look for the tuning key https://github.com/serpent-os/boulder/blob/main/data/macros/arch/base.yml

Other `stone.yml` topics to consider documenting

Comments
Common workflows: bumping, updating
Directly link to more examples stones
package key

Botanium

A nice guide to let the community members start packaging, but still complicated and has many manual steps for noobs like me.

ermo

Botanium A nice guide to let the community members start packaging, but still complicated and has many manual steps for noobs like me.

Unavoidably so, as we're in pre-alpha where we have a rough idea of how stuff will end up working, but reserve the right to make changes. Hence, it'd be an extra burden to maintain up to date documentation, which is why we haven't really attempted doing so as of yet.

In summary, our target audience in the current stage of tooling (non-)maturity is primarly people who are (very) used to packaging software across a range of packaging formats.

As we flesh out the tooling and workflows more and address misfeatures and bugs, the tooling will stabilise and it'll make sense to begin to onboard new packagers with comparatively less experience.

Keep checking in periodically; we're happy to have you here!

ReillyBrogan

monitoring.yml

monitoring.yml is used for automating the detection of package updates and CVEs. It has the following format:

releases:
  id: 205088
  ignore:
    # We only update to the n-1 stable release. So for now we're only interested in 252.x updates
    - "253.*"
security:
  cpe:
  - vendor: systemd_project
    product: systemd
  - vendor: freedesktop
    product: systemd
  ignore:
  - CVE-2022-55555

Breaking this down:

releases.id refers to the Anitya ID. Anitya is a service ran by Fedora volunteers that integrates with a ton of package sources such that new releases are detected automatically. To determine this go to https://release-monitoring.org and search for the package. You may need to figure out which of similar-looking packages this refers to. Once you've identified the correct package you can see the ID in the URL bar for that package.
releases.ignore is a list (or array) of regex rules for ignoring particular releases that are returned from the Anitya API. Anitya is not particularly smart and has no filtering on the Anitya side, so we use this regex to exclude any tags/releases that Anitya thinks are "newer" than our version but are actually not. We can also use this to exclude releases that we are not ready to update to. Note that certain regex characters can break yaml formatting, in that case you may want to enclose the regex in quotation marks. Please add a comment above each ignore entry explaining why you are ignoring that version/string for future reference.

CPE:
The easiest way known to figure out the CPE is to run the following command (replacing systemd with whatever the package name is):

curl -s -X POST https://cpe-guesser.cve-search.org/search -d "{\"query\": [\"systemd\"]}" | jq .

This will return an output that looks like this:

[
  [
    39295,
    "cpe:2.3:a:ubuntu_developers:systemd"
  ],
  [
    103137,
    "cpe:2.3:a:lennart_poettering:systemd"
  ],
  [
    106991,
    "cpe:2.3:a:systemd_project:systemd"
  ],
  [
    107046,
    "cpe:2.3:a:freedesktop:systemd"
  ]
]

This output is broken down into IDs and CPE strings. The numeric IDs are useless to us and can be ignored. The CPE string is separated by semicolons and the second to last and last fields are vendor and product respectively. Some CPEs are for the same product but through a more specific vendor (like ubuntu_developers) and can be ignored. Others are probably not the canonical (heh) CPE that CVEs are most likely to be published under (like lennart_poettering) and can also be ignored. In this example the most likely vendors that a CVE will be published under are systemd_project and freedesktop so we pull those into our monitoring.yml (this is why the CPE section is a list/array).

When querying this API please lowercase the product name otherwise you are unlikely to get a result (like libXinerama -> libxinerama). You may also need to try a few different variations if the package is known by different names as the CPE search is not a fuzzy search.

CPEs also only exist for products/packages that have had CVEs reported against them in the past. This means that some packages do not have CPEs. In this case please put in the following (replacing the date)

# No known CPE, checked 20230323
security: {}

In general it should be fairly easy to know whether a package should have a CPE (IE it has had CVEs in the past) if you are familiar with the package. If you are unsure and the CPE API is returning an empty result please ask on Matrix or here on the forums and we'll do our best to help you find it or tell you that it most likely does not have a CPEs.

davidjharder

@ReillyBrogan how do you look up cpe vendor/product?

ReillyBrogan

davidjharder That post was a WIP and I just updated it.

davidjharder

Here's something for patching

Patching

Use the %patch macro to apply patches during the setup stage

setup       : |
    %patch %(pkgdir)/0001-stateless-config.patch 
    %patch %(pkgdir)/use-some-words-to-describe-the-patch.patch

All patches should be placed in the pkg directory. You can organize the pkg directory with sub-directorties if needed.

├── stone.yml
├── pkg
│   ├── 0001-stateless-config.patch 
│   └── use-some-words-to-describe-the-patch.patch

Notes

Patching order goes by the order in stone.yml, not by filename
Give the patches a descriptive filename. Example: 3rd-patch.patch is not descriptive enough, 0001-stateless-config.patch is better
Unless you wrote the patch, explain the source of the patch in the patch file. Include git format-patch headers, or add a comment

Examples

The openssh package carries a bunch of patches, organized in sub-directories: https://github.com/snekpit/main/blob/87c177774d2c60612eba85164443949670311934/console/openssh/stone.yml

Source

Refer to Boulder source for the macro definition: https://github.com/serpent-os/boulder/blob/main/data/macros/actions/misc.yml

Future

Patching series macro?