Will Maidsafe be in Debian repositories?

I am personally using yunohost it has lots of apps ready to go! had some hickups with nextcloud I dont know if I failed to maintain in somehow or it was yunohost problem

Hi @jonas

I work for Maidsafe and I’m interested in whether we might be able to provide some support for this work.

Just to be clear, this work is to enable installing two packages via apt? Like so:

apt-get install safe-network-node
apt-get install safe-network-cli

However, if I’m honest, in general I don’t know a great deal about what goes into getting anything in the repositories. I hope you might forgive my ignorance and possibly answer some questions please?

Why do you need to have a mirror of the codebase? Is that a standard thing for all packages?

I don’t completely understand what this means. Does every crate in the entire dependency graph need to also be mirrored inside this Debian Gitlab?

When you build Rust code, do you use an alternate source from crates.io for pulling in the dependencies?

How do you normally keep up to date with upstream?

What exactly do you mean here in terms of testing? That the binaries would install correctly? Or that they run and work correctly?

I notice your version, 0.46.5, is very old. Development of the network has been proceeding so fast, that this will bear almost no resemblance to what we have now. Can we get a newer version?

Cheers,

Chris

10 Likes

Hi @chriso

Correct, the goal of my work is that on an official pristine Debian system you can type (your commands above, or) apt install safe-network-node safe-network-cli and get sn_node and sn_cli installed - integrated with the system, and security-tracked and upgraded as integral part of that.

Yes: All code distributed as part of Debian must be built from source in a pristine Debian environment, and must be possible to rebuild using only the distributed system.

Yes (in one interpretation of your question): All crates must be included in source form as part of Debian, and built from source to produce all binary packages part of Debian.

Needed code (crates and C libraries and build tools) need to be included with Debian, but need not be duplicated in each source package. In fact it is strongly discouraged to duplicate code in Debian, which is the main reason the current draft package is not released into Debian yet: I want to reduce the amount of embedded crates, ideally to zero, before formally pushing it to Debian.

Also, Salsa is not officially part of Debian, but is just a common (by far!) place to prepare packages to be pushed to Debian. Debian predates the invention of git, and a “Debian source package” really is a couple of tarballs (one with upstream code and one with packaging code) and a cryptographically signed metadata file.

But loosely speaking, yes: All crates in the graph must be packaged for Debian!

Yes: When a Debian package is built only other binary Debian packages can be pulled in as build-dependencies.

So each and every build-dependency need to exist as a Debian binary package (or cheating and embedding, as I do now for the draft package).

I normally maintain packages with a "slightly slower release frequency :wink:

[Not sure if you are asking how well I usually keep up, or how I techincally do, or…]

For each package update (ideally each upstream release, but can skip some or many upstream releases, there is no strict rule), I download the upstream release tarball (or when that is not available - as is the case for crates - I synthesize a tarball from upstream git release tag), and update my packaging tarball to align with that newer upstream code.

I maintain (alone or as part of teams) roughly 600 packages in Debian. By far the most of them have much fewer dependencies than to this one, though.

Here I meant invitations for regular not-so-geeky users to help out.

…but I guess your interest is more how strong testing we can expect.

Currently there is very sloppy testing: I simply execute cargo test, skipping some tests I experienced failing (likely because they rely on network but maybe I miss some environment setup) and ignore the result.

Obviously I hope that can change to check all tests and to fail the build if any of the tests fail.

Besides the build-time testing I also expect to add runtime tests, when package is officially in Debian and I can rely on Debian infrastructure for that.

Perhaps it is helpful to you to see how a Debian developer “dashboard” looks like - here is one for another (very simple) Rust package: precious - Debian Package Tracker

And here is one for a non-Rust package with test failures, that has caused the package to be kicked out of the “testing” branch of Debian: eye - Debian Package Tracker

Certainly: The version listed in initial post of the “ITP” (intent to package) bugreport reflects where work initially began, more than 4 months ago. To check which version the packaging is currently aligned with, see topmost section of the package changelog.

Feel free to ask more questions. We can also meet for an audio chat if you like - I host a lightweight videoroom service myself at https://live.jones.dk/ that I prefer to use for that purpose. Just say a time that is suitable for you - I am generally online and flexible with time (rarely in fixed-time meetings).

10 Likes

Here’s a summary of reasons MaidSafe might appreciate their code being officially in Debian: AdvantagesForUpstream - Debian Wiki

3 Likes

Hi @jonas ,

Thanks for the response. It’s interesting to get some insight into how the process works.

Can I just clarify I understand this correctly please? Does this mean, for example, if I installed the current LTS version of Debian on a VM, given the source for that LTS distribution, I would be able to build that same version without installing any dependencies?

If you don’t mind please, I also have other general questions related to this. They aren’t related to anything in particular regarding a package for Safe, and are more just out of curiosity. What is the source for the Debian distribution? What I mean by that is, the source package isn’t just say, a bunch of C code? It’s something that’s pulling together all the packages and stuff? From what I understand, the Linux kernel is a separate thing that’s common to all distros. Does the Debian source include a copy of the kernel source at a particular version? Or do you pull the kernel source in as part of your build process?

I think I understand what you’re saying here. You want the source for each individual crate? So if Safe shares the same crate with another Rust-based package, you’ll only have one copy of the source? (Though I don’t think I quite understand how you would deal with multiple versions of each crate.)

Btw, we normally use musl rather than glibc. Not sure if that’s something you need to take into account?

The thing I don’t understand here is, if you’re using cargo build, how do you get that to use an alternate source for the dependent crates? I did do a brief bit of Googling, but I can’t see how you can hook into Cargo to do that.

What is “embedding” in this context please? Specifically I’d be interested to know what you’re doing to hook into Cargo (if that’s what you’re doing).

When you say “upstream release tarball”, is that source or binary? If it helped at all, we could easily extend our Github releases to include a source tarball and I’d be happy to spec it the way you needed it. Or is getting the source from the tag enough?

Right, I was just wondering if it was something similar to the community testing on this forum. I wonder if those would help satisfy any criteria for testing for Debian? You can see an example of a ‘Comnet’ session here.

Just FYI, getting all the tests to execute for Safe, as per what we do in our CI and release process, does require a running network. Unfortunately it’s not just as simple as running cargo test. I would be happy to help you get a similar test suite to what we use for release and CI. Just to say though, it will change over time, so I’m not sure how you would keep up with those changes.

Are these automated? Or are they reports from someone who ‘manually’ ran some test procedure for a given package?

Thanks for taking the time. Right now I would prefer to just keep the discussion text based, if you don’t mind.

Cheers,

Chris

7 Likes

Correct.

BTW, you seem to use Ubuntu lingo above, and Debian release cycle is different: Debian LTS means the continuation of a subset of Debian beyond its regular lifespan of ~3 years to ~5 years.

Very roughly, regular Ubuntu releases compare to Debian testing and Ubuntu LTS releases to Debian stable.

I totally love your questions. All of them, and probably all to come as well. So feel free to throw them at me without the kind and polite wrapping :slight_smile:

Debian source includes everything needed to compute, even on a desert island with no internet access. That includes source code for a kernel (i.e. Linux for most architectures but a *BSD kernel or the Hurd on more experimental ones), and including source code for the GCC compiler and binutils to actually build the kernel and everything else.

All code in Debian is organized as Debian packages, which exist in source form and binary form. A Base System consisting of about 140 packages are always installed using a special bootstrapping process, and then both those and the remaining ~50’000 packages are (in almost all cases) added/updated/removed/purged using the APT package management tool.

Correct.

To take an example: The crate clap is included in Debian as source package rust-clap which builds binary package librust-clap-dev which virtually “provides” librust-clap-2-dev (“Provides” is alternative names, kinda equivalent to symlinks). Other packages declare a dependency on that package, either using its main name if any version goes, or more commonly using its provided versioned name to stick to that branch of the crate.

Later on, clap v3 is released, and binary package librust-clap-dev is changed to provide librust-clap-3-dev and stop providing librust-clap-2-dev.
It turns out, however, that reverse dependencies exist in Debian that still need clap v2 (and cannot be coerced to change that, by upgrading or by applying Debian-specific patches). So a new source package rust-clap-2 is introduced, with (typically main, but could be virtually provided) name librust-crate-2-dev.

The idea is that the unversioned source package tracks the “head” of upstream releases, and optionally versioned source packages are “branched out” as needed to cover non-newest upstream branches as needed. And then those versioned packages can be dropped when demand diminishes for them.

With rust crates you have the option of “yanking” a release. Technically we comparably have the option of “dropping a package” but where crates are offers for use, all Debian packages are actively used so more care is needed before “yanking” in Debian.

Possibly.

Default compiler in Debian is GCC, and default libc is glibc.

Debian packages are generally (but not always) intended for execution on a Debian system, where glibc is already available and used by core tools as a shared library. So most commonly there is no benefit in linking with another more lightweight libc.

But if there is a special reason to link specially, which makes sense for the scope of Debian use, then we do have musl available as well. And a range of alternate compilers too.

First I collect all sources and tools needed for building a package, then I turn off internet and build it.

Commonly when using cargo, it does multiple things at once, but with some careful structuring you can separate the process, to ensure that when you call cargo build then it only builds.

This is similar to how git pull internally does a git fetch and then a git merge, which makes sense in casual use but you can choose to be more strict when needed.

Essentially I do cargo vendor as part of preparing a package for being buildable in a non-networked clean-room environment.

The concrete commands that I invoke are documented in debian/README.source - that file exists when Debian package maintainers want to communicate to their peers that something unusual is needed to manage the source package. Ideally - when all dependent crates are in Debian - I can delete that file because then only regular Debian build routines are necessary to maintain the package.

More detailed what those documented commands does internally, you can see by looking at the get-vendorlibs target in debian/rules - that file is a regular make file, containing a set of special targets mandatory for all Debian packages, and optionally some targets not used during regular build but included for convenience, like here.

It is source! Debian never reuse binary code prebuilt by upstream or others. In fact, if your git repo contains binary blobs then we in Debian adds an extra routine to repackage your released code with those binary parts removed.

Thanks, but really no need: For your style of development - marking releases using git tags in a publicly accessible git repository - there is no need for you to do more.

For now I need help testing if it works at all.

Later I expect that some of those participating in community tests grab sources released by MaidSafe, some use self-compiled binaries, and some install from Debian (or from a package built by me, until package is ready for official Debian inclusion).

Thanks. I am aware of your CI tests.

For Debian packaging, we distinguish between build-time tests and runtime tests.

Build-time tests are constrained - no internet!

Runtime tests can vary - some runtime tests involve a full recompilation of the package. Some involve freezing time to check that a recompile is then bit-by-bit identical. Some might include booting up a virtual system to e.g. replace the bootloader and test if that virtual system succeeds in rebooting after that.

Thanks - it would indeed be helpful to identify subsets of tests suitable as Debian build-time tests (did the build succeed not only to compile “something” but “something that does some basics of what is expected of it”?), and as Debian runtime tests (does the binary interact as expected with the surrounding system?).

NB! I am no expert in composing testsuites. And maybe my choice of words are not the best - possibly instead of talking about “runtime tests” I should use the more common word “CI tests”.

Debian runtime tests are intended to catch if something else in Debian causes the package to no longer work.

Maybe it makes sense to also include tests to discover if something else in the World outside Debian changes as well - e.g. connect to a live external Safe network and interact with it a bit.

Debian does not have endless resources, though - so I am not sure we should include a shitload of tests in the Debian context…

[quote=“chriso, post:25, topic:5520”]

Are these automated? Or are they reports from someone who ‘manually’ ran some test procedure for a given package?

The tests listed at the right side in the middle are fully automated.

Some tests are semi-automated (e.g. one Debian developer has access to a huge compute farm and occationally rebuilds all or certain subsets of Debian in various ways), but results from those - and from manual tests - will appear in the right side at the top as bugreports. And if bugs are severe (e.g. FTBFS - “failure to build from source”) then they will also appear in the middle at the top half.

Perfectly fine :slight_smile:

8 Likes

Hi @jonas

Sorry for taking some time to get back to you. This may be more like a longer exchange of emails rather than a quick back and forth, as I wanted to sit down and take some time to carefully read and try and understand what you’ve said.

I appreciate you answering all the questions. The deeper insight into Linux and Debian here is really interesting. I do just want to be clear, I’m not trying to exploit your time. Even though I’m asking some general questions, ultimately all of this is aimed at getting my head into a position where I can understand exactly how I can help.

I think there could actually be some mutual benefits. One example would be with offline builds. At the moment Maidsafe doesn’t have any specific requirement for that, but given the nature of the project, I believe it could come in the future. Plus, it’s also something that’s just pretty interesting from a technical point of view.

Now, to address specific things.

Thanks for the correction on that!

Since you mentioned it, I thought I might try and take clap as an example to try and understand packaging in more detail. I think I’ve found the source of the package here. I understand your explanation for versioning, but now I’d please like to ask some questions to improve my general understanding:

  • What defines what this package does when we run apt-get install -y librust-clap-dev ? It looks like the debian/control file is important. I can see it obviously defines the dependent packages. I was expecting this to show me what files get put where, or something like that, but it doesn’t (at least not that I can see). In the apt-get output it runs a .deb. Is that what defines what it actually does? How do you get from this package to the .deb? Is that what the package builds to? So is there a tool that takes this package source and creates a .deb?
  • Even though I’m asking the above, I ran it on a VM, and I can see the effects of the apt-get install are to put the source at /usr/share/cargo/registry. So I get that we could now use that source by pointing cargo vendor there?
  • How does this source package rust-clap end up as librust-clap-dev? The control file has Package: librust-clap-dev. Is that what defines it?
  • This site allows you to search all these source packages, but when you build a distribution, there must be some process that’s pulling all the packages together and building them, right? Can you link to that?

Related to all this and to help me understand, before getting into how this stuff would apply to the safe_network repo, I think it would be useful if I could play around with a little sandbox. So I could write a tiny little Rust binary that has a dependency on a couple of small crates that themselves have very little dependencies. Then if I could go through the development workflow for writing a package, which would include actually running the package on a Debian system, that would really help me put all the pieces together. Is there any documentation you could link to that describes something like that?

Just to address this part of what you said, I think as I said above, I wouldn’t mind getting my head around a sandbox project first.

I want to just clarify something here. Based on other things you said in this section, when we say “tests” in this context, we’re talking about tests for the Debian package, as opposed to testing some specific behaviour in the application that the package installed?

There are tests you can run that are ‘pure’ unit tests, i.e., they don’t rely on a running network. I can help identify those.

Cheers,

Chris

5 Likes

…and even if you were, I am enjoying it. Not sure how I can put such “concerns” of yours to rest. :slight_smile:

2 Likes

Correct. Classic build logic e.g. using GNU autotools typically contains three steps - configure probes environment and decides what to build here, make compiles source into binary form, and make install installs compiled code and other assets into the system and does integrations stuff like setting access rights and register with man database.

Debian packaging wraps around such build systems and extends with tasks to scoop up the configured+built+installed stuff and packaging it into one or more arballs (yes, not a typo: Debian binary packages use an image format called ar, similar but simpler than tar and zip).

There are overwhelming piles of documentation… :slight_smile:

For understanding Debian source package structure and how that is turned into one or more Debian binary packages, this seems a good starting point.

To understand the upgrade flow of a single package - how “upstream” issues new releases from time to time, and Debian package may squeeze in intermedary packaging releases in-between those, see this neat little diagram.

For understanding Debian package management from the point of view of a single package - how installing, updating, removing a single package interacts with the surrounding operating system, try start from here.

For understanding Debian package management from a publishing point of view - how to serve source and binary packages for consumption by Debian(-derived) systems, then try start here.

To get an understanding of the daily churn of Debian development - how packages uploaded to Debian get autobuilt, tested, and published, and how they migrate from the evolutionary stages of a Debian system - from unstable through testing to stable - then that’s pretty complex but one summary view is this.

None of the above are specific to Rust packages, so some of your concrete questions are still unanswered after trawling through all of that - e.g. how exacly the system path /usr/share/cargo/registry is used. For understanding those details, you will need to dive deeper into the helper tools used by the Debian Rust team.

…and then on top of (or slightly besides) what the Debian Rust team has semi-formalized, I have invented some techniques of my own.

One thing I do differently than the Debian Rust team is how to track maintenance work: The Rust team tracks all of the 1000+ Rust crates they maintain in a single git repo, I prefer the approach far more common in Debian of separately tracking the packaging of each upstream project (where an upstream project might consist of a single source or - like the safe-network project - a “worksheet” of multiple tightly related sources).

Another thing I have invented is a streamlined management of embedding vendorlibs: The get-vendorlibs target in the debian/rules file of the Debian source packages I have composed that involve building Rust crates. Since embedding other projects is strongly discouraged in Debian, this technique is mainly intended for draft packages. Ideally when officially uploading to Debian, each and every upstream crate needed as a dependency of the packaged project is packaged as a separate Debian source package (where the built binary package installs the Rust code below /usr/share/cargo/registry), but a few crates might be too specific and/or too tiny to package on its own - as an example I chose to embed one crate in the official package for Rust project precious.

Mostly, yes - but it is a gray area, I suspect…

Most packages in Debian are more loosely linked than Rust code - scripting languages fully resolve links at runtime, and most C code is compiled as shared objects. For those, Debian testing might catch changes in other Debian components affecting this package.

Since Rust code is statically linked, such tests rarely makes sense. Instead it might make sense to fully rebuild the package in CI testing, but I am hesitant at enabling that because it is a heavy task to impose on Debian infrastructure, and I am not really familiar with the limits (most of my experience in my 20+ years of Debian package maintenance has been with runtime-linked or shared code).

That would be helpful indeed.

Probably the most helpful would be if not done once, but if sensible (and technically possible and not too much of a hassle) to generally hint throughout the project tests that are of different type - as I expect other distributors (and perhaps some advanced end-users as well) might benefit from this as well.

Ideal to me would be if your source code were coded to consult same environment variables as agreed upon in the perl community, meaning that simply running cargo test would run only a commonly sensible subset of tests, and only when EXTENDED_TESTING=1 is set in environment (or that variable is set to any non-empty value, really) are tests included that require network or other special advanced environment, or are slow. And similar for other variables (read the linked text, or the official text).

8 Likes

As an end-user of Debian for ten years, this is an interesting discussion. More to Debian packaging than I would have imagined! Impressive.

It shows! Thank you very much for helping get the Safe Network into Debian!

6 Likes

Hi @TylerAbeoJordan - thanks for pushing this conversation early on.

…and thanks for using and seeing the value in Debian :slight_smile:

If you can spare the time, then I can use the help of Debian users to test the packages that I am drafting.

5 Likes

Cool! Will you post a test-repo here when ready?

1 Like

I thought I’d point out that supporting Debian means supporting MANY other distros that are based on Debian as well.

2 Likes

Depending on what you mean, I already did that on April 17th: I posted a link in this thread and mentioned that it “lists ways you can help”.

Please scroll back and check that link, and tell me if that is what you need to help out.
Thanks.

Indeed. Debian maintains a list here that you might find helpful.

5 Likes

First of the MaidSafe packages has now been packaged for official Debian inclusion: The crate xor_name has been pushed to Debian as now sits in NEW queue awaiting a final screening and (hopefully) approval by the Debian FTP-master team.

[below was supposed to be a separate post, but this stupid forum software doesn’t allow me to structure my communication that way…]

One way you might be able to help is to check among your peer developers if anyone is caring for issues filed at the Github issue trackers for your various Rust projects.

I have filed a few bugreports but so far received no response on any of them. Nothing fatally flawed, so perhaps you are simply busy with more important stuff. But I just realized that I don’t recall receiving any response from those issues, and wondered if perhaps it is because you use some other issue tracker internally and perhaps forgot to turn off the Github issue trackers (if that is even possible) or in other ways communicate loudly enough where you prefer receiving issue reports.

10 Likes

Hi Jonas,

I’ll be responding to your larger post towards the end of the week, or maybe on the weekend, but let me give you a quick response to this one.

Regarding how I can help, all the questions I’ve been asking have been in an effort to move towards being able to make a contribution to your process and help with the Maidsafe crates we’d want to get approved for Debian. Right now, I’m doing about a half day a week off mainline development on a little project to make our licensing consistent across all our repos. What I’d like to do is propose to Josh that I could continue to devote some time every week for issues that are off mainline development, and the Debian contributions would be one of those issues. I feel though that I need to understand it better before I take the proposal to him, and that’s why I’ve been asking all the questions.

Regarding the issues that get raised on the repos, yeah, managing and responding to those is something we don’t do very well. I’ll do a little audit of all the issues across all the repos and speak to Josh about that. I can’t guarantee when it will be, but I promise I’ll get back to you.

Oh also, glad to hear that one of our crates went in.

Cheers,

Chris

@jonas I swear, half the time I ‘reply’ to a particular post on here, it doesn’t actually register the reply properly.

4 Likes

Don’t forget there’s two buttons and the big blue ‘reply’ is replies to topic. It’s the small black black arrow in the post to reply to the post.

2 Likes

Yeah, this happens when I’m replying to the post.

Which it just did there - haha!

2 Likes

Thanks for skimming throught the issues, @chriso - and for learning about Debian packaging.

Not exactly: It is queued to hopefully get in: NewQueue - Debian Wiki

3 Likes