Deterministic builds

When I build safe_vault it creates a different binary file to when maidsafe builds safe_vault.

This is not really ideal; the functionality is identical but the file itself is not.

So this topic is about exploring the path toward deterministic builds for at least the vault but hopefully the other SAFE software also.

Bitcoin does deterministic builds with gitian (for windows and mac) and is looking to move to guix (which is currently used for the linux build). You can see from the massive commit Add deterministic Guix builds this is not a trivial thing to do.

My initial investigations have given rise to some basic questions:

  • Why is the maidsafe linux build of safe_vault done with x86_64-unknown-linux-musl
    rather than x86_64-unknown-linux-gnu (see the *-musl suffix on the release page)? For me the default rust toolchain installed by rustup is gnu, and the travisCI is also gnu (see L435 of the travis build log for vault 0.20.1), so why is maidsafe using musl for their build? I had an error with the openssl package when trying to build safe_vault with the musl toolchain. Which toolchain would be preferred if we try moving toward deterministic builds?

  • I looked at the vault binaries (using xxd safe_vault) and searched for the text maidsafe and home and didn’t see anything that immediately stood out as being specific to the build environment. An introduction to deterministic builds seems like a good starting place to get an idea of how complex it is to manage sources of variation (although not specifically about rust builds).

  • What is the value of deterministic builds, are they important, and should they be attempted or worked towards? Can / should they be used for all maidsafe products (eg browser, frontend etc) or only the vault? Can we do fine without them?

This topic doesn’t seem to have been discussed too much on the forum from what I can see… @bluebird discussed it a couple of times (here is one such time) and @sfultong uses NixOS which is known for deterministic builds. Anyone else got experience or opinions about this topic?

And for those that just want a good read, try Reflections on Trusting Trust.

21 Likes

This allows the binary to run on all flavours of linux. So the 64 bit will run on all 64 bit machines. If we don’t then we need to provide a load of builds for differing versions of glibc.

It is also a step towards deterministic builds. On release we should be using a fixed cargo.lock to aid this as well. We can go further though and vendor such apps.

21 Likes

If people are interested, I could try setting up a nix build script for SAFE binaries.

11 Likes

Musl target is also more secure. A quote from yourself one year ago:

8 Likes

I agree that Deterministic Builds is a good goal.

I’m not sure if this is fully achievable with rust toolchain yet. Eg, see https://github.com/rust-lang/rust/issues/34902

Related, there is Codechain for signing.

In code we trust: Secure multiparty code reviews with signatures and hash chains.

The most common signing mechanism for open-source software is using GPG signatures. For example, GPG is used to sign Git commits and Debian packages. There is no built-in mechanism for key rotation and key compromise. And if forced to, a single developer can subvert all machines which trust the corresponding GPG key.

That’s where the Codechain tool comes in. It establishes code trust via multi-party reviews recorded in unmodifiable hash chains.

Codechain allows to only publish code that has been reviewed by a preconfigured set of reviewers. The signing keys can be rotated and the reviewer set flexibly changed.

Every published code state is uniquely identified by a deterministic source tree hash stored in the hash chain, signed by a single responsible developer.

Codechain uses files to store the hash chain, not a distributed “blockchain”.

4 Likes

Is there a way to also specify / force the version of rust to use? My searching didn’t give any clues.

@sfultong, do you have any opinions regarding nix vs guix?


Some more notes on my dabbling toward maybe some progress:

One of the tough things I faced in generating a musl version of vault was the dependency on openssl.
cargo build --target x86_64-unknown-linux-musl

Looking through the build script for vault (see fleming/Dockerfile.build) there’s a fair bit of screwing around to build openssl with musl (L25-L40). This particular part is failing when I try to build vault with musl on my machine (ie not using a docker container).

OpenSSL is not actually needed directly by the vault. The dependency tree (reduced just to show openssl dependency) stems from self_update

├── self_update v0.5.1
│   ├── reqwest v0.9.20
│   │   ├── hyper-tls v0.3.2
│   │   │   ├── native-tls v0.2.3
│   │   │   │   ├── openssl v0.10.24
│   │   │   │   │   └── openssl-sys v0.9.49
│   │   │   │   ├── openssl-probe v0.1.2
│   │   │   │   └── openssl-sys v0.9.49 (*)

when I removed self_update feature the vault compiled without needing to point to the specific musl version of openssl.

The DOPENSSL_NO_SECURE_MEMORY flag used in the build of openssl also comes with a reasonable warning (see this github issue comment): “Obviously it’s not recommended to compile things this way (its clearly better to have the secure memory feature).”

It seems like possibly a good first step to remove the openssl dependency, what do you think? I know self_update is cool but is it worth dragging in the complexity of openssl for it? Just poking around for opinions, not trying to step on any toes here so apologies if I’ve misunderstood some aspect!

5 Likes

We have done a ton of work to remove that actually. Rust_sodium needed it and we deprecated that for exactly that reason/ My feeling is we build 100% in rust and then lock it down as well as audit it.

8 Likes

Removing self_update has another benefit, the vault binary goes from 15.6 MB to 11.7 MB, ie that feature adds about 4 MB.

4 Likes

@sfultong, do you have any opinions regarding nix vs guix?

Not strong ones. I’m more familiar with nix, but nix is a weirder language. Guix is scheme-based, so it’s more friendly to people who know scheme. I think nix has more adoption, probably mainly because it came first.

1 Like

I managed to reproduce the same build on two different machines.

  • Remove openssl dependency from safe_vault (see this commit) only because I couldn’t get it to build with musl. Maybe it’s possible to do deterministic builds still with openssl, but I chose to remove it.
  • musl-gcc 9.2.1 20191008
  • rustc 1.39.0 4560ea788 2019-11-04
  • safe_vault 0.20.1 8dee8601 2019-12-04
  • cargo build --release --target x86_64-unknown-linux-musl

This builds on two different machines to both give the same sha256:

$ sha256sum target/x86_64-unknown-linux-musl/release/safe_vault
eaeed686b7183314f26c85f2461963e95041dd6e483906d7371be1a4e4b1245a

This doesn’t necessarily mean it’s deterministic, just that it’s reproducable!

So it seems like a possible step toward the goal of deterministic builds.

On the topic of openssl, it would be a pretty big headline (for the geeks anyhow) for the new internet to work without any openssl dependency. That particular library is so critical to the existing internet but often seen as a bit of a weakness. I mean, just look at the popular topics on hacker news about openssl. It’s not pretty!

I have some thoughts on self_update too but maybe getting too far off topic.

9 Likes