Content-addressed derivations -> Dynamic Derivations Nix roadmap

We at Obsidian Systems were very pleased with the reception of Sandstone (slides) at Planet Nix. As the end of the talk says, it's time to bring these experiments to a close --- we want to use these things! The rest of this document is a deep dive --- almost brain dump! --- into what is left, at various scales of granularity.

Roadmap

Content-Addressing derivations

High level

Wrapping up the core of this long-experimental feature is the first step. (One of the reasons for that is that using dynamic derivations requires content-addressing derivations, because derivations themselves are always content-addressed.)

Completely moving the whole ecosystem over to content-addressing derivations is the ultimate goal, but this doesn't need to coincide with wrapping up the core of the experiment. For example, as others have written out, "sed-ing" binaries to rewrite self-references is unlikely to work in general. That's fine for me --- we'll simply keep input-addressing in the cases where it doesn't work. (Not only is this expedient, this also incentivizes trying to modify packages to stop needing self-references, which I think is a good thing to do regardless.)

So what does "wrapping up the core of the experiment" entail? For the big test is "don't put junk in the cache". I am OK with the "client side" missing various conveniences, like tooling to understand trust map conflicts, or fancier garbage collection. So long as there is an still input-addressed Nixpkgs, no one will be "forced" to use them (by network effects) and so client UX issues can just be dodged by "just opting out". On the "server side", however, I don't want anything sketchy to be going on, because I don't want people to accidentally opt into issues, especially highly nuanced "cache semantics" issues, that they didn't sign up for. Cached build artifacts, even local ones but especially shared internet-accessible ones, are potentially very long-lived. If we get the roll-out wrong, we open ourselves up to "cache poisoning" issues, which because of the distributed nature of Nix stores and copying, may be hard to completely eradicate. I don't want content-addressing derivations to be responsible for any of those.

Medium level

Drilling deeper, what does "ensuring the binary cache is sound" entail? I think the essential issue is Nix#11896. "deep realisations" --- build trace key-value pairs where the key includes derivations that depend on other derivations' outputs --- are fundamentally ambiguous. This ambiguity makes them hard to verify/challenge, and hard to know when they conflict --- two deep realisations may implicitly make incompatible assumptions about the outputs of those dependency derivations. We currently have a notion of "dependent realisations" that seeks to address this issue, but I do not think this mechanism is sound, and it is certainly not consistently implemented.

The simplest thing to do is....just rip out deep realisations. Build trace keys should always be derivations that just depend on "opaque" store objects. When those store objects are content-addressed, this completely solves the problem.

When those store objects are input-addressed, we can also retroactively fix this with additional content-addresses (Nix#11919) (think "the data with hash foo is mounted at store path bar), but frankly I am not sure whether this should be a "blocker" --- mixed content- and input- addressing was always supposed to be a transitional phase, and the ambiguities are no worse than what input-addressing everything does, so it may not make sense to "invest in the outgoing" thing. We'll see.

There are two downsides to "just do shallow addressing only" which are

1. We break the current client-side logic for garbage-collecting realisations

2. Nix#11928 We regress with the current scheduling logic, causing build build-time inputs to be built/downloaded unnecessarily when the downstream thing we actually need should just be substitute exists but was built slightly differently.

Re (1): once again, I am quite willing to defer polishing something that is client-side, and thus has problems that the user is free to side-step entirely by opting out. We can always delete all realisations locally (there are no hard references between shallow realisations -- no "closure property"), so that sledgehammer can always be presented as a fail-safe last resort to unbreak anyone's machine that ran out of disk space. Again, the current way we GC realisations (leveraging those "dependent realisations") is not necessarily a good or the only way to do things --- in fact, because the relationships between realisations are "soft" and not "hard", I see this as a situation where there are many possible "policies", and choosing between them is a matter of opinion. Multiple policy/opinion territory is a clear place to cut scope for the first version.

The second downside, however, I consider more serious --- it would be really annoying to always download GCC whenever you just want some cached binary built with GCC. Yes, you can GC that GCC right away, so there is no wasted disk space, but there is still the wasted time waiting for the download, and wasted network usage. Downloading to then delete is not a solution, but just exposes how artificial and silly the status quo is.

Nix#11928 is thus something I consider required to fix if we're going to get rid of deep realisations (as I propose). The good thing is that we can simply change the scheduling logic so it's no longer a problem. The fix is conceptually simple enough: we can resolve derivations (normalize their inputs) without actually downloading those inputs. We can just look up build trace key-value pairs and substitute within the derivation accordingly. The less good news is that it is a bit harder than it sounds to implement, because the scheduling code is such a confusing mess.

Low level

This in turn leads me to Nix#12663. To make progress on the schedule code (and actually a bunch of other issues, which I'll hopefully get to), we need to untangle scheduling and building. Only then we'll we have a "clean workbench" upon which we can address reworking the scheduling logic for Nix#11928 (and the other issues too). This might sound hard, but it actually isn't so bad --- it's just long overdue. (Not doing this and attempting to fix the issues anyways is much harder.)

After Planet Nix, @L-as and I started on a "bottom up" approach to this, which is the one outlined in Nix#12663.

You should now just read that issue, it attempts to lay out a roadmap also --- if I said more here I would be just inlining the ticket.

So far, we got Nix#12630, Nix#12662, and Nix#12658 done, and Nix#12668 "on deck". This will get local building pretty well "off to the side". Then we do something similar for remote building (maybe just moving the hook code, or maybe indulging a little scope creep and getting rid of it altogether per Nix#5025). At that point, the building logic (local and remote cases) will be completely "out of the way", and we should be able to solve Nix#11928. And at that point, we can (with some stop-gap for local GC) fix Nix#11896, just ripping out shallow derivations.

Along with / right after doing Nix#11896, we can also do Nix#11897. This is a good simple cleanup --- the scheduling changes and lack of deep realisations mean that there is absolutely use hash derivations "modulo fixed-output derivations", because resolved derivations never depend on fixed-output derivations (because they never depend on any derivation's output at all). We can go back to just using derivation paths.

Hydra

With the Nix changes done, the next task is getting Hydra to work with the revamped system. This is especially important given my "server first" approach --- I want to see us building at scale to find and eradicate problems before I worry about regular users actually using this stuff. This should be a very simple fix --- Hydra already computes deep and shallow realisations and uploads both. It just needs to stop doing the former.

One interesting thing to note is we should also upload the resolved derivations that the shallow realisation refers to --- otherwise we're uploading something with a dangling reference, which would be bad! We've never uploaded .drv files to binary caches from Hydra before, but I think this will be fine. We want a record of what we've done.

Documentation

Equally important to making these implementation changes is properly documenting everything. The store language for many years was not at all documented in the manual (except for bits and pieces scattered in other sections). As I celebrated in my Planet Nix talk, we've finally been addressing this, and the new "Nix Store" chapter of the manual is growing fast. I want to keep paying off "documentation debt" during all this, and make sure the new way content-addressing derivations work is completely documented. Ideally a stabilizing RFC need not contain any technical detailed design --- it can just link a version of the Nix manual where "experimental" warnings have been removed, because all the normative spec material has already been written down.

A small project that's part this is getting rid of the term "realisation" that I've been using above Nix#11895. I don't like it at all, it is a vague English word (to me), and too close to "realise" which we already use for building input- and content-addressed derivations alike. The linked issue contains a discussion of alternatives, I lean towards something with "build trace" in it.

Rollout, Nixpkgs, RFC

Whereas the above is mostly "technical stuff I can just do without having to ask anyone for for permission", this part is squarely on community buy-in. I think what follows is a good process to follow, but, of course, no one knows for sure how the community will react until they do.

This is the roadmap I have in mind; the "...." indicates perhaps more intermediate steps to gain confidence in the new way things work before a major "flip the switch" milestone.

1. Implement and document, per the above
2. Do a lot of builds of Nixpkgs, publicly, with a public cache. (This should involve Hydra, it may or may not involved hydra.nixos.org and even cache.nixos.org.) (It's not crazy to imagine it does, because content-addressing derivations should invisible to anyone not using them.)
3. Gather feedback, fix issues.
4. RFC ....
5. Nixpkgs is content-addressed by default. Some packages remain input addressed, to e.g. work around self-reference issues. ....
6. Nixpkgs doesn't use any input-addressing.

Dynamic derivations

Shortly after dynamic derivations was initially merged ~2 years ago, we had to revert part of the implementation because a race condition regression in substituting multiple outputs. I reverted the revert (Nix#12591), thinking I had finally fixed the bug, for the version of Nix I used in my Sandstone talk. Unfortunately we found that that the bug reappeared (prior to a release, this time, though! Thanks @roberth) I could barely reproduce the bug --- yes with specific exact drv paths, no with a test in the test suite (it relies on std::map ordering). I made a janky patch (Nix#12558) which did fix my janky non-committable repro. @roberth quickly found that fixed the one problem just to introduce another, and so we didn't merge.

Well, it turns out janky fixes don't like me as much as I don't like them. The proper fix here --- once again --- is to clean up the scheduling code more broadly.

The race condition stems from Derivation::addWantedOutputs. This function exists because DerivationGoal currently awkwardly handles both downloading derivation outputs --- for which we hardly care whether store objects come from the derivation or not --- and building derivation outputs --- for which we will produce all outputs like it or not. addWantedOutputs adds an additional "wanted output" to derivation goal if we still might substitute individual outputs, so we download that output too.

This is an unclear way way to solve a problem which shouldn't exist. If there was a proper separation of goals that build and goals that substitute, we wouldn't need to mutate "wanted" anythings: we could have a single-output substitution goal type, and an all-outputs build goal type; the former would create the latter after all attempts to substitute fail. There is no need to mutate the former, because multiple derivation output substitution goals for the same derivation (but different outputs) would all end up blocking on the same building goal anyways. Without "wanted" mutation, there is no addWantedOutputs, and (I'm pretty sure) no race condition.

An irony of the status quo is we in fact do have something called DrvOutputSubstitutionGoal! It just isn't sufficient to get rid of "wanted outputs" from DerivationGoal.

With that fixed, the core of dynamic derivations really is done! There are some usability improvements such as

• Nix#8602 Restricted form of "recursive Nix", to give people better incentives / guardrails. (Imperatively requesting builds from inside a build is a sledgehammer, and probably not one we'd ever want to allow in Nixpkgs, for example.

• Nix#12727 @roberth's idea (which we have yet to talk about) of derivations that return deriving paths rather than store paths

But I don't think either of these will be too much work. Dynamic derivations is a relatively "cheap" extension to content-addressing derivations.