From 4af167d27bc7532ab169919c723715a75d75ae09 Mon Sep 17 00:00:00 2001 From: Ali Abrar Date: Thu, 17 Apr 2025 20:48:34 +0000 Subject: [PATCH] Update README.md --- README.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index a8000bd..02c3971 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Content-addressed derivations -> Dynamic Derivations Nix roadmap -We at Obsidian were very pleased with the reception of [Sandstone](https://github.com/obsidiansystems/sandstone) ([slides](https://github.com/obsidiansystems/sandstone/blob/slides/planet-nix-2025.pdf)) at [Planet Nix](https://planetnix.com/). +We at Obsidian Systems were very pleased with the reception of [Sandstone](https://github.com/obsidiansystems/sandstone) ([slides](https://github.com/obsidiansystems/sandstone/blob/slides/planet-nix-2025.pdf)) at [Planet Nix](https://planetnix.com/). As the end of the talk says, it's time to bring these experiments to a close --- we want to use these things! The rest of this document is a deep dive --- almost brain dump! --- into what is left, at various scales of granularity. @@ -33,7 +33,7 @@ I don't want content-addressing derivations to be responsible for any of those. Drilling deeper, what does "ensuring the binary cache is sound" entail? I think the essential issue is [Nix#11896]. "deep realisations" --- build trace key-value pairs where the key includes derivations that depend on other derivations' outputs --- are fundamentally ambiguous. -This ambiguous makes them hard to verify/challenge, and hard to know when they conflict --- two deep realisations may implicitly make incompatible assumptions about the outputs of those dependency derivations. +This ambiguity makes them hard to verify/challenge, and hard to know when they conflict --- two deep realisations may implicitly make incompatible assumptions about the outputs of those dependency derivations. We currently have a notion of "dependent realisations" that seeks to address this issue, but I do not think this mechanism is sound, and it is certainly not consistently implemented. The simplest thing to do is....just rip out deep realisations. @@ -55,10 +55,10 @@ We can always delete *all* realisations locally (there are no hard references between shallow realisations -- no "closure property"), so that sledgehammer can always be presented as a fail-safe last resort to unbreak anyone's machine that ran out of disk space. Again, the current way we GC realisations (leveraging those "dependent realisations") is not necessarily a good or the only way to do things ---- in fact, because the relationships between realisations are "soft" and not "hard", I very this as a situation where there are many possible "policies", and choosing between them is a matter of opinion. +--- in fact, because the relationships between realisations are "soft" and not "hard", I see this as a situation where there are many possible "policies", and choosing between them is a matter of opinion. Multiple policy/opinion territory is a clear place to cut scope for the first version. -Downside two however I consider more series +The second downside, however, I consider more serious --- it would be really annoying to always download GCC whenever you just want some cached binary built with GCC. Yes, you can GC that GCC right away, so there is no wasted disk space, but there is still the wasted time waiting for the download, and wasted network usage. Downloading to then delete is not a solution, but just exposes how artificial and silly the status quo is. @@ -66,12 +66,12 @@ Downloading to then delete is not a solution, but just exposes how artificial an [Nix#11928] is thus something I consider required to fix if we're going to get rid of deep realisations (as I propose). The good thing is that we can simply change the scheduling logic so it's no longer a problem. The fix is conceptually simple enough: we can resolve derivations (normalize their inputs) without actually downloading those inputs. -We just look up build trace key-value pairs and substitute within the derivation accordingly. -The less good news is that it is a bit harder than it sounds to implement, because the scheduling code was such a confusing mess. +We can just look up build trace key-value pairs and substitute within the derivation accordingly. +The less good news is that it is a bit harder than it sounds to implement, because the scheduling code is such a confusing mess. #### Low level -This in turn leans me to [Nix#12663]. +This in turn leads me to [Nix#12663]. To make progress on the schedule code (and actually a bunch of other issues, which I'll hopefully get to), we need to untangle scheduling and building. Only then we'll we have a "clean workbench" upon which we can address reworking the scheduling logic for [Nix#11928] (and the other issues too). This might sound hard, but it actually isn't so bad --- it's just long overdue. @@ -114,7 +114,7 @@ The linked issue contains a discussion of alternatives, I lean towards something #### Rollout, Nixpkgs, RFC -Whereas the above is mostly "technical stuff I can just *do* without having to ask anyone for for permission", this part is squarely on community by-in. +Whereas the above is mostly "technical stuff I can just *do* without having to ask anyone for for permission", this part is squarely on community buy-in. I think what follows is a good process to follow, but, of course, no one knows for sure how the community will react until they do. This is the roadmap I have in mind; the "...." indicates perhaps more intermediate steps to gain confidence in the new way things work before a major "flip the switch" milestone. @@ -160,7 +160,7 @@ Without "wanted" mutation, there is no `addWantedOutputs`, and (I'm pretty sure) > An irony of the status quo is we in fact *do* have something called `DrvOutputSubstitutionGoal`! > It just isn't sufficient to get rid of "wanted outputs" from `DerivationGoal`. -With that fixed....the core of dynamic derivations really is done! +With that fixed, the core of dynamic derivations really is done! There are some usability improvements such as - [Nix#8602] Restricted form of "recursive Nix", to give people better incentives / guardrails. -- 2.49.0