Compare commits
2 Commits
22ac1403b6
...
adbfbd2366
Author | SHA1 | Date | |
---|---|---|---|
adbfbd2366 | |||
4af167d27b |
18
README.md
18
README.md
@ -1,6 +1,6 @@
|
|||||||
# Content-addressed derivations -> Dynamic Derivations Nix roadmap
|
# Content-addressed derivations -> Dynamic Derivations Nix roadmap
|
||||||
|
|
||||||
We at Obsidian were very pleased with the reception of [Sandstone](https://github.com/obsidiansystems/sandstone) ([slides](https://github.com/obsidiansystems/sandstone/blob/slides/planet-nix-2025.pdf)) at [Planet Nix](https://planetnix.com/).
|
We at Obsidian Systems were very pleased with the reception of [Sandstone](https://github.com/obsidiansystems/sandstone) ([slides](https://github.com/obsidiansystems/sandstone/blob/slides/planet-nix-2025.pdf)) at [Planet Nix](https://planetnix.com/).
|
||||||
As the end of the talk says, it's time to bring these experiments to a close --- we want to use these things!
|
As the end of the talk says, it's time to bring these experiments to a close --- we want to use these things!
|
||||||
The rest of this document is a deep dive --- almost brain dump! --- into what is left, at various scales of granularity.
|
The rest of this document is a deep dive --- almost brain dump! --- into what is left, at various scales of granularity.
|
||||||
|
|
||||||
@ -33,7 +33,7 @@ I don't want content-addressing derivations to be responsible for any of those.
|
|||||||
Drilling deeper, what does "ensuring the binary cache is sound" entail?
|
Drilling deeper, what does "ensuring the binary cache is sound" entail?
|
||||||
I think the essential issue is [Nix#11896].
|
I think the essential issue is [Nix#11896].
|
||||||
"deep realisations" --- build trace key-value pairs where the key includes derivations that depend on other derivations' outputs --- are fundamentally ambiguous.
|
"deep realisations" --- build trace key-value pairs where the key includes derivations that depend on other derivations' outputs --- are fundamentally ambiguous.
|
||||||
This ambiguous makes them hard to verify/challenge, and hard to know when they conflict --- two deep realisations may implicitly make incompatible assumptions about the outputs of those dependency derivations.
|
This ambiguity makes them hard to verify/challenge, and hard to know when they conflict --- two deep realisations may implicitly make incompatible assumptions about the outputs of those dependency derivations.
|
||||||
We currently have a notion of "dependent realisations" that seeks to address this issue, but I do not think this mechanism is sound, and it is certainly not consistently implemented.
|
We currently have a notion of "dependent realisations" that seeks to address this issue, but I do not think this mechanism is sound, and it is certainly not consistently implemented.
|
||||||
|
|
||||||
The simplest thing to do is....just rip out deep realisations.
|
The simplest thing to do is....just rip out deep realisations.
|
||||||
@ -55,10 +55,10 @@ We can always delete *all* realisations locally
|
|||||||
(there are no hard references between shallow realisations -- no "closure property"),
|
(there are no hard references between shallow realisations -- no "closure property"),
|
||||||
so that sledgehammer can always be presented as a fail-safe last resort to unbreak anyone's machine that ran out of disk space.
|
so that sledgehammer can always be presented as a fail-safe last resort to unbreak anyone's machine that ran out of disk space.
|
||||||
Again, the current way we GC realisations (leveraging those "dependent realisations") is not necessarily a good or the only way to do things
|
Again, the current way we GC realisations (leveraging those "dependent realisations") is not necessarily a good or the only way to do things
|
||||||
--- in fact, because the relationships between realisations are "soft" and not "hard", I very this as a situation where there are many possible "policies", and choosing between them is a matter of opinion.
|
--- in fact, because the relationships between realisations are "soft" and not "hard", I see this as a situation where there are many possible "policies", and choosing between them is a matter of opinion.
|
||||||
Multiple policy/opinion territory is a clear place to cut scope for the first version.
|
Multiple policy/opinion territory is a clear place to cut scope for the first version.
|
||||||
|
|
||||||
Downside two however I consider more series
|
The second downside, however, I consider more serious
|
||||||
--- it would be really annoying to always download GCC whenever you just want some cached binary built with GCC.
|
--- it would be really annoying to always download GCC whenever you just want some cached binary built with GCC.
|
||||||
Yes, you can GC that GCC right away, so there is no wasted disk space, but there is still the wasted time waiting for the download, and wasted network usage.
|
Yes, you can GC that GCC right away, so there is no wasted disk space, but there is still the wasted time waiting for the download, and wasted network usage.
|
||||||
Downloading to then delete is not a solution, but just exposes how artificial and silly the status quo is.
|
Downloading to then delete is not a solution, but just exposes how artificial and silly the status quo is.
|
||||||
@ -66,12 +66,12 @@ Downloading to then delete is not a solution, but just exposes how artificial an
|
|||||||
[Nix#11928] is thus something I consider required to fix if we're going to get rid of deep realisations (as I propose).
|
[Nix#11928] is thus something I consider required to fix if we're going to get rid of deep realisations (as I propose).
|
||||||
The good thing is that we can simply change the scheduling logic so it's no longer a problem.
|
The good thing is that we can simply change the scheduling logic so it's no longer a problem.
|
||||||
The fix is conceptually simple enough: we can resolve derivations (normalize their inputs) without actually downloading those inputs.
|
The fix is conceptually simple enough: we can resolve derivations (normalize their inputs) without actually downloading those inputs.
|
||||||
We just look up build trace key-value pairs and substitute within the derivation accordingly.
|
We can just look up build trace key-value pairs and substitute within the derivation accordingly.
|
||||||
The less good news is that it is a bit harder than it sounds to implement, because the scheduling code was such a confusing mess.
|
The less good news is that it is a bit harder than it sounds to implement, because the scheduling code is such a confusing mess.
|
||||||
|
|
||||||
#### Low level
|
#### Low level
|
||||||
|
|
||||||
This in turn leans me to [Nix#12663].
|
This in turn leads me to [Nix#12663].
|
||||||
To make progress on the schedule code (and actually a bunch of other issues, which I'll hopefully get to), we need to untangle scheduling and building.
|
To make progress on the schedule code (and actually a bunch of other issues, which I'll hopefully get to), we need to untangle scheduling and building.
|
||||||
Only then we'll we have a "clean workbench" upon which we can address reworking the scheduling logic for [Nix#11928] (and the other issues too).
|
Only then we'll we have a "clean workbench" upon which we can address reworking the scheduling logic for [Nix#11928] (and the other issues too).
|
||||||
This might sound hard, but it actually isn't so bad --- it's just long overdue.
|
This might sound hard, but it actually isn't so bad --- it's just long overdue.
|
||||||
@ -114,7 +114,7 @@ The linked issue contains a discussion of alternatives, I lean towards something
|
|||||||
|
|
||||||
#### Rollout, Nixpkgs, RFC
|
#### Rollout, Nixpkgs, RFC
|
||||||
|
|
||||||
Whereas the above is mostly "technical stuff I can just *do* without having to ask anyone for for permission", this part is squarely on community by-in.
|
Whereas the above is mostly "technical stuff I can just *do* without having to ask anyone for for permission", this part is squarely on community buy-in.
|
||||||
I think what follows is a good process to follow, but, of course, no one knows for sure how the community will react until they do.
|
I think what follows is a good process to follow, but, of course, no one knows for sure how the community will react until they do.
|
||||||
|
|
||||||
This is the roadmap I have in mind; the "...." indicates perhaps more intermediate steps to gain confidence in the new way things work before a major "flip the switch" milestone.
|
This is the roadmap I have in mind; the "...." indicates perhaps more intermediate steps to gain confidence in the new way things work before a major "flip the switch" milestone.
|
||||||
@ -160,7 +160,7 @@ Without "wanted" mutation, there is no `addWantedOutputs`, and (I'm pretty sure)
|
|||||||
> An irony of the status quo is we in fact *do* have something called `DrvOutputSubstitutionGoal`!
|
> An irony of the status quo is we in fact *do* have something called `DrvOutputSubstitutionGoal`!
|
||||||
> It just isn't sufficient to get rid of "wanted outputs" from `DerivationGoal`.
|
> It just isn't sufficient to get rid of "wanted outputs" from `DerivationGoal`.
|
||||||
|
|
||||||
With that fixed....the core of dynamic derivations really is done!
|
With that fixed, the core of dynamic derivations really is done!
|
||||||
There are some usability improvements such as
|
There are some usability improvements such as
|
||||||
|
|
||||||
- [Nix#8602] Restricted form of "recursive Nix", to give people better incentives / guardrails.
|
- [Nix#8602] Restricted form of "recursive Nix", to give people better incentives / guardrails.
|
||||||
|
Loading…
x
Reference in New Issue
Block a user