Update README.md
This commit is contained in:
		
							parent
							
								
									22ac1403b6
								
							
						
					
					
						commit
						4af167d27b
					
				
							
								
								
									
										18
									
								
								README.md
									
									
									
									
									
								
							
							
						
						
									
										18
									
								
								README.md
									
									
									
									
									
								
							| @ -1,6 +1,6 @@ | ||||
| # Content-addressed derivations -> Dynamic Derivations Nix roadmap | ||||
| 
 | ||||
| We at Obsidian were very pleased with the reception of [Sandstone](https://github.com/obsidiansystems/sandstone) ([slides](https://github.com/obsidiansystems/sandstone/blob/slides/planet-nix-2025.pdf)) at [Planet Nix](https://planetnix.com/). | ||||
| We at Obsidian Systems were very pleased with the reception of [Sandstone](https://github.com/obsidiansystems/sandstone) ([slides](https://github.com/obsidiansystems/sandstone/blob/slides/planet-nix-2025.pdf)) at [Planet Nix](https://planetnix.com/). | ||||
| As the end of the talk says, it's time to bring these experiments to a close --- we want to use these things! | ||||
| The rest of this document is a deep dive --- almost brain dump! --- into what is left, at various scales of granularity. | ||||
| 
 | ||||
| @ -33,7 +33,7 @@ I don't want content-addressing derivations to be responsible for any of those. | ||||
| Drilling deeper, what does "ensuring the binary cache is sound" entail? | ||||
| I think the essential issue is [Nix#11896]. | ||||
| "deep realisations" --- build trace key-value pairs where the key includes derivations that depend on other derivations' outputs --- are fundamentally ambiguous. | ||||
| This ambiguous makes them hard to verify/challenge, and hard to know when they conflict --- two deep realisations may implicitly make incompatible assumptions about the outputs of those dependency derivations. | ||||
| This ambiguity makes them hard to verify/challenge, and hard to know when they conflict --- two deep realisations may implicitly make incompatible assumptions about the outputs of those dependency derivations. | ||||
| We currently have a notion of "dependent realisations" that seeks to address this issue, but I do not think this mechanism is sound, and it is certainly not consistently implemented. | ||||
| 
 | ||||
| The simplest thing to do is....just rip out deep realisations. | ||||
| @ -55,10 +55,10 @@ We can always delete *all* realisations locally | ||||
| (there are no hard references between shallow realisations -- no "closure property"), | ||||
| so that sledgehammer can always be presented as a fail-safe last resort to unbreak anyone's machine that ran out of disk space. | ||||
| Again, the current way we GC realisations (leveraging those "dependent realisations") is not necessarily a good or the only way to do things | ||||
| --- in fact, because the relationships between realisations are "soft" and not "hard", I very this as a situation where there are many possible "policies", and choosing between them is a matter of opinion. | ||||
| --- in fact, because the relationships between realisations are "soft" and not "hard", I see this as a situation where there are many possible "policies", and choosing between them is a matter of opinion. | ||||
| Multiple policy/opinion territory is a clear place to cut scope for the first version. | ||||
| 
 | ||||
| Downside two however I consider more series | ||||
| The second downside, however, I consider more serious | ||||
| --- it would be really annoying to always download GCC whenever you just want some cached binary built with GCC. | ||||
| Yes, you can GC that GCC right away, so there is no wasted disk space, but there is still the wasted time waiting for the download, and wasted network usage. | ||||
| Downloading to then delete is not a solution, but just exposes how artificial and silly the status quo is. | ||||
| @ -66,12 +66,12 @@ Downloading to then delete is not a solution, but just exposes how artificial an | ||||
| [Nix#11928] is thus something I consider required to fix if we're going to get rid of deep realisations (as I propose). | ||||
| The good thing is that we can simply change the scheduling logic so it's no longer a problem. | ||||
| The fix is conceptually simple enough: we can resolve derivations (normalize their inputs) without actually downloading those inputs. | ||||
| We just look up build trace key-value pairs and substitute within the derivation accordingly. | ||||
| The less good news is that it is a bit harder than it sounds to implement, because the scheduling code was such a confusing mess. | ||||
| We can just look up build trace key-value pairs and substitute within the derivation accordingly. | ||||
| The less good news is that it is a bit harder than it sounds to implement, because the scheduling code is such a confusing mess. | ||||
| 
 | ||||
| #### Low level | ||||
| 
 | ||||
| This in turn leans me to [Nix#12663]. | ||||
| This in turn leads me to [Nix#12663]. | ||||
| To make progress on the schedule code (and actually a bunch of other issues, which I'll hopefully get to), we need to untangle scheduling and building. | ||||
| Only then we'll we have a "clean workbench" upon which we can address reworking the scheduling logic for [Nix#11928] (and the other issues too). | ||||
| This might sound hard, but it actually isn't so bad --- it's just long overdue. | ||||
| @ -114,7 +114,7 @@ The linked issue contains a discussion of alternatives, I lean towards something | ||||
| 
 | ||||
| #### Rollout, Nixpkgs, RFC | ||||
| 
 | ||||
| Whereas the above is mostly "technical stuff I can just *do* without having to ask anyone for for permission", this part is squarely on community by-in. | ||||
| Whereas the above is mostly "technical stuff I can just *do* without having to ask anyone for for permission", this part is squarely on community buy-in. | ||||
| I think what follows is a good process to follow, but, of course, no one knows for sure how the community will react until they do. | ||||
| 
 | ||||
| This is the roadmap I have in mind; the "...." indicates perhaps more intermediate steps to gain confidence in the new way things work before a major "flip the switch" milestone. | ||||
| @ -160,7 +160,7 @@ Without "wanted" mutation, there is no `addWantedOutputs`, and (I'm pretty sure) | ||||
| > An irony of the status quo is we in fact *do* have something called `DrvOutputSubstitutionGoal`! | ||||
| > It just isn't sufficient to get rid of "wanted outputs" from `DerivationGoal`. | ||||
| 
 | ||||
| With that fixed....the core of dynamic derivations really is done! | ||||
| With that fixed, the core of dynamic derivations really is done! | ||||
| There are some usability improvements such as | ||||
| 
 | ||||
| - [Nix#8602] Restricted form of "recursive Nix", to give people better incentives / guardrails. | ||||
|  | ||||
		Loading…
	
	
			
			x
			
			
		
	
		Reference in New Issue
	
	Block a user