Some ghc-options should not be considered part of build hash

[ezyang]

This is related to #3883 but it is a distinct problem so I wanted to make a ticket for it by itself.

Some ghc-options don't have any affect on build outputs, e.g., -v, but if someone specifies it via ghc-options, e.g., in their cabal.project file, it will cause us to rebuild all deps because elabProgramArgs has changed, which changes the hash, which means we need to rebuild. This is not good.

It's unclear how to go about fixing this. We could build in knowledge to cabal-install what arguments should and should not be hashed. But we'd be constantly chasing the most up-to-date set, and there would always be some time where we wouldn't have gotten it exactly right. Alternately, we could let the user specify, "use this option when building, but don't include it in the hash." But here it seems easy for a user to shoot themselves in the foot, ESP when it's a package being installed to the global store; worst case scenario you'd have to blow away the entire store because you corrupted it.

CC @bgamari who reported this most recently.

[DanielG]

How about adding a command to GHC that tells cabal which flags should influence the hash? For old versions this knowledge would still have to be included but at least then we don't need to chase after GHC.

[ezyang]

That's an interesting idea. The relevant code in GHC is fingerprintDynFlags in FlagChecker, we'd have to reverse engineer the flags that affect fields in DynFlags here.

It's worth noting GHC's recompilation avoidance doesn't always get this right. See #3891. So it' still challenging to compute the flags, even on GHC's side ;)

[23Skidoo]

Yep, just bumped into this when I tried to enable -Werror for my project.

[mightybyte]

I also hit this when I did cabal new-build --ghc-option=-fno-code

[hvr]

I think that -fno-code is a bit special here, as it does affect the .o output... by not producing any! This could have the ability to confuse cabal quite a lot. Maybe this rather demands a special "dry" mode to invoke, i.e. cabal new-build --no-code which takes care of passing this to ghc and not get confused about the .o files not being generated. What's your use-case for -fno-code?

[mightybyte]

No-code is great for getting something to compile. It's much faster than anything else I know of, and significantly speeds up the compile-fix-errors loop. The ideal situation in my mind would be to have a cabal tc (for type check) command that does this. Making it a separate command makes it more convenient and discoverable (as well as being a little quicker to type).

[merijn]

I think this issue is a bit stuck in idealism over pragmatism. As far as I can tell everyone agrees that:

1. cabal needs a way to filter flags that don't affect build outputs from the hash
2. hard-coding isn't great because that means manually keeping the flags list in cabal in sync with GHC.

So now this ticket is stuck on the "devise scheme to figure out which flags can be safely used". I (of course) support that idea, but would like to argue that we shouldn't let "perfect" be the enemy of "good". Since we need a code-path to filter out flags anyway, why add said code and, for now, hard-code a set of common flags that can be safely filtered out. Then once we figure out a more sustainable/future-proof way of dealing with GHC options that don't affect output we just have to drop the hard-coded set and replace it with whatever method people come up with.

This would make new-build a significant bit more pleasant to use right now while introducing no obstacles for a more perfect future solution. If someone can point to where the hashing code is I'm even willing to see if I can hack this in somehow.

[alexbiehl]

@merijn PackageHashConfigInputs is the input to the package hash function for package and program configuration. The ghc-options property is put into the pkgHashProgramArgs currently.

Construction of PackageHashConfigInputs happens here:

cabal/cabal-install/Distribution/Client/ProjectPlanning.hs

Lines 3496 to 3529 in 066b2a1

	packageHashConfigInputs :: ElaboratedSharedConfig
	-> ElaboratedConfiguredPackage
	-> PackageHashConfigInputs
	packageHashConfigInputs
	ElaboratedSharedConfig{..}
	ElaboratedConfiguredPackage{..} =
	PackageHashConfigInputs {
	pkgHashCompilerId = compilerId pkgConfigCompiler,
	pkgHashPlatform = pkgConfigPlatform,
	pkgHashFlagAssignment = elabFlagAssignment,
	pkgHashConfigureScriptArgs = elabConfigureScriptArgs,
	pkgHashVanillaLib = elabVanillaLib,
	pkgHashSharedLib = elabSharedLib,
	pkgHashDynExe = elabDynExe,
	pkgHashGHCiLib = elabGHCiLib,
	pkgHashProfLib = elabProfLib,
	pkgHashProfExe = elabProfExe,
	pkgHashProfLibDetail = elabProfLibDetail,
	pkgHashProfExeDetail = elabProfExeDetail,
	pkgHashCoverage = elabCoverage,
	pkgHashOptimization = elabOptimization,
	pkgHashSplitSections = elabSplitSections,
	pkgHashSplitObjs = elabSplitObjs,
	pkgHashStripLibs = elabStripLibs,
	pkgHashStripExes = elabStripExes,
	pkgHashDebugInfo = elabDebugInfo,
	pkgHashProgramArgs = elabProgramArgs,
	pkgHashExtraLibDirs = elabExtraLibDirs,
	pkgHashExtraFrameworkDirs = elabExtraFrameworkDirs,
	pkgHashExtraIncludeDirs = elabExtraIncludeDirs,
	pkgHashProgPrefix = elabProgPrefix,
	pkgHashProgSuffix = elabProgSuffix
	}

As a first approximation you could strip irrelevant options with something like this: Map.alter strip_irrelevant_ghc_options "ghc" pkgHashProgramArgs

[hvr]

hard-coding isn't great because that means manually keeping the flags list in cabal in sync with GHC.

to be fair, we do already hardcode knowledge about GHC (and other compilers) in lib:Cabal (and that's where we have to encode that information); so there is a precedent to embed knowledge about GHC. It's just annoying that GHC's CLI grammar is non-trivial, so we can't just blindly filter out tokens from the list of CLI args; we have to do a bit more work.

[merijn]

ok, looking at the code the hashing appears to initiate in one single point: buildAndInstallUnpackedPackage, which is also the first function in the chain to be in IO. So my idea is to propagate a set of filters from there into the hashing functions, which means that any future replacement process can just hook into the IO in buildAndInstallUnpackedPackage to compute a filter dynamically to replace any initial static filter.