runtime: VirtualFree of 0 bytes failed with errno=487 (ERROR_INVALID_ADDRESS)

[heschi]

Funny looking trybot failure on 32-bit Windows: https://storage.googleapis.com/go-build-log/0c8f6cc0/windows-386-2008_c0fd3e00.log

cc @mknyszek maybe?

[mknyszek]

The 0 bytes thing is a red herring, the code tries to decommit successively smaller chunks because of how windows mapping works.

It's almost certainly related to my code somehow, and maybe related to #35445.

[mknyszek]

On second thought... probably not related to #35445. The page size on 386 is 4 KiB.

[heschi]

@stamblerre @matloob @ianthehat

[mknyszek]

@heschik have you been seeing this often? Do you have more examples? I've been unable to reproduce thus far, and also I don't see it happening at all in CI which leads me to believe it's fairly rare.

I have one suspicion: the background scavenger is scavenging unreserved memory. The evidence here is that:

• The failure is ERROR_INVALID_ADDRESS (though MSDN won't tell me under what conditions different error codes are set).
• The page to be scavenged is at the very top of a palloc chunk, which would be the first thing the scavenger sees.

There is a check that guards against this though, so if this is indeed the case, that must be erroneously passing. But if that check is erroneously passing, then it's possible to allocate into unmapped memory, which is not a failure mode I've seen yet (though certainly possible).

[heschi]

It happens frequently on the x/tools builders:
https://build.golang.org/log/77fcae35bcf997b299f2290d37d1a7055aba5a4e
https://build.golang.org/log/da05324aafea64620a340f8d0891e59916edcf1e
https://build.golang.org/log/3738682eca81e0e3ff0a89a72c74c27158fe78d0
https://build.golang.org/log/301b41503517225d7d640ee81462da9d45343e7b
https://build.golang.org/log/053b0b7757d23ab3c76c1c632ce9529855f78204
I've also seen it on a couple of trybot runs if you need more.

edit: looking more closely, https://build.golang.org/log/3738682eca81e0e3ff0a89a72c74c27158fe78d0 is a different crash in the same place. Suspicious?

[mknyszek]

Ah, excellent. Thank you @heschik.

This supports my hypothesis, though it's curious that it's the same test every time. It's probably just quite good at tickling some edge case.

I was able to reproduce once in a gomote (unfortunately it dumped the output into my terminal window and I lost all the useful info) but it's timing out now. Now that I can reproduce though it should be easier to get to the bottom of this.

[mknyszek]

I think I found the problem. Will push a fix in a bit.

[gopherbot]

Change https://golang.org/cl/206978 mentions this issue: runtime: check summary before scavenging in fast path

[mknyszek]

@heschik confirmed this fixes the problem in the x/tools test (10 consecutive runs and no failure; the tests in that package take a while to run). It's a bit wild to me how consistently that test was able to trigger this, but it was nice to have a consistent reproducer. :)