v2.7 expands .env file values

[antonagestam]

In prior versions, this .env file would result in the variable BASIC_AUTH_CLIENTS being set to the value {"a_client":"md5$ldjDcN3a5x7x$46b3330841bbe4974ef43cb8b97034e"}:

BASIC_AUTH_CLIENTS={"a_client":"md5$ldjDcN3a5x7x$46b3330841bbe4974ef43cb8b97034e"}

In version 2.7 this results in the following error message:

WARN[0000] The "ldjDcN3a5x7x" variable is not set. Defaulting to a blank string. 
Invalid template: "{\"a_client\":\"md5$ldjDcN3a5x7x$46b3330841bbe4974ef43cb8b97034e\"}"

.env files used to be "dumb" and not allow any expansions, which is a rigid but stable behavior. This change is problematic because it isn't backwards compatible, and will require synchronized upgrades across environments.

Output of docker compose version:

Docker Compose version v2.7.0

[ulyssessouza]

In Docker Compose V2 dollar signs have to be escaped by doubling it ($$).
Another solution would be surrounding the whole string with single quotes (''), like in:

BASIC_AUTH_CLIENTS ='{"a_client":"md5$ldjDcN3a5x7x$46b3330841bbe4974ef43cb8b97034e"}'

This indicates that the string mustn't be evaluated.

[antonagestam]

@ulyssessouza What are you referring to by Docker Compose V2, spec or implementation? Your statement at least seems to contradict this document, is that not the correct spec to look at?

The value of VAL is used as a raw string and not modified at all. If the value is surrounded by quotes (as is often the case for shell variables), the quotes MUST be included in the value passed to containers created by the Compose implementation.

If you are referring to the version of the implementation, I don't think that's true either for versions prior to 2.6 or perhaps 2.7?

[ulyssessouza]

I mean the implementation. So this git repo.
Thanks for the reference. I'll propose a change in the spec for that.

I don't remember exactly when the behaviour was introduced, but it was with godotenv. So I think that was since the beginning of version 2.

[laurentlbm]

This behaviour was not present for env_file before 2.7, I assume it was added because of this "fix": Fix environment variable expansion

It has caused me problems, as it seems to have for many others, and I find it a bit frustrating. The specification explicitly says that values are used "as a raw string and not modified at all." I seriously think this change has a broader impact than you realize: docker-compose >= 2.7 now doesn't implement the specification correctly and environment files aren't backward-compatible anymore... It's not just about expansion, a value surrounded by quotes isn't treated the same anymore! Many users might be affected by this change and not even realize it (yet).

The change in behaviour could be reverted, but that would affect all those who had to fix their environment files and those who depend on this feature. If it's not reverted and the specification is updated, it needs to be clear that the behaviour has changed and that not all implementations work the same.

Either way, I think this change needs to be communicated quickly and broadly so that people can react and fix their code.

[antonagestam]

@laurentlbm That's what I suspected too, but it was a little too cumbersome to downgrade or build previous versions of docker-compose for me locally to prove it.

[kiere]

I downloaded the version of Docker Desktop for Mac that I had before (4.9, I believe) and reinstalled. That solved the issue without making any changes, so it's definitely new.

[nephthys]

I downloaded the version of Docker Desktop for Mac that I had before (4.9, I believe) and reinstalled. That solved the issue without making any changes, so it's definitely new.

Same problem on the latest version of Docker Desktop for Mac (4.11.1 with Apple chip, Docker Compose version v2.7.0).

If I roll back to Docker Desktop 4.9.1 (Docker Compose version v2.6.0), it works again.