Support for rejecting non-canonical encodings

[emilypi]

Consider the following Base64-encoded string: "ZE==". What is the correct result of decoding the string "ZE=="?

Answer: It is not valid Base64, but it still satisfies the decoder's understanding of Base64 encoded data. Unfortunately, there is no way to construct such a result from binary, which leads to confusion - the decoder in base64-bytestring is not smart enough to differentiate such data. In fact, this value never round trips:

П> decode "ZE=="
Right "d"
П> encode "d"
"ZA=="
П> fmap encode (decode "ZE==")
Right "ZA=="

A more correct implementation should fail with an "invalid input" error. Or we can leave it as is and leave a note about the support status for "impossible by construction" inputs to the decoder.

[hvr]

I recall stumbling upon this but wasn't sure about the spec;

However, the docstring for our decode operation says

Decode a base64-encoded string. This function strictly follows the specification in RFC 4648.

While we also have a "lenient" variant of that function. Assuming the RFC 4648 in fact forbids ZE== (which is what I assume you're claiming), I'd expect decode to reject it; while decodeLenient to tolerate it...

[emilypi]

Assuming the RFC 4648 in fact forbids ZE== (which is what I assume you're claiming)

It's not specified because (go figure) the RFC is wrong and incomplete in many respects. Not just w.r.t. specifying reasonable bounds for what is and is not considered "valid base64", but also what is and is not considered correct base64. In this case, ZE== is externally valid base64, but not internally in the sense that it round trips. There is an implicit assumption made in the RFC that assumes data will be decoded from encoded data, and so never mentions it.

You're not alone though - nearly every base64 implementation has this flaw, and it's relatively benign because there is no way to construct such a value from bytestring data. The way to fix this is to track invalid states via some clever bitmasking when decoding, which requires a change to the algorithm and is guaranteed to regress performance.

I'd expect decode to reject it; while decodeLenient to tolerate it...

Ideally that's what you'd want, yeah.

[hvr]

This section from RFC 4648 is also relevant:

3.5.  Canonical Encoding

   The padding step in base 64 and base 32 encoding can, if improperly
   implemented, lead to non-significant alterations of the encoded data.
   For example, if the input is only one octet for a base 64 encoding,
   then all six bits of the first symbol are used, but only the first
   two bits of the next symbol are used.  These pad bits MUST be set to
   zero by conforming encoders, which is described in the descriptions
   on padding below.  If this property do not hold, there is no
   canonical representation of base-encoded data, and multiple base-
   encoded strings can be decoded to the same binary data.  If this
   property (and others discussed in this document) holds, a canonical
   encoding is guaranteed.

   In some environments, the alteration is critical and therefore
   decoders MAY chose to reject an encoding if the pad bits have not
   been set to zero.  The specification referring to this may mandate a
   specific behaviour.

So the RFC does give terminology to this degree of freedom; and we MAY want to add opt-in support for rejecting "non-canonical" encodings in a future release.

[emilypi]

Since we now support head validations for malformed padded and unpadded base64 + base64url, I say yes on this, and I'm going to begin work on this.