Clarify number and position of wildcard labels

[weppos]

I made a commit a few weeks ago that introduced a rule like *.*.private.domain and the commit caused the build to fail.

According to our website, that is supposed to be a valid format:

Wildcards are not restricted to appear only in the leftmost position, but they must wildcard an entire label. (I.e. ..foo is a valid rule: *bar.foo is not.)

@rockdaboot mentioned a potential incompatibility of Chromium if we allow multiple wildcards. libpsl is currently not compatible with multiple wildcards, and to be fair I haven't tested my Ruby implementation either.

@gerv @sleevi can we clarify whether multiple wildcard labels are accepted? Specifically, we should be more clear if the following rules are valid:

// multiple leading wildcards (common case)
*.*.foo.bar

// single wildcard, but inside the rule
foo.*.bar

// multiple willdcards, inside the rule
foo.*.*.bar

// multiple willdcards, inside the rule, non consecutive
foo.*.bar.*.baz

// I suppose this is invalid
foo.*

The current list definition doesn't explicitly deny these rules, they are supposed to be valid.

Once the decision is taken, I think we should:

1. Update the website to make it more clear
2. Add some corresponding tests in the test file
3. Should the rules be considered invalid, I'll also add the corresponding rules to the linter I'm working on,
4. Should the rules be considered valid, we should ping the various lib maintainers

[rockdaboot]

// single wildcard, but inside the rule
foo.*.bar

This would not allow exceptions right now, since (AFAIR) the exclamation mark has to be the first character (and must not be followed by a dot).
If this form will be allowed, the rules for exceptions must IMO change (e.g. foo.!baz.bar).

[gerv]

I believe the Mozilla implementation also didn't allow multiple wildcards. It seems like there are lots of implementations out there which don't. Given that their use is quite niche (what was the use case for the rule you added?), perhaps we need to bow to the inevitable and update the spec to match reality.

[sleevi]

Right, the Chrome code doesn't presently, but we could likely fix that. But in the implementations I've seen, I haven't seen any support for multi-wildcards, despite what the website says.

[User4574]

Hi

As you can see from our pull request here:

#179

we need to get the double wildcard issue addressed, as we need our customer domain suffix in the PSL.

We name machines under MACHINE.GROUP.ACCOUNT.CLUSTER.bigv.io, so ...bigv.io would be ideal, or *..uk0.bigv.io.

Thanks
Næþ'n

[gerv]

When you say "get it addressed", do you mean "update all the PSL-using code in the universe to support double wildcards"? It seems to me that the most likely way for this to be addressed would be to document what appears to be a fairly universal client limitation - that is, that double wildcards are not supported.

[weppos]

@gerv @sleevi I'm curious, what would be the implications of having one of those entries in the list as of today?

Would it cause some sort of crash, or will it simply "not work as expected"?

I'm asking because it may make sense in the long run to support it. And if this is the direction we want to try to take, we may simply document it as "compatibility not guaranteed". Clients and consumers will eventually update their codebase.

In other words, we allow, but we document as "it may not work on all clients". Different is if rolling out today would cause Firefox or Chrome to potentially crash.

What do you think?

[gerv]

I suspect the answer is implementation-dependent... We'd have to try it and see.

[sleevi]

@weppos I don't think we can make a demarcation between "crash" and "not-work". That's the sort of decision PSL consumers should take. For example, how would we quantify if a double-wildcard caused entries subsequent on the PSL to be ignored due to parsing issues? We don't have the means to quantify that.

That said, I'm not fundamentally opposed to changing, so long as major platforms will end up supporting them.

[myronmarston]

@benkirzhner and I are working on an Elixir public suffix library and have found it much, much easier to support only a single left-most wildcard than to support multiple wildcards or wildcards at any position. If we only support a single wildcard at the left-most position, our algorithm can do a very simple, constant-time set membership check (e.g. ["*" | suffix] in wild_card_rules) to see if a domain matches a wild card rule. Full wildcard support as per the spec is much more complicated, and we would have to do a slow linear scan of all the rules to compare a domain against each or put the wildcard rules into some kind of a tree data structure and traverse that.

Given that up to now, the only usage of wildcards in the data file has been in the left-most position...is it really necessary to add extra complexity to every implementation to support wildcards at any level and multiple wildcards?

If it is decided to support the added complexity, I would ask that the tests be updated to include examples of multiple wildcards and wildcards at less common positions so implementors can use those in their test suites.

[rockdaboot]

*.label and *.*.label would allow fast lookups as we do it now.

One or more wildcards somewhere inbetween (e.g. foo.*.bar) result in slower and more complex lookup algorithms.

Despite from that, we need new exception rules.
E.g. when having *.*.bar, we would need exceptions with wildcards like !*.doo-*.bar.

[weppos]

@rockdaboot @gerv @sleevi given:

• the conversation above
• most (if not all) the PSL consumers/libraries today assume one single *
• this is consistent with other notations (e.g. the CA use of * in the subjectAltName and the dsName)

barring any objection I am going to update the format documentation and the site to permit the use of one single wildcard, and only as the left outermost label.

Should the need change, we can always revisit the decision. But it looks like there is no current practical application of it.