Learning Time Dependent Choice

The two notions of learnability we consider are the PAC model and the active model. We now state the standard definitions of PAC and active learning in the context of preference relations. Most of the following definitions for the PAC setting are taken from [2] since the setup involving preference relations is identical. These definitions of course apply to the more general setting of concept learning (for example, see [6]).

A collection $\mathcal{P}$ of preference relations is (PAC) learnable if there is a learning rule $\sigma$ such that for every $0<\varepsilon,\delta<1$, there is $s(\varepsilon,\delta)\in\mathbb{N}$ such that for every $n\geq s(\varepsilon,\delta)$, $\succsim\in\mathcal{P}$, and $\mu\in\Delta(X\times X)$,

where

is the hypothesis preference relation produced by the learning rule²² 2 $\mu^{n}$ denotes the product measure induced by $\mu$ on $(X\times X)^{n}$.. The quantity $s(\varepsilon,\delta)$ is called the sample complexity of the learning rule $\sigma$.

The complexity of learning is commonly quantified by the Vapnik-Chervonenkis (VC) dimension, which we now define. A set of points $\{(x^{1},y^{1}),\ldots,(x^{n},y^{n})\}$ from $X\times X$ is shattered by a model of preferences $\mathcal{P}$ if for every vector of labels $(a_{1},\ldots,a_{n})\in\{0,1\}^{n}$, there is a preference relation $\succsim\in\mathcal{P}$ that realizes the labelling, i.e. for $i=1,\ldots,n$ we have that $x^{i}\succsim y^{i}$ if and only if $a_{i}=1$. In this case, $\mathcal{P}$ is said to rationalize the dataset $\{((x^{1},y^{1}),a_{1}),\ldots,((x^{n},y^{n}),a_{n})\}$. The VC dimension of $\mathcal{P}$, denoted by $VC(\mathcal{P})$, is the largest integer $n$ such that there exist $n$ points that are shattered by $\mathcal{P}$.

Blumer et al. [6] in 1989 proved that learnability is equivalent to having a finite VC dimension³³ 3 This result requires $\mathcal{P}$ to satisfy a certain measurability requirement. We note in Section 4 that the models of choice we consider all satisfy said requirement..

The VC dimension (denoted by $d$ for the remainder of this subsection) also plays a role in the sample complexity of learning a model of preferences. In the same paper, Blumer et al. [6] show that any algorithm that outputs a hypothesis consistent with the data seen is a valid learning rule requiring sample complexity

In 2016, Hanneke [14] showed that these bounds (after a small improvement) are tight: the optimal sample complexity of PAC learning is

The other learning model we consider is the active learning framework, where the analyst has some control over the data from which he learns. In stream-based selective sampling, points drawn according to an unknown distribution are presented to the analyst as before, but without the labels. The analyst can choose whether or not to query the label of a given point, and the complexity of the learning rule is measured by label complexity, i.e. the number of labels requested by the analyst. Disagreement based active learning refers to the paradigm in which the learner only requests labels on points that significantly reduce the hypothesis space. The disagreement of a preference model with respect to the underlying distribution is quantified through the disagreement coefficient $\theta$, which is defined in Section 5. A finite disagreement coefficient implies (for the underlying distribution) an exponential improvement in label complexity over the sample complexity of PAC learning. For example, the CAL algorithm [9, 10, 13], a simple disagreement based learning algorithm, yields a label complexity of

In the membership queries model, the analyst is allowed to request the label for any point at any time. There appears to be a dearth of literature/results pertaining to the complexity of membership query algorithms for learning when the hypothesis space is infinite. One explanation for this is that improvements to the “passive” disagreement based methods used in the stream-based setting would need specific information about the problem domain: disagreement based methods are designed to work on a very general class of concept learning problems without assuming anything about the learning space. In our case, we have specific details about how the preference relations take shape. Thus, the membership query model turns out to be an interesting and useful perspective to use in the study of learning preference models.

For a more detailed survey of active learning, see [10].

We now present the definitions for the discounted utility models of intertemporal choice. An agent chooses between plans or vectors in $X=\mathbb{R}^{T}$ that encode payoffs over $T$ time periods. A preference relation over plans is a binary relation $\succsim\subseteq\mathbb{R}^{T}\times\mathbb{R}^{T}$.

The most important model of intertemporal choice is the discounted utility model, in which the agent’s payoffs $x_{t}$ for having chosen a plan $x\in\mathbb{R}^{T}$ are reduced, or discounted, as $t$ increases from $1$ to $T$. In its most general form, we can characterize the preference relations that follow time discounting as follows:

We use the following notation for the preference models arising from the three most commonly studied discounting functions $D$:

• •

  $\mathcal{P}_{\mathcal{D}}$ denotes the set of preferences that satisfy the discounted utility model.

• •

  $\mathcal{P}_{\mathcal{ED}}$ denotes the set of preferences that satisfy the discounted utility model with exponential discounting: $D(t)=\delta^{t}$ for $\delta\in(0,1)$.

• •

  $\mathcal{P}_{\mathcal{HD}}$ denotes the set of preferences that satisfy the discounted utility model with hyperbolic discounting: $D(t)=\frac{1}{1+t\alpha}$ for $\alpha>0$.

• •

  $\mathcal{P}_{\mathcal{QHD}}$ denotes the set of preferences that satisfy the discounted utility model with quasi-hyperbolic discounting: $D(t)=1$ if $t=1$, $D(t)=\beta\cdot\delta^{t-1}$ if $t>1$ for $\beta,\delta\in(0,1)$.

For a more thorough exposition on the various discounted utility models of intertemporal choice, see [7].

The first part of our paper investigates the situation of an analyst trying to learn the preference relation by which an agent makes choices, but has no control over what choices he gets to observe, and is agnostic to the process by which they are drawn. We thus adopt the PAC learning model.

The agent chooses between plans that encode payoffs over $T$ periods of time and evaluates the total payoff of a plan vector $x\in\mathbb{R}^{T}$ according to private weights $w_{1},\ldots,w_{T}$ that he multiplicatively applies to each state: $\text{payoff}(x)=\sum_{t=1}^{T}w_{t}x_{t}.$ This defines a model of preference relations, which we denote by $\mathcal{P}_{\mathcal{W}}$, where for any $\succsim\in\mathcal{P}_{\mathcal{W}}$, there exists a vector of weights $w=(w_{1},\ldots,w_{T})\in\mathbb{R}^{T}$ such that

In [2], it is shown that $T-1\leq VC(\mathcal{P}_{\mathcal{W}})\leq T+1$. In the context of choice over time, however, this model is extremely general and does not capture any of the intuitive notions of how an agent values payoffs when they are delayed⁴⁴ 4 In [2] the complete control over weights is used to model choice under uncertainty, which calls for such generality since the agent’s beliefs/weights are given by an element of the probability simplex on $\mathbb{R}^{T}$.. For example, the discounted utility models of intertemporal choice require the weights to be of a particular functional form. Moreover, when there is no structure to the discount function we cannot improve the bounds on $\mathcal{P}_{\mathcal{W}}$:

This leads us to the motivating question of the first part of the paper: what structural conditions can we impose on the weights $w_{1},\ldots,w_{T}$ such that this bound can be improved?

We investigate the situation where the agent computes his weights by evaluating polynomials at a private parameter $\delta$. Specifically, let $Q_{1},\ldots,Q_{T}$ be polynomials of degree at most $d$, and suppose the agent evaluates total payoff of a plan vector $x\in\mathbb{R}^{T}$ by $\text{payoff}(x)=\sum_{t=1}^{T}Q_{t}(\delta)x_{t}$. Consequently, let $\mathcal{P}_{\mathcal{PW}}$ be the model of preference relations parametrized by $\delta$ such that

This class of preference models allows us to approximate preference relations where the weights are given by any real valued functions – we choose $Q_{1},\ldots,Q_{T}$ to be the appropriate Taylor polynomials. Moreover, existing models of intertemporal choice fit this characterization – for example $\mathcal{P}_{\mathcal{ED}}$ and $\mathcal{P}_{\mathcal{HD}}$.

We additionally consider a slightly larger class of preference models where the agent has a private parameter $\beta$ (in addition to $\delta$) that in evaluating total payoff of a plan vector $x\in\mathbb{R}^{T}$ allows the agent to modify the constant term $\sum_{t=1}^{T}Q_{t}(0)x_{t}$ of the polynomial $\sum_{t=1}^{T}Q_{t}(\delta)x_{t}$. This model aims to more generally capture the effects of the $\beta$ parameter in quasi-hyperbolic discounting. For polynomials $Q_{1},\ldots,Q_{T}$ of degree at most $d$, let $\mathcal{P}_{\mathcal{BPW}}$ be the model of preference relations parametrized by $\beta$ and $\delta$ such that $x\succsim y$ if and only if

Our main results show that with this additional structure on the preference model, we can achieve an exponential improvement in the bounds for the VC dimension of $\mathcal{P}_{\mathcal{W}}$ obtained in [2]⁶⁶ 6 It is important to note that the classes $\mathcal{P}_{\mathcal{PW}}$ and $\mathcal{P}_{\mathcal{BPW}}$ are defined for a given $Q_{1},\ldots,Q_{T}$. That is, the analyst knows $Q_{1},\ldots,Q_{T}$, and is trying to learn the parameters $\beta$ and $\delta$. If the $Q_{1},\ldots,Q_{T}$ are private information only available to the agent, we are in no better shape than in the case of $\mathcal{P}_{\mathcal{W}}$..

Note that when $Q_{1},\ldots,Q_{T}$ have degree at most polynomial in $T$, we obtain an exponential improvement over the linear growth of $VC(\mathcal{P}_{\mathcal{W}})$. We show that in this case, we get a tight (asymptotic) bound of $\log T$:

An interesting feature of Theorems 3.2 and 3.3 is that for fixed $Q_{1},\ldots,Q_{T}$ with degrees at most $T-1$, $VC(\mathcal{P}_{\mathcal{PW}})$ and $VC(\mathcal{P}_{\mathcal{BPW}})$ satisfy the same asymptotic bounds, so giving the agent an extra parameter that allows control over the constant term of the polynomial $\sum_{t=1}^{T}Q_{t}(\delta)x_{t}$ does not introduce a significant amount of richness to the model.

Applying Theorems 3.2 and 3.3 to the discounted utility models, we have:

Thus, $\mathcal{P}_{\mathcal{D}}$, $\mathcal{P}_{\mathcal{ED}}$, $\mathcal{P}_{\mathcal{HD}}$, and $\mathcal{P}_{\mathcal{QHD}}$ are all learnable. $\mathcal{P}_{\mathcal{D}}$ requires a minimum sample size that grows linearly with $T$, while $\mathcal{P}_{\mathcal{ED}}$, $\mathcal{P}_{\mathcal{HD}}$, and $\mathcal{P}_{\mathcal{QHD}}$ require a minimum sample size that grows logarithmically in $T$.

The main technique in proving Theorems 3.2 and 3.3 is interpreting the shattering criteria as a statement about the sign combinations achieved by a collection of polynomials. The upper bound on the VC dimension follows from an upper bound on the number of sign combinations a collection of polynomials can achieve. In demonstrating the lower bound on the VC dimension, we construct a set of points that is shattered by finding polynomials achieving all possible sign combinations – chosen according to a Hamiltonian path in the $\log(T-1)$-dimensional hypercube.

Our other results concern the active learning framework, which broadly deals with situations in which the analyst has some control over the choices he observes and learns from. The two models we consider are stream-based selective sampling and learning via membership queries. A large body of active learning research is devoted to the stream-based model, specifically focusing on disagreement based algorithms – a class of learning algorithms that instructs the analyst only to request labels on points he sees that reduce the hypothesis space significantly. In the most general setting of concept learning, this is a useful framework since the error guarantees can be described using the same setup as the PAC model. Moreover without additional information about the problem domain, it is unclear how to devise efficient algorithms that are more specific in instructing the analyst on what questions to ask.

We find that the stream-based model is in general difficult to analyze for the preference relations we work with. This difficulty seems to arise from the apparent need to quantify disagreement in order to explicitly write down learning guarantees. Though in most general situations it is unclear how to quantify disagreement for our preference relations, we present a redeeming situation for which we are able to provide a precise analysis of the learning guarantees for $\mathcal{P}_{\mathcal{ED}}$. Here, the analyst can learn $\mathcal{P}_{\mathcal{ED}}$ with an exponential improvement in label complexity over the guarantees provided by the PAC model. This is achieved via a computation of the disagreement coefficient (defined in Section 5) of $\mathcal{P}_{\mathcal{ED}}$ for a specific distribution.

The measure $\mu$ we construct is induced by the product Lebesgue measure on $(0,1)^{T-1}$, and allows us to precisely translate statements about disagreement into statements about the roots of polynomials arising from a given choice. Once we have defined $\mu$, the calculation of $\theta$ follows from basic probability arguments.

Now, in our case the analyst has structural information regarding the preference relation of the agent he is questioning. We find that allowing the analyst full control over the membership queries he makes yields a learning algorithm that, despite its simplicity, takes advantage of this extra structure and yields a significant improvement in complexity over the stream-based setting. Additionally, the membership queries model naturally describes an experimental environment in which the analyst is able to ask the agent questions in real time.

We show that when the preference model satisfies some relatively benign structural requirements, even very naive algorithms outperform the guarantees provided by CAL in the stream-based setting. The example algorithm we give, relying on a simple binary search, has a query complexity of $O(\log 1/\varepsilon)$, which gets rid of the $\log T$ dependence in Theorem 3.5.

The class of preference models is defined as follows: let $g_{1},\ldots,g_{T}:\mathbb{R}\to\mathbb{R}$ be a collection of functions satisfying the properties listed in Section 5.3 and consider the model of preference relations $\mathcal{P}$ parametrized by $\delta$ where $x\succsim y$ if and only if $\sum_{t=1}^{T}g_{t}(\delta)x_{t}\geq\sum_{t=1}^{T}g_{t}(\delta)y_{t}$⁷⁷ 7 As before, the $g_{1},\ldots,g_{T}$ are known to the analyst.. We have

The remainder of the paper is devoted to proving the results discussed in this section.

While PAC learnability is a positive result, it does not take into account the computational complexity of computing a hypothesis. Blumer et. al. [6] show that any learning rule that outputs a hypothesis consistent with the data seen yields with high probability a hypothesis that has very low error. However, if the problem of outputting a consistent hypothesis is computationally intractable, PAC learnability on its own is perhaps unsatisfying. In this section we note that the discounted utility models of intertemporal choice are efficiently learnable. This is due to an algorithm of Grigor’ev and Vorobjov [12] for solving a system of polynomial inequalities.

For notational convenience, it will be useful to write $\mathcal{P}=\{\mathcal{P}^{T}\}_{T\geq 1}$, for each of the models above, where $\mathcal{P}^{T}$ is the collection of preference relations for a given $T$. Moreover, suppose acts are chosen from $[-1,1]^{T}$ instead of $\mathbb{R}^{T}$. It is clear that this does not change any of the analysis above.

Polynomial learnability, as defined by Blumer et. al. [6], stipulates that the learning rule be computable in $\text{poly}(1/\varepsilon,1/\delta,T)$-time (where $\varepsilon$ and $\delta$ denote the error threshold and confidence threshold respectively). Polynomial learnability is equivalent to the task of outputting a hypothesis consistent with the given data set in polynomial time [6].

We have just shown that $VC(\mathcal{P}^{T})\sim\log(T-1)$. Using techniques involving algebraic geometry, Grigor’ev and Vorobjov [12] present an algorithm to solve a system of $N$ polynomial inequalities with a $\text{poly}(N,T)$ runtime. This serves as our r-poly hy-fi, and thus we obtain:

For notational convenience, $\succsim_{\delta}$ will to refer to the preference relation in $\mathcal{P}_{\mathcal{ED}}$ with discounting factor $\delta$.

Let $\mu$ be a distribution on $\mathbb{R}^{T}\times\mathbb{R}^{T}$. $\mu$ induces a metric on $\mathcal{P}_{\mathcal{ED}}$ by $d(\succsim_{\delta},\succsim_{\gamma})=\mu(\succsim_{\delta}\triangle\succsim_{\gamma})$, and thus we can define the closed ball of radius $R$ centered at $\succsim_{\delta}$ by

For $V\subseteq\mathcal{P}_{\mathcal{EU}}$, the disagreement region of $V$, $\operatorname{Dis}(V)$ is defined by

Intuitively, $\operatorname{Dis}(V)$ is the collection of points $(x,y)$ such that we can find two hypothesis relations in the current version space that rank $x$ and $y$ differently.

If $\succsim_{\delta}$ is the target preference relation, the disagreement coefficient of $\succsim_{\delta}$ with respect to $\mu$ is the quantity

In this subsection, we define a distribution $\mu$ on $\mathbb{R}^{T}\times\mathbb{R}^{T}$ and show that the disagreement coefficient of $\mathcal{P}_{\mathcal{ED}}$ with respect to $\mu$ is $2$.