Barrier Certificates for Assured Machine Teaching

From an optimal control perspective, a learning system (e.g., a machine learning algorithm, or a human learner) defines a dynamical system where the state (i.e., learner’s hypothesis) is driven by training data [1]. In this respect, machine teaching, i.e., the algorithmic framework of designing an optimal training set for learning a target hypothesis, can be viewed as optimal control for learning [2]. In a typical setting of machine teaching, the target hypothesis is given to the algorithm, and the goal of the teacher (machine) is to generate a minimal sequence of training examples such that the target hypothesis can be learned by a learner (human or another machine) from a finite set of hypotheses.

One popular learner’s model studied in the machine teaching literature is the version space learner. In such settings, the learner maintains a subset of hypotheses that are consistent with the examples received from a teacher, and outputs a hypothesis from this subset. Based on different assumptions on the learner’s behavior, multiple variants of the version space learner model has been studied in algorithmic machine teaching, leading to different notions of teaching complexity: For instance, (i) the “worst-case” model [3] assumes that the learner’s behavior is completely unpredictable, and (ii) the “preference-based” model [4] assumes that she has a global preference over the hypotheses. These models are typically studied under the batch setting, where the teacher constructs a set of examples and provides them to the learner at once. Recently, [5] considered the state-dependent preference-based model, which generalizes the preference-based model of [4] to the adaptive setting. The state-dependent preference-based model assumes that the learner’s choice of next hypothesis depends on some local preferences defined by the learner’s state (i.e., the current hypothesis). In the sequential machine teaching setting, the teacher, after showing each example, obtains feedback about the hypothesis that the learner is currently entertaining; such feedback is further utilized to guide the selection of future teaching examples.

In this paper, we use notions from hybrid systems analysis framework to study the state-dependent preference-based machine teaching model with the aim of verifying whether a given machine teaching method has assured teaching performance. We first show that state-dependent preference-based machine teaching model can be represented by a POMDP. Once this POMDP is formulated, we show that the evolution of the beliefs over the states of this POMDP can be described by a discrete-time switched system (also see [6, 7]). We use barrier certificates to verify whether the beliefs of this POMDP belong to some subset of the reachable belief space, which, in turn, corresponds to the probability of teaching of a hypothesis. From a computational standpoint, we show that these barrier certificates can be decomposed and constructed using SOS programming. We demonstrate the efficacy of our proposed methodology by comparing and analyzing two machine teaching methods.

The rest of this paper is organized as follows. We describe the state-dependent teaching model in the next section. In Section III, we propose a POMDP representation for machine teaching. In Section IV, we briefly discuss a hybrid system that describe the evolution of this POMDP. In Section V, we formulate a set of conditions based on barrier certificates for verifying the teaching performance and show how the calculations can be decomposed. In Section VI, we propose a computational approach using SOS programming to find the barrier certificates. We elucidate the proposed method with an example in Section VII and conclude the paper in Section VIII.

Notation: $\mathds{R}$ and $\mathds{N}$ denote the sets of real numbers and non-negative integers $\{0,1,2,\ldots\}$, respectively. $\mathds{N}_{\geq l}$, with $l\in\mathds{N}$, denotes $\{l,l+1,l+2,\ldots\}$. $\mathcal{R}[x]$ accounts for the set of polynomial functions with real coefficients in $z\in\mathds{R}^{n}$, $p:\mathds{R}^{n}\to\mathds{R}$ and $\Sigma\subset\mathcal{R}$ is the subset of polynomials with an SOS decomposition; i.e., $p\in\Sigma[x]$ if and only if there are $p_{i}\in\mathcal{R}[x],~i\in\{1,\ldots,k\}$ such that $p=p_{i}^{2}+\cdots+p_{k}^{2}$.

We now state the adaptive machine teaching protocol, and describe the state-dependent learner’s model of [5].

Given the state-dependent teaching model as described in Section II, we can represent machine teaching as a sequential decision making under uncertainty scenario. To this end, we propose a POMDP representation for the learner based on the state-dependent teaching model. The POMDP model can be described as follows.

Here, the observation model $O(y_{t}\mid h_{t},z_{t})$ defines how the version space gets updated. When referring to the “version space” learners, we are implicitly considering the “noise-free” setting, i.e., all consistent hypotheses are uniformly distributed, or equivalently, $O(y_{t}\mid h_{t},z_{t})$ is binary. Moreover, according to (1), the transition function $T(h,z_{t-1},h^{\prime})$ defines a uniform distribution: the learner only goes to the hypotheses $h^{\prime}$ that are the most preferred; hence, $T$ induces a uniform distribution over the most preferred hypothesis according to the preference function $\sigma$.

The learner starts with an initial hypothesis $h_{0}$ and over a sequence of trials, in which an example $z_{t}\in\mathcal{Z}$ is shown and the learner receives a corresponding observation $y_{t}\in\mathcal{Y}$, develops a belief in the new hypothesis $h$. Then, the hypothesis belief evolves according to

The objective of a teaching policy is then to assure that the learner learns the target hypothesis $h^{*}\in\mathcal{H}$ in $t^{*}$ number of trials. That is,

where we refer to $0<\lambda\leq 1$ as the teaching performance. In addition, given a teaching policy, we are often interested in finding the minimum number of trials such that the learner learns a target hypothesis, i.e.,

Ideally, given a pre-specified number of trials ${t^{*}}$, a teaching algorithm is perfect, if $\lambda=1$, i.e., the probability of learning the target hypothesis after ${t^{*}}$ number of examples is one. However, achieving a perfect teaching algorithm in $t^{*}$ number of trials may not be realistic. In practice, it is desirable that we teach the target hypothesis with teaching performance $\lambda\geq 0.75$.

Checking whether (3) holds by solving the learning POMDP directly is a PSPACE-hard problem [10]. In this section, we show that the learning POMDP can be represented as a special hybrid system [11], specifically, a discrete-time switched system [12, 13, 14].

The belief update equation (2) can be characterized as a discrete-time switched system, where the actions $a\in A$ define the switching modes. Formally, the hypothesis belief dynamics (2) can be described as

where $b$ denote the belief vector belonging to the belief unit simplex $\mathcal{B}$ and $b_{0}=p_{0}$. In (5), $z\in\mathcal{Z}$ denote the examples that can be interpreted as the indices for the switching modes, $y\in\mathcal{Y}$ are the observations representing inputs, and $t\in\mathds{N}_{\geq 1}$ denote the discrete time instances. The (rational) vector fields $\{f_{z}\}_{z\in\mathcal{Z}}$ with $f_{z}:[0,1]^{|\mathcal{Z}|}\times\mathcal{Y}\to[0,1]^{|\mathcal{Z}|}$ are described as the vectors with rows

where $f_{z}^{h^{\prime}}$ denotes the $h^{\prime}$th row of $f_{z}$.

We consider two classes of problems in learning POMDP verification:

• 1.

  Arbitrary-Policy Verification: This case corresponds to analyzing (5) under arbitrary switching with switching modes determined by the examples $z\in\mathcal{Z}$.

• 2.

  Fixed-Policy Verification: This corresponds to analyzing (5) under state-dependent switching. In fact, a teaching policy $\pi:\mathcal{B}\to\mathcal{Z}$ (a mapping from the hypothesis beliefs into examples) determines regions in the belief space where each mode (example) is active.

Both cases of switched systems with arbitrary switching and state-dependent switching are well-known in the systems and controls literature (see [15, 16] and references therein).

In the following, we describe a method based on barrier certificates to verify the teaching performance as given by (3). We then focus on the two cases of arbitrary policy verification and fixed-policy verification. We further show that in both cases, the calculation of the barrier certificates can be decomposed.

In order to check the teaching performance, we consider following teaching-failure set

which is the complement of (3).

We have the following result.

In this section, we propose techniques for finding the barrier certificates and checking whether a teaching performance is satisfied using SOS programming [21, 22].

In order to cast the conditions of Theorem 1-3 into SOS programs, we need polynomial/rational variables and require the associated sets to be semi-algebraic. Fortunately, these requirements naturally fit our problem. The hypothesis belief space is a semi-algebraic set. Moreover, the right-hand side of the belief update equation (5) is composed of rational functions in the belief states $b_{t}(h)$, $h\in\mathcal{H}$. That is,

Furthermore, the teaching-failure set (6) is a semi-algebraic set.

At this point, we present conditions based on SOS programs to verify a given teaching performance of a teaching algorithm.

In order to illustrate the proposed framework, we consider a toy scenario, where the teacher aims to teach/steer a human learner to reach a goal state in a physical environment. Each hypothesis/node corresponds to some unexplored territory, and there exists an example which flags the territory as explored. The learner prefers local moves, and if all neighboring territories are explored, the learner jumps to the next closest one.

The physical environment is characterized by a $4\times 4$ lattice corresponding to $16$ hypotheses. The target hypothesis is located at $h^{*}=(4,4)$. The teacher has $16$ choices of locations on the lattice to show to the student as examples. The student then receives two labels based on its answer $y\in\{-1,1\}$. The preference function $\sigma(h^{\prime};h)$ is given by the minimum distance between hypotheses described by $\ell_{1}(h^{\prime};h)$.

In this example, we compare two teaching algorithms in the adaptive setting, where the teacher observes the learner’s hypothesis at each iteration. The Myopic algorithm is a greedy approach which, at each iteration, picks the teaching example such that after observing the label, the worst-case rank of the target hypothesis in the learner’s resulting version space is the smallest. The Ada-L algorithm aims to teach the learner some intermediate hypothesis at each iteration, i.e., it aims to direct the learner to transit to a hypothesis that is “closer” to the target hypothesis. For more details of the algorithms please refer to [5].

Each algorithm provides a set of policies for which we seek to find the minimum number of trials such that the following teaching performance is assured

To this end, we minimize the number of trials $t^{*}$ such that (27)-(29) are satisfied. We start by a large number of trials ($16$ in this case) and decrease it until no barrier certificate can be found to verify the teaching performance. We fix the degree of variables $B_{i}$, $p^{l_{1}}_{i}$, $p^{l_{2}}_{i}$, $p^{l_{3}}_{i}$, and $p^{f}_{i}\in{\Sigma}[b]$, $i\in\{1,2,\ldots,N\}$ in Corollary 3 to $2$ and search for the certificates. In order to check the SOS conditions formulated in Section VI, we use diagonally-dominant-SOS (DSOS) relaxations of the SOS programs implemented through the SPOTless tool [23] (for more details see [24, 25]).

The results on finding the minimum number of trials $t^{*}$ for which the teaching performance is satisfied were as follows.

We presented a method based on barrier certificates to assure the performance of machine teaching algorithms. Our computational method was in terms of SOS programs, where we used DSOS relaxations. It was shown in [26] that using sparse SOS (SSOS) programs leads to more efficient and less conservative results. Future work can explore the use of more scalable SOS relaxations such as SSOS.