GaussianProcessesforMachineLearning资源-CSDN下载

需积分: 10 136 浏览量 2013-04-12 22:58:12 上传评论 1 收藏 675KB PDF 举报

高斯过程是一种在机械学习中广泛使用的非参数概率模型，它通过高斯过程回归（Gaussian Process Regression，GPR）和高斯过程分类（Gaussian Process Classification，GPC）等技术来处理回归和分类问题。在高斯过程模型中，协方差函数（covariance function）扮演着至关重要的角色。协方差函数被用于编码我们关于要学习的函数的假设，它定义了输入空间中点之间的相似性，或者说是“邻近性”。本章将介绍几种常用的协方差函数，并探讨它们的性质。需要对协方差函数进行一些基本术语的定义。例如，平稳协方差函数（stationary covariance function）是指只依赖于输入空间中两点之间差值的函数，即形式为f(x - x')。由于它在输入空间中对平移具有不变性，因此可以称为平稳的。此外，平稳过程可以具有常数均值，并且其协方差函数对平移保持不变。当一个过程的所有有限维分布都相同，并且对输入空间的平移保持不变时，该过程被称为严格平稳的。对于协方差函数来说，它必须满足正定（positive semidefinite）的性质，这是构建有效高斯过程模型的一个基本要求。正定性质确保了协方差矩阵是半正定的，这在数学上保证了概率模型的合法性和合理性。例如，平方指数协方差函数是一种常用的协方差函数，它适用于连续输入空间，并且对距离函数的尺度变化具有良好的灵活性。此外，本章还介绍了如何通过已有的协方差函数来构造新的协方差函数。这可能包括将不同的协方差函数以某种方式结合起来，以便更好地适应数据的特定特性。特征函数分析（eigenfunction analysis）是协方差函数研究中的一个重要内容。Mercer定理是一个关键理论，它说明了在某些条件下，可以将协方差函数表示为特征函数和特征值的积分形式。这一理论不仅对理解协方差函数的内在结构具有重要意义，而且在实际应用中也有助于简化计算过程。在高斯过程的研究中，还涉及到了输入域为结构化对象（如字符串和树等）时的协方差函数定义。这扩展了高斯过程的应用范围，使其能够处理更加复杂和非欧几里得类型的数据结构。例如，当输入域是字符串时，需要特殊的协方差函数来捕捉字符串之间的相似性，并在此基础上进行有效的学习和预测。对于高斯过程的分类问题，本章中虽然没有详细展开，但协方差函数同样起着重要作用。在GPC中，协方差函数有助于定义数据点之间的相似性，从而可以用来推断新点的类别。与回归问题类似，GPC同样需要选择合适的协方差函数来刻画输入空间的特性。总结来说，高斯过程中的协方差函数不仅在理论上具有重要意义，还在实际应用中发挥着关键作用。通过选择和设计合适的协方差函数，可以使得高斯过程模型能够更有效地学习数据中的复杂模式，以及对未见数据进行更准确的预测。对于想要深入研究高斯过程的读者，理解协方差函数的本质和性质是不可或缺的一步。

资源推荐

资源详情

资源评论

C. E. Rasmussen & C. K. I. Williams, Gaussian Processes for Machine Learning, the MIT Press, 2006,

ISBN 026218253X.



2006 Massachusetts Institute of Technology. www.GaussianProcess.org/gpml

Chapter 4

Covariance Functions

We have seen that a covariance function is the crucial ingredient in a Gaussian

process predictor, as it encodes our assumptions about the function which we

wish to learn. From a slightly diﬀerent viewpoint it is clear that in supe rvised

learning the notion of similarity between data points is crucial; it is a basic similarity

assumption that points with inputs x which are close are likely to have similar

target values y, and thus training points that are near to a test point should

be informative about the prediction at that point. Under the Gaussian process

view it is the covariance function that deﬁnes nearness or similarity.

An arbitrary function of input pairs x and x

will not, in general, b e a valid valid covariance

functions

covariance function.

The purpose of this chapter is to give examples of some

commonly-used covariance functions and to examine their properties. Section

4.1 deﬁnes a number of basic terms relating to covariance functions. Sec tion 4.2

gives e xamples of stationary, dot-product, and other non-stationary covariance

functions, and also gives some ways to make new ones from old. Section 4.3

intro duces the important topic of eigenfunction analysis of covariance functions,

and states Mercer’s theorem which allows us to express the covariance function

(under certain conditions) in terms of its eigenfunctions and eigenvalues. The

covariance functions given in section 4.2 are valid when the input domain X is

a subset of R

. In section 4.4 we describe ways to deﬁne covariance functions

when the input domain is ove r structured objects such as strings and trees.

4.1 Preliminaries

A stationary covariance function is a function of x − x

. Thus it is invariant stationarity

to translations in the input space.

For example the squared exponential co-

To be a valid covariance function it must be positive semideﬁnite, see eq. (4.2).

In stochastic process theory a process which has constant mean and whose covarianc e

function is invariant to translations is called weakly stationary. A process is strictly sta-

tionary if all of its ﬁnite dimensional distributions are invariant to translations [Papoulis,

1991, sec. 10.1].

C. E. Rasmussen & C. K. I. Williams, Gaussian Processes for Machine Learning, the MIT Press, 2006,

ISBN 026218253X.



2006 Massachusetts Institute of Technology. www.GaussianProcess.org/gpml

80 Covariance Functions

variance function given in equation 2.16 is stationary. If further the covariance

function is a function only of |x − x

| then it is called isotropic; it is thus in-isotropy

variant to all rigid motions. For example the squared e xponential covariance

function given in equation 2.16 is isotropic. As k is now only a function of

r = |x − x

| these are also known as radial basis functions (RBFs).

If a covariance function depends only on x and x

through x · x

we c all itdot product covariance

a dot product covariance function. A simple example is the covariance function

k(x, x

) = σ

+ x · x

which can be obtained from linear regression by putting

N(0, 1) priors on the coeﬃcients of x

(d = 1, . . . , D) and a prior of N(0, σ

)

on the bias (or constant function) 1, see eq. (2.15). Another important example

is the inhomogeneous polynomial kernel k(x, x

) = (σ

+ x · x

)

where p is a

positive integer. Dot product covariance functions are invariant to a rotation

of the coordinates about the origin, but not translations.

A general name for a function k of two arguments mapping a pair of inputskernel

x ∈ X, x

∈ X into R is a kernel. This term arises in the theory of integral

operators, where the operator T

is deﬁned as

f)(x) =

k(x, x

)f(x

) dµ(x

), (4.1)

where µ denotes a measure; see section A.7 for further explanation of this point.

A real kernel is said to be symmetric if k(x, x

) = k(x

, x); clearly covariance

functions must b e symmetric from the deﬁnition.

Given a set of input points {x

|i = 1, . . . , n} we can compute the Gram

matrix K whose entries are K

= k(x

, x

). If k is a covariance function weGram matrix

call the matrix K the covariance matrix.covariance matrix

A real n × n matrix K which satisﬁes Q(v) = v

Kv ≥ 0 for all vectors

v ∈ R

is c alled p ositive semideﬁnite (PSD). If Q(v) = 0 only when v = 0positive semideﬁnite

the matrix is positive deﬁnite. Q(v) is called a quadratic form. A symmetric

matrix is PSD if and only if all of its eigenvalues are non-negative. A Gram

matrix corresponding to a general kernel function need not be PSD, but the

Gram matrix corresponding to a covariance function is PSD.

A kernel is said to be positive semideﬁnite if

k(x, x

)f(x)f(x

) dµ(x) dµ(x

) ≥ 0, (4.2)

for all f ∈ L

(X, µ). Equivalently a kernel function which gives rise to PSD

Gram matrices for any choice of n ∈ N and D is positive sem ideﬁnite. To

see this let f be the weighted sum of delta functions at each x

. Since such

functions are limits of functions in L

(X, µ) eq. (4.2) implies that the Gram

matrix corresponding to any D is PSD.

For a one-dimensional Gaussian proc es s one way to understand the charac-

teristic length-scale of the process (if this exists) is in terms of the number ofupcrossing rate

upcrossings of a level u. Adler [1981, Theorem 4.1.1] states that the expected

Informally speaking, readers will usually be able to substitute dx or p(x)dx for dµ(x).

C. E. Rasmussen & C. K. I. Williams, Gaussian Processes for Machine Learning, the MIT Press, 2006,

ISBN 026218253X.



2006 Massachusetts Institute of Technology. www.GaussianProcess.org/gpml

4.2 Examples of Covariance Functions 81

numbe r of upcrossings E[N

] of the level u on the unit inte rval by a zero-mean,

stationary, almost surely continuous Gaussian process is given by

E[N

] =

2π

−k

(0)

k(0)

exp



−

2k(0)



. (4.3)

If k

(0) does not exist (so that the process is not mean square diﬀerentiable)

then if such a process has a zero at x

then it will almost surely have an inﬁnite

numbe r of zeros in the arbitrarily small interval (x

, x

+δ) [Blake and Lindsey,

1973, p. 303].

4.1.1 Mean Square Continuity and Diﬀerentiability ∗

We now describe mean square continuity and diﬀerentiability of stochastic pro-

cesses, following Adler [1981, sec. 2.2]. Let x

, x

, . . . be a sequence of points

and x

∗

be a ﬁxed point in R

such that |x

− x

∗

| → 0 as k → ∞. Then a

process f (x) is continuous in mean square at x

∗

if E[|f (x

) − f(x

∗

] → 0 as mean square continuity

k → ∞. If this holds for all x

∗

∈ A where A is a subset of R

then f(x) is s aid

to be continuous in mean square (MS) over A. A random ﬁeld is continuous in

mean square at x

∗

if and only if its covariance function k(x, x

) is continuous

at the point x = x

= x

∗

. For stationary covariance functions this reduces

to checking continuity at k(0). Note that MS continuity does not necessarily

imply sample function continuity; for a discussion of sample function continuity

and diﬀerentiability see Adler [1981, ch. 3].

The mean square derivative of f(x) in the ith direction is deﬁned as

∂f(x)

∂x

= l. i. m

h→0

f(x + he

) − f(x)

, (4.4)

when the limit exists, where l.i.m denotes the limit in mean square and e

mean square

diﬀerentiability

is the unit vector in the ith direction. The covariance function of ∂f(x)/∂x

is given by ∂

k(x, x

)/∂x

∂x

. These deﬁnitions can be extended to higher

order derivatives. For stationary processes, if the 2kth-order partial derivative

∂

k(x)/∂

. . . ∂

exists and is ﬁnite at x = 0 then the kth order partial

derivative ∂

f(x)/∂x

. . . x

exists for all x ∈ R

as a mean square limit.

Notice that it is the properties of the kernel k around 0 that determine the

smoothness properties (MS diﬀerentiability) of a s tationary process.

4.2 Examples of Covariance Functions

In this section we consider covariance functions where the input domain X is

a subset of the vector space R

. More general input spaces are considered in

section 4.4. We start in section 4.2.1 with stationary covariance functions, then

consider dot-product covariance functions in section 4.2.2 and other varieties

of non-stationary covariance functions in section 4.2.3. We give an overview

of some commonly used covariance functions in Table 4.1 and in section 4.2.4

C. E. Rasmussen & C. K. I. Williams, Gaussian Processes for Machine Learning, the MIT Press, 2006,

ISBN 026218253X.



2006 Massachusetts Institute of Technology. www.GaussianProcess.org/gpml

82 Covariance Functions

we describe general methods for constructing new kernels from old. There

exist several other good overviews of covariance functions, see e.g. Abrahamsen

[1997].

4.2.1 Stationary Covariance Functions

In this section (and section 4.3) it will be convenient to allow kernels to be a map

from x ∈ X, x

∈ X into C (rather than R). If a zero-mean process f is complex-

valued, then the covariance function is deﬁned as k(x, x

) = E[f(x)f

∗

)],

where

∗

denotes complex conjugation.

A stationary covariance function is a function of τ = x − x

. Sometimes in

this case we will write k as a function of a single argument, i.e. k(τ).

The covariance function of a stationa ry process can be represented as the

Fourier transform of a p ositive ﬁnite measure.

Theorem 4.1 (Bochner’s theorem) A complex-valued function k on R

is theBochner’s theorem

covariance function of a weakly stationary mean square continuous complex-

valued random process on R

if and only if it can be represented as

k(τ ) =

2π is· τ

dµ(s) (4.5)

where µ is a positive ﬁnite measure. 

The statement of Bochner’s theorem is quoted from Stein [1999, p. 24]; a proof

can be found in Gihman and Skorohod [1974, p. 208]. If µ has a density S(s)spectral density

power spectrum

then S is known as the spectral density or pow er spectrum corresponding to k.

The construction given by eq. (4.5) puts non-negative power into each fre-

quency s; this is analogous to the requirement that the prior covariance matrix

on the weights in equation 2.4 b e non-negative deﬁnite.

In the case that the spectral density S(s) exists, the covariance function and

the spectral density are Fourier duals of e ach other as shown in eq. (4.6);

this

is known as the Wiener-Khintchine theorem, see, e.g. Chatﬁeld [1989]

k(τ ) =

S(s)e

2π is· τ

ds, S(s) =

k(τ )e

−2π is· τ

dτ . (4.6)

Notice that the variance of the process is k(0) =

S(s) ds so the power spectrum

must be integrable to deﬁne a valid Gaussian proces s.

To gain some intuition for the deﬁnition of the p ower spectrum given in

eq. (4.6) it is important to realize that the complex exponentials e

2π is· x

are

eigenfunctions of a stationary kernel with resp e ct to Lebesgue measure (see

section 4.3 for further details). Thus S(s) is, loosely speaking, the amount of

power allocated on average to the eigenfunction e

2π is· x

with frequency s. S(s)

must eventually decay suﬃciently fast as |s| → ∞ so that it is integrable; the

See Appendix A.8 for details of Fourier transforms.

剩余25页未读，继续阅读

评论收藏

内容反馈

PassionalP

粉丝: 0

Gaussian Processes for Machine Learning

Gaussian process for machine learning

Gaussian processes

gaussian process

Gaussian程序

Gaussian Processes for Machine Learning机器学习的高斯过程

电子书 Gaussian Processes for Machine Learning

GaussianProcessesforMachineLearning

高斯过程回归源码（含实验数据）

报销明细-ysk_zhangdan_baoxiao_machinelearning_

matlab集成c代码-Gaussian-Processes-for-Machine-Learning:Williams[2006年]的《用于

Gaussian Process for Machine Learning 一书的配套matlab代码

advances in gaussian processes

高斯过程机器学习在膨胀土等级分类中的应用.pdf

高斯过程在机器学习中的应用.doc

利用GPML V4.2工具箱实现高斯过程回归（Gaussian process regression，GPR）的多变量数据预测

Bayesian Reasoning and Machine Learning

gpml.zip_gpml_多输入多输出_多高斯过程_高斯过程_高斯过程建模

Bishop Pattern Recognition and Machine Learning

基于高斯过程机器学习的注塑过程建模及工艺参数优化设计.pdf

机器学习中的高斯过程

用正则表达式过滤日志

使用spark在阿里云平台跑任务报错no space left on device

最新资源