Learning to Optimize with Guarantees: A Complete Characterization of Linearly Convergent Algorithms

Guarantees of fast convergence are crucial whenever optimization must be executed under tight computational budgets, as in large-scale machine learning (ML) or real-time model predictive control (MPC). Worst-case linear convergence guarantees have been developed for iterative optimization algorithms over several classes of objective functions, whose structure—e.g., strong convexity, smoothness, or gradient dominance—can be exploited by first-order schemes such as standard gradient descent and Nesterov’s accelerated method [22]. A growing body of work leverages the analogy between worst-case convergence rates and robust-control techniques such as integral quadratic constraints (IQCs), leading to characterizations of accelerated algorithms with provably optimal rates across families of convex functions [15, 27, 29].

Worst-case rate guarantees establish a baseline performance in terms of the number of iterations required to achieve a certain level of precision. However, the performance of an algorithm in a specific application does not depend solely on its worst-case convergence rate. First, there exist fundamental trade-offs between the speed of convergence and the robustness of algorithmic behavior; see, for instance, the speed/covariance trade-off for accelerated methods in strongly convex optimization analyzed in [21]. This raises the question of how to appropriately define algorithm performance. A second challenge is that scenarios encountered in applications rarely span the entirety of the space of problem instances for which the worst-case guarantee is tight, resulting in overly conservative average-case performance. This introduces another trade-off: how to tailor an algorithm performance to specific instances without compromising the original uniform guarantees. A prime example of such a situation is MPC [20], where the optimization problems to be solved online often share the same objective and system dynamics constraints, and only differ in the initial state. In such cases, a solver tailored to this sub-family of problems could converge in significantly fewer iterations than a general-purpose algorithm tuned for optimal worst-case convergence rates.

The learning to optimize (L2O) literature addresses the challenge of adopting user-defined performance metrics beyond mere convergence rates and designs algorithms that are tailored to such metrics using ML. For instance, [4] proposes an algorithm performance metric that balances convergence speed with solution precision, and accordingly designs neural network (NN) update rules. However, general-purpose NN update rules come with no guarantees. Convergence with learned updates has been addressed through possibly conservative safeguarding mechanisms [11], or by limiting the use of ML components to learning initializations [25] or to hyperparameter tuning [12] of classical algorithms. These approaches demonstrate performance exceeding that of state-of-the-art classical algorithms upon training and inherit their convergence guarantees—yet the reliance on classical update rules inherently limits the expressivity of these learning-based design methods.

Beyond the optimal tuning of classical algorithms, another line of research seeks to use ML to design entirely new convergent update rules, aiming to discover application-specific shortcuts unknown to classical update rules. This has been achieved by taking simple gradient descent as a baseline and enhancing it through learned optimal deviations from such gradient-based updates. The work [19] characterizes the class of all and only those deviation functions that ensure convergence to stationary points in nonconvex, unconstrained smooth optimization, enabling learned optimization for user-defined performance metrics and outperforming finely tuned Adam [14] in NN training. The work [5] uses deep learning to train deviations from gradient descent and saturates these updates with the norm of measured gradients, ensuring convergence for composite convex optimization. The numerical studies of [19, 5] empirically demonstrate that convergence rates superior to those of classical algorithms can be achieved through training on gradient descent perturbations.

Our main goal is to address a question that has remained open in the literature on L2O. Given any algorithm with guaranteed linear convergence over a class of optimization problems, how can we improve its average-case performance over a subclass of problems of interest without sacrificing its guarantees over the entire class? A theoretical study of these trade-offs is important towards making learned optimization a standard and reliable component of linearly convergent algorithm design.

The goal of characterizing all convergent optimization algorithms for a class of problems is analogous to the canonical control-theoretic problem of parameterizing all stabilizing controllers for a given system, initially studied for linear systems [30, 3] and more recently for nonlinear and in particular neural-network systems (see, e.g. [28, 8, 18]). Structurally, our approach is analogous in that it augments an existing convergent algorithm with a decaying perturbation, whereas the Youla parameterization augments an existing stabilizing controller with a stable perturbation. Connection can also be drawn between our work and the recently-studied input-to-state stability properties of gradient algorithms applied to optimization problems satisfying the Polyak–Łojasiewicz (PL) condition [6]. In this paper we study a wider class of optimization algorithms and provide a characterization of perturbations achieving precise bounds on convergence rates.

Contributions: Given any existing optimization algorithm that achieves linear convergence to a set of fixed points at a specified rate—henceforth the baseline algorithm—our main contributions are as follows. First, we characterize conditions on the baseline algorithm under which adding exponentially decaying perturbations preserves the same linear convergence rate, up to a higher-order polynomial term. These conditions identify fundamental trade-offs between the frequency of perturbations and their worst-case impact on the linear convergence rate. Second, we establish a completeness result for linearly convergent optimization: every update rule that converges linearly at a given rate can be written as the sum of the baseline algorithm and a suitably designed exponentially decaying augmentation of its update rule that preserves linear convergence. Finally, we instantiate our results for augmenting state-of-the-art linearly convergent algorithms, notably 1) Nesterov’s accelerated gradient (NAG) [22] method for strongly convex and smooth optimization, 2) gradient descent for classes of nonconvex Polyak–Łojasiewicz (PL) functions, and 3) projected gradient descent for convex optimization with polyhedral feasible sets. Numerical examples showcase the potential of learned linearly convergent optimization in augmenting the performance of classical algorithms under tight iteration budgets.

Notation: The set of all sequences $\mathbf{x}=(x_{0},x_{1},x_{2},\ldots)$ where $x_{t}\in\mathbb{R}^{n}$ for all $t\in\mathbb{N}$ is denoted as $\ell^{n}$. For a function $g:\mathbb{R}^{n}\rightarrow\mathbb{R}^{m}$, we write $g\left(\mathbf{x}\right)=(g(x_{0}),g(x_{1}),\ldots)\in\ell^{m}$. For $m\in\mathbb{N}$, we use $\mathcal{P}_{m}(x)$ to denote the set of positive and monotonically non-decreasing polynomials of degree at most $m$ in the variable $x$ with $p(1)\geq 1$. We define the set of fixed points of an operator $\pi$, assumed non-empty, as $\mathtt{Fix}_{\pi}$. For $m\in\mathbb{N}$ and $\gamma\in(0,1)$, we denote by $\ell_{exp}(m,\gamma)$ the class of signals $\mathbf{x}$ for which there exists a polynomial $p_{m}(t)\in\mathcal{P}_{m}(t)$ such that $|x_{t}|\leq p_{m}(t)\gamma^{t}$, where $|\cdot|$ denotes any vector norm. We write $\mathbb{I}_{S}$ for the indicator function of a set $S$ ($0$ if $x\in S$, $+\infty$ otherwise). The distance from $x$ to a set $S$ is defined as $\mathrm{dist}(x,S)=\inf_{y\in S}|x-y|$. For $u,v\in\mathbb{R}^{n}$, $u\leq v$ indicates the element‐wise inequality.

We consider composite optimization problems of the form

where $x\in\mathbb{R}^{d}$ is the decision variable, $f:\mathbb{R}^{d}\to\mathbb{R}$ is proper and $\beta$-smooth, and $g:\mathbb{R}^{d}\to\mathbb{R}\cup\{+\infty\}$ is convex, proper, and lower semi-continuous, but potentially nonsmooth. We let $F(x)=f(x)+g(x)$ for brevity, and we assume that the set of optimizers $\mathcal{X}^{\star}=\operatorname*{arg\,min}_{x\in\mathbb{R}^{d}}~F(x)$ is non-empty. In particular, we note that (1) subsumes constrained optimization problems of the form

where $f_{0}:\mathbb{R}^{d}\to\mathbb{R}$ is $\beta$-smooth, and each function $f_{i}:\mathbb{R}^{d}\to\mathbb{R}$ with $i\in[1,M]$ defines a non-empty convex feasibility set $\mathcal{X}_{i}\subseteq\mathbb{R}^{d}$. In fact, one can rewrite (2) as an instance of (1) letting $f(x)=f_{0}(x)$ and $g(x)=\max_{i\in[1,M]}~\mathbb{I}_{\mathcal{X}_{i}}(x)$.

A standard method to solve problem (1) is to analytically construct iterations of the form:

where $\xi_{t}\in\mathbb{R}^{n}$ is the state variable, the decision $x_{t}\in\mathbb{R}^{d}$ is the output variable, and the operator $\pi$ is designed so that its set of fixed points $\mathtt{Fix}_{\pi}$—the set of points $\xi^{\star}$ such that $\pi(F,\xi^{\star})=\xi^{\star}$—is related to $\mathcal{X}^{\star}$ through $\phi$; that is, a point $x^{\star}\in\mathcal{X}^{\star}$ can be reconstructed from a point $\xi^{\star}\in\mathtt{Fix}_{\pi}$ as per $x^{\star}=\phi(F,\xi^{\star})$.

A key metric for the performance of algorithms (3) when applied to a class of problems $F\in\mathcal{F}$ is how fast they converge to $\mathtt{Fix}_{\pi}$. Classical optimization algorithms often come with convergence guarantees that hold for the worst-case instance of $F\in\mathcal{F}$. However, optimal control methods such as MPC require efficiently finding solutions to the instances of (1) that are encountered during deployment, where the objective $F$ is drawn from a specific distribution $\mathcal{D}_{\mathcal{F}}$ over the class $\mathcal{F}$. Motivated as such, in this work we investigate the following question.

Given a set of problem instances $F\sim\mathcal{D}_{\mathcal{F}}$ and a baseline algorithm $\pi$ to solve (1), how can we improve its average-case performance over $\mathcal{D}_{\mathcal{F}}$, while retaining worst-case convergence guarantees over the entire class $\mathcal{F}$?

In particular, this paper characterizes algorithms $\bm{\nu}$ that achieve linear convergence to $\mathtt{Fix}_{\pi}$ for classes of functions $\mathcal{F}$. We will showcase how to leverage such characterization for learning-based algorithm design in Section 4.

To enhance the average-case performance of a baseline algorithm $\pi$ on problem instances $F\sim\mathcal{D}_{\mathcal{F}}$, we aim to design augmentations $v_{t}\in\mathbb{R}^{n}$ to its update rule that do not jeopardize its convergence guarantees. Specifically, we will consider augmented update rules defined by the iterations:

and we will establish conditions ensuring that $\bm{\nu}$ is linearly convergent as per (4).

In this section, we establish our main results on how introducing an augmentation term $v_{t}$ affects the worst-case linear convergence guarantees of a given baseline optimization algorithm $\pi$. We first abstract away from the specific form of $\pi$ and the class of functions $\mathcal{F}$ it is designed to optimize; we only assume that $\pi$ is a linearly convergent fixed-point algorithm as per Definition 1. In Section 3.1, we present corollaries that reveal several classes of problems (1) and corresponding baseline algorithms $\pi$ that are compatible with our framework.

Consider a baseline algorithm $\pi\in\widehat{\mathtt{pExp}}^{\pi}_{\mathcal{F}}(\gamma)$ achieving linear convergence as per Definition 1. The property (4) implies that the signal $\operatorname{dist}(\xi_{t},\mathtt{Fix}_{\pi})$ decays exponentially up to a polynomial factor $p(t)$, for any initial condition $\xi_{0}\in\mathbb{R}^{n}$ and any objective $F\in\mathcal{F}$. We now characterize to what extent injecting exponentially decaying signals $v_{t}$ in the iterates of (5) can deteriorate the convergence guarantee of $\pi$.

We report the proof of Theorem 1 in Appendix A. Theorem 1 establishes a trade-off between how often we inject an exponentially decaying perturbation—as measured by $N\in\mathbb{N}$—and the degradation of the convergence rate. In particular, if $\pi\in\mathtt{Exp}^{\pi}_{\mathcal{F}}(\gamma)$, then $p(1)=1$ by definition and we observe that the asymptotic rate $\gamma$ does not change even with $N=1$, as only the order of the polynomial factor in (4) is affected. This means that one can apply perturbation $\mathbf{v}\in\ell_{exp}(m,\gamma)$ at every time instant as per (5) without deteriorating the asymptotic convergence rate. For the general case where $\pi\in\mathtt{pExp}^{\pi}_{\mathcal{F}}(m,\gamma$), instead, the convergence rate increases at most to the value $\sqrt[N]{p(N)}\gamma\in(\gamma,1)$. As expected, when $N$ tends to infinity, we recover the original rate of the baseline algorithm because $\lim_{N\to\infty}\sqrt[N]{p(N)}=1$; this corresponds to the case $v_{t}=0$ for all $t$.

It remains crucial to understand how large is the class of linearly convergent algorithms $\xi_{t}=\nu_{t}(F,\xi_{t:0})$ that can be achieved by perturbing a baseline algorithm $\pi$ with an exponentially decaying learned update $v_{t}$ as per (5). Our next result establishes conditions on the baseline algorithm $\pi$ that guarantee that any algorithm in $\mathtt{pExp}^{\pi}_{\mathcal{F}}(m,\gamma$) can be represented—provided that such target algorithm satisfies the following regularity condition.

In other words, the definition above excludes pathological cases of linearly convergent algorithms that can cycle indefinitely among the points in $\mathtt{Fix}_{\pi}$ even when $\operatorname{dist}(\xi_{t},\mathtt{Fix}_{\pi})=0$. As an example, consider the case where the objective $F$ is constant, the baseline algorithm is $\xi_{t+1}=\pi(F,\xi_{t})=\xi_{t}$ with $\xi_{0}\neq 0$, and $\mathtt{Fix}_{\pi}=\mathbb{R}^{n}$. In this case, the algorithm $\xi_{t+1}=\nu_{t}(F,\xi_{t:0})=-\xi_{t}$ converges to $\mathtt{Fix}_{\pi}$ yet is not regular according to Definition 2. In fact, by definition of $\bm{\nu}$, we have $u_{t}=-2\xi_{0}\neq 0$ when $t$ is even and $u_{t}=2\xi_{0}\neq 0$ when $t$ is odd, hence the resulting $\mathbf{u}$ is not exponentially decaying.

We are ready to present our completeness result.

A few remarks are in order. First, the completeness property of Theorem 2 is key in the context of automating the design of augmented algorithms, e.g., by learning from sampled problems, as it implies that (5) encompasses all regular linearly convergent algorithms $\bm{\sigma}\in\mathtt{pExp}^{\pi}_{\mathcal{F}}(m,\gamma$). Second, when we learn an augmentation term $\mathbf{v}\in\ell_{exp}(m,\gamma)$ by searching over the entire space of exponentially decaying updates, it is desirable that the baseline algorithm satisfies the stronger condition $\pi\in\mathtt{Exp}^{\pi}_{\mathcal{F}}(\gamma)$. This is because, if $\pi\in\mathtt{pExp}^{\pi}_{\mathcal{F}}(m,\gamma~)\setminus\mathtt{Exp}^{\pi}_{\mathcal{F}}(\gamma)$, Theorem 1 only guarantees linear convergence—with the worsened rate $\sqrt[N]{p(N)}\,\gamma$—for those $\mathbf{v}$ chosen exactly as in (6). In other words, without $\pi\in\mathtt{Exp}^{\pi}_{\mathcal{F}}(\gamma)$, most perturbations in $\ell_{exp}(m,\gamma)$ may not preserve linear convergence. Third, the assumption that $\pi(F,\xi)$ is Lipschitz continuous in $\xi$ is mild; we will show in the following that this condition is satisfied for baseline algorithms widely used for convex and composite optimization.

The rest of this section is dedicated to establishing how Theorem 1 and Theorem 2 can be used to augment existing solvers for composite optimization problems in the form (1) drawn from specific classes $\mathcal{F}$.

In this section, we illustrate a natural application of our characterization of linearly convergent algorithms to design new update rules with improved average‑case performance relative to a baseline linearly convergent algorithm while preserving its worst‑case convergence rate. We begin by formulating a meta‑optimization problem over the augmentation sequence $v_{t}(F,\xi_{t:0})$, where we aim at minimizing the expected cost incurred by the augmented algorithm under a given data distribution while enforcing a target worst‑case convergence rate. Beyond the special case in which the algorithmic costs are quadratic in the iterates, the augmentation sequences are linear functions, and the objective functions $F\in\mathcal{F}$ are quadratic, the meta‑optimization is nonconvex. This motivates parameterizing $v_{t}(F,\xi_{t:0})$ as an exponentially decaying NN update and learning its parameters via empirical algorithmic cost minimization. We conclude by discussing the significant speed-ups achieved by learned augmentations through examples drawn from linear regression and linear quadratic MPC.²²2The Python source code that reproduces our experiments is publicly available at https://github.com/andrea-martin/LinearlyConvergentL2O.