Stable concurrent learning and control of dynamical systems is the subject of adaptive control. Despite being an established field with many practical applications and a rich theory, much of the development in adaptive control for nonlinear systems revolves around a few key algorithms. By exploiting strong connections between classical adaptive nonlinear control techniques and recent progress in optimization and machine learning, we show that there exists considerable untapped potential in algorithm development for both adaptive nonlinear control and adaptive dynamics prediction. We begin by introducing first-order adaptation laws inspired by natural gradient descent and mirror descent. We prove that when there are multiple dynamics consistent with the data, these non-Euclidean adaptation laws implicitly regularize the learned model. Local geometry imposed during learning thus may be used to select parameter vectors—out of the many that will achieve perfect tracking or prediction—for desired properties such as sparsity. We apply this result to regularized dynamics predictor and observer design, and as concrete examples, we consider Hamiltonian systems, Lagrangian systems, and recurrent neural networks. We subsequently develop a variational formalism based on the Bregman Lagrangian. We show that its Euler Lagrange equations lead to natural gradient and mirror descent-like adaptation laws with momentum, and we recover their first-order analogues in the infinite friction limit. We illustrate our analyses with simulations demonstrating our theoretical results.
Adaptation is an online learning problem concerned with control or prediction of the dynamics of an unknown nonlinear system. This task is accomplished by constructing an approximation to the true dynamics through the online adjustment of a vector of parameter estimates under the assumption that there exists a fixed vector of parameters that globally fits the dynamics. The overarching goal is provably safe, stable, and concurrent learning and control of nonlinear dynamical systems.
Adaptive control theory is a mature field, and many results exist tailored to specific system structures (Ioannou & Sun, 2012; Narendra & Annaswamy, 2005; Slotine & Li, 1991). An adaptive control algorithm typically consists of a parameter estimator coupled in feedback to the controlled system, and the estimator is often strongly inspired by gradient-based optimization algorithms. A significant difference between standard optimization algorithms and adaptive control algorithms is that the parameter estimator must not only converge to a set of parameters that leads to perfect tracking of the desired trajectory, but the system must remain stable throughout the process. The additional requirement of stability prevents the immediate application of optimization algorithms as adaptive control algorithms, and stability must be proved by jointly analyzing the closed-loop system and estimator.
Significant progress has been made in adaptive control even for nonlinear systems in the linearly parameterized setting, where the dynamics approximation is of the form for some known regressor matrix and vector of parameter estimates . Examples include the adaptive robot trajectory controller of Slotine and Li (1987) and the neural network-based controller of Sanner and Slotine (1992), which employs a mathematical expansion in physical nonlinear basis functions to uniformly approximate the unknown dynamics.
Unlike its linear counterpart, solutions to the adaptive control problem in the general nonlinearly parameterized setting have remained elusive. Intuitively, this is unsurprising: guarantees for gradient-based optimization algorithms typically rely on convexity, with a few notable exceptions such as the Polyak-Lojasiewicz condition (Polyak, 1963). In the linearly parameterized setting, the underlying optimization problem will be convex. When the parameters appear nonlinearly, the problem is in general nonconvex and difficult to provide guarantees for.
In this work, we provide new provably globally convergent algorithms for both the linearly and nonlinearly parameterized adaptive control problems, along with new insight into existing adaptive control algorithms for the linearly parameterized setting. Our results for nonlinearly parameterized systems are valid under the monotonicity assumptions of Tyukin, Prokhorov, and van Leeuwen (2007) and the convexity assumptions of Fradkov (1980). These monotonicity assumptions are equivalent to those commonly satisfied by generalized linear models in statistics (Kakade, Kalai, Kanade, & Shamir, 2011).
1.1 Description of Primary Contributions
Our contributions can be categorized into two main advances.
We further develop a class of natural gradient and mirror descent-like algorithms that have recently appeared in the literature in the context of physically consistent inertial parameter learning in robotics (Lee, Kwon, & Park, 2018) and geodesically convex optimization (Wensing & Slotine, 2020). We prove that these algorithms implicitly regularize the learned system model in both the linearly parameterized and nonlinearly parameterized settings.
We construct a general class of higher-order in-time adaptive control algorithms that incorporate momentum into existing adaptation laws. We prove that our new momentum algorithms are stable and globally convergent for both linearly parameterized and nonlinearly parameterized systems.
Unlike standard problems in optimization and machine learning, explicit regularization terms cannot be naively added to adaptive control algorithms without affecting stability and performance. Our approach enables a provably stable and globally convergent implementation of regularization in adaptive control. We demonstrate the utility of these results through examples in the context of dynamics prediction, such as sparse estimation of a physical system's Hamiltonian or Lagrangian function, and estimating the weights of a continuous-time recurrent neural network model.
It is well known in adaptive control that the true parameters are only recovered when the desired trajectory satisfies a strong condition known as persistent excitation (Narendra & Annaswamy, 2005; Slotine & Li, 1991). In general, an adaptation law need only find parameters that enable perfect tracking, and very little is known about what parameters are found when the estimator converges without persistent excitation. Our proof of implicit regularization provides an answer and shows that standard Euclidean adaptation laws lead to parameters of minimum norm.
For the second contribution, we utilize the Bregman Lagrangian (Betancourt, Jordan, & Wilson, 2018; Wibisono, Wilson, & Jordan, 2016; Wilson, Recht, & Jordan, 2016) in tandem with the velocity gradient methodology (Andrievskii, Stotskii, & Fradkov, 1988; Fradkov, 1980, 1986; Fradkov, Miroshnik, & Nikiforov, 1999) to define a general formalism that generates higher-order in-time (Morse, 1992) velocity gradient algorithms. Our key insight is that the velocity gradient formalism provides an optimization-like framework that encompasses many well-known adaptive control algorithms and that the velocity gradient “loss function” can be placed directly in the Bregman Lagrangian.
1.2 Summary of Related Work
Our work continues in a long-standing tradition that utilizes a continuous-time view to analyze optimization algorithms, and here we consider a nonexhaustive list. Diakonikolas and Jordan (2019) develop momentum algorithms from the perspective of Hamiltonian dynamics, while Maddison, Paulin, Teh, O'Donoghue, and Doucet (2018) use Hamiltonian dynamics to prove linear convergence of new optimization algorithms without strong convexity. Muehlebach and Jordan (2019, 2020) study momentum algorithms from the viewpoint of dynamical systems and control. Boffi and Slotine (2020) analyze distributed stochastic gradient descent algorithms via dynamical systems and nonlinear contraction theory. Su, Boyd, and Candès (2016) provide an intuitive justification for Nesterov's accelerated gradient method (Nesterov, 1983) through a limiting differential equation. Continuous-time differential equations were used as early as 1964 by Polyak to derive the classical momentum or “heavy ball” optimization method (Polyak, 1964). In all cases, continuous time often affords simpler proofs, and it enables the application of physical intuition when reasoning about optimization algorithms. Given the gradient-based nature of many adaptive control algorithms, the continuous-time view of optimization provides a natural bridge from modern optimization to modern adaptive control.
Despite the simplicity of proofs in continuous time, finding a discretization that provably retains the convergence rates of a given differential equation is challenging. In a significant advance, Wibisono et al. (2016) showed that many accelerated methods in optimization can be derived via a variational point of view from a single mathematical object known as the Bregman Lagrangian. The Bregman Lagrangian leads to second-order mass-spring-damper-like dynamics, and careful discretization provides discrete-time algorithms such as Nesterov's celebrated accelerated gradient method (Nesterov, 1983). We similarly use the Bregman Lagrangian to generate our new adaptive control algorithms, which generalize and extend a recently developed algorithm due to Gaudio, Gibson, Annaswamy, and Bolender (2019).
Progress has been made in nonlinearly parameterized adaptive control in a number specific cases. Annaswamy, Skantze, and Loh (1998), Ai-Poh Loh, Annaswamy, and Skantze (1999), and Kojić and Annaswamy (2002) develop stable adaptive control laws for convex and concave parameterizations, though they may be overly conservative and require solving optimization problems at each time step. Astolfi and Ortega (2003) and Liu, Ortega, Su, and Chu (2010) develop the immersion and invariance (I&I) approach, and prove global convergence if a certain monotone function can be constructed. Ortega, Gromov, Nuño, Pyrkin, and Romero (2019) use a similar approach for system identification. Tyukin et al. (2007) consider dynamical systems satisfying a monotonicity assumption that is essentially identical to conditions required for learning generalized linear models in machine learning and statistics (Goel & Klivans, 2017; Goel, Klivans, & Meka, 2018; Kakade et al., 2011), and develop provably stable adaptive control algorithms for nonlinearly parameterized systems in this setting. Fradkov (1980), Andrievskii et al. (1988), Fradkov (1986), and Fradkov et al. (1999) develop the velocity gradient methodology, an optimization-like framework for adaptive control that allows for provably global convergence under a convexity assumption. As mentioned in section 1.1, this framework, in tandem with the Bregman Lagrangian, is central to our development of momentum algorithms.
Our work is strongly related to and inspired by a line of recent work that analyzes the implicit bias of optimization algorithms in machine learning. Soudry, Hoffer, Nacson, Gunasekar, and Srebro (2018) and Gunasekar, Lee, Soudry, and Srebro (2018b, 2018a) characterize implicit regularization of common gradient-based optimization algorithms such as gradient descent with and without momentum, as well as natural gradient descent and mirror descent in the settings of regression and classification. Azizan, Lale, and Hassibi (2019) and Azizan and Hassibi (2019) arrive at similar results via a different derivation based on results from control. Similarly, Belkin, Hsu, Ma, and Mandal (2019) consider the importance of implicit regularization in the context of the successes of deep learning. Our results are the adaptive control analogues of those presented in these papers.
1.3 Paper Outline
The paper is organized as follows. In section 2, we present some required mathematical background on direct adaptive control in the linearly and nonlinearly parameterized settings. In section 3 we analyze the implicit bias of adaptive control algorithms, while in section 4 we consider general observer and dynamics predictor design, Hamiltonian dynamics prediction, control of Lagrangian systems, and estimation of recurrent neural networks. In section 5 we provide background for our development of momentum algorithms, including a review of the velocity gradient formalism (section 5.1) and the Bregman Lagrangian (section 5.2). In section 6 we present adaptive control algorithms with momentum, and we extend them to the non-Euclidean setting in section 7. We illustrate our results via simulation in section 8, and we conclude with some closing remarks and future directions in section 9.
2 Direct Adaptive Control
In this section, we provide an introduction to direct adaptive control for both linearly parameterized and nonlinearly parameterized systems, along with a description of some natural gradient-like adaptive laws that have appeared in the recent literature.
2.1 Linearly Parameterized Dynamics
The fundamental utility of defining the variable is its conversion of the adaptive control problem for the th-order system, equation 2.1, to an adaptive control problem for the first-order system, equation 2.6. Our results may be simply extended to other error models (Ai-Poh Loh et al., 1999; Narendra & Annaswamy, 2005) of the form 2.6, or error models with similar input-output guarantees, as summarized by lemma 72.
We will use to denote the equivalent first-order system to (2.1), , where and .
In this work, we make a mild additional assumption that simplifies some of the proofs.
The dynamics is locally bounded in and uniformly in . That is, if and , then
2.2 Nonlinearly Parameterized Dynamics
While a difficult problem in general, significant progress has been made for the nonlinearly parameterized adaptive control problem under the assumption of monotonicity, and several notions of monotonicity have appeared in the literature (Astolfi & Ortega, 2003; Liu et al., 2010; Ortega et al., 2019; Tyukin, 2011; Tyukin et al., 2007). We consider one such notion as presented by Tyukin et al. (2007), which is captured in the following assumption.
2.3 The Bregman Divergence and Natural Adaptation Laws
The choice of enables the design of adaptation algorithms that respect physical Riemannian constraints (Lee, Wensing, & Park, 2020) obeyed by the true parameters, as in the estimation of mass properties in robotics (Wensing, Kim, & Slotine, 2018). Similarly, it allows one to introduce a priori bounds on parameter estimates without resorting to parameter projection techniques by choosing to be a -barrier function (Wensing & Slotine, 2020). In section 3.1, we further prove that the choice of implicitly regularizes the learned system model.
with strongly convex and ensures that over the maximal interval of existence of for nonlinearly parameterized systems categorized by assumption 2. The proof is identical to that of Tyukin et al. (2007). The implementation of this algorithm in PI form will be described in remark 13 and is based on a correspondence between mirror descent and natural gradient descent in continuous time. This algorithm can be seen as the adaptive control equivalent of a mirror descent or natural gradient extension of the GLMTron of Kakade et al. (2011), and this correspondence will be considered in greater detail in section 6.
In the linearly parameterized setting, rather than the Lyapunov-like function , the Lyapunov-like function may be used for any positive-definite matrix . This shows stability of the adaptation law , where the choice of the matrix offers an additional design flexibility.
In some practical applications, as in adaptive robot control, the estimated parameters may correspond to physical constants. In this case, the weighted parameter estimation error term not only provides additional design flexibility through the elements of in the adaptation law, but is necessary for physical consistency of units. Indeed, the usual Lyapunov-like function shows that must be chosen so that the parameter estimation error term has the same units as the tracking error term . Similar considerations apply when replacing this standard parameter estimation error term with the Bregman divergence , which has the same units as . In this case, must be chosen to have the same units as the tracking error term, for example, by introducing a diagonal matrix of constants to ensure consistent dimensions.
3 Natural Gradient Adaptation and Implicit Regularization
In this section, we show that the “natural” adaptation algorithms of the previous section implicitly regularize the learned system model.
3.1 Implicit Regularization and Adaptive Control
With deep networks as the predominant example, modern machine learning often considers highly overparameterized models that are capable of interpolating the training data (achieving zero error on the training set) while still generalizing well to unseen examples. The classical principles of statistical learning theory emphasize a trade-off between generalization performance and model capacity, and predict that in the highly overparameterized regime, generalization performance should be poor due to a tendency of the model to fit noise in the training data. Nevertheless, empirical evidence indicates that deep networks and other modern machine learning models do not obey classical statistical learning wisdom (Belkin et al., 2019) and can even generalize with significant label noise (Zhang, Bengio, Hardt, Recht, & Vinyals, 2016).
More surprisingly, the ability to simultaneously fit label noise in the training data yet generalize to new examples has been observed in overparameterized linear models (Bartlett, Long, Lugosi, & Tsigler, 2020; Muthukumar, Vodrahalli, & Sahai, 2019). A possible explanation for the ability of highly overparameterized models to generalize when optimized using simple first-order algorithms is their implicit bias—that is, the tendency of an algorithm to converge to a particular (e.g., minimum norm) solution when there are many that interpolate the training data (Azizan & Hassibi, 2019; Azizan et al., 2019; Gunasekar et al., 2018a, 2018b; Soudry et al., 2018).
In adaptive control, the possibility of there being many possible parameter vectors that lead to zero tracking error is not unique to the overparameterized case. Unless the trajectory is persistently exciting2 (Narendra & Annaswamy, 2005; Slotine & Li, 1991), it is well known that will not converge to the true parameters in general. Depending on the complexity of the trajectory, there may even be many solutions in the underparameterized case where . To achieve perfect tracking, the adaptation algorithm need only fit the unknown dynamics along the trajectory rather than the whole state space, so that the effective number of parameters may be less than .
The wealth of possible solutions in the linearly parameterized case is captured by the time-dependent null space of : when , we can conclude that , and hence that where for all . This observation also highlights that any element of the null space may be added to the parameter estimates without affecting the value of .3 In the overparameterized case when , the set of parameters that achieve zero tracking error is not unique regardless of the complexity of the desired trajectory. By deriving a continuous-time extension of a recent proof of the implicit bias of mirror descent algorithms (Azizan & Hassibi, 2019; Azizan et al., 2019), we now show that the natural adaptive laws of the previous section implicitly regularize . This proof of implicit regularization provides an answer to the question, with infinitely many parameter vectors that achieve zero tracking error, which does adaptation choose?
The assumptions of proposition 1 provide a setting where theoretical insight may be gained into the implicit regularization of adaptive control algorithms, but they are stronger than needed. In general, the parameters found by an adaptive controller need not converge to a constant despite the fact that .4 Similarly, even in the case that the parameters converge, it is not strictly required that along the entire trajectory, as this condition is satisfied asymptotically. Numerical simulations in section 8 will demonstrate the implicit regularization of parameters found by adaptive control along the entire trajectory.
We may make a similar claim in the nonlinearly parameterized setting captured by assumption 2. To do so, we require an additional assumption.
For any vector of parameters and the true parameters , implies that .
For the class of systems 2.11, a sufficient condition for assumption 3 is that and that the map is invertible at every . We may now state our implicit regularization result for nonlinearly parameterized systems.
Algorithm 3.4 must be implemented in PI form due to the appearance of . The use of the PI form in , equations 2.13 to 2.16, is complicated by the presence of the inverse Hessian of . To implement equation 2.12, the Euclidean variant may be implemented through the usual PI form for an auxiliary variable , and then the controller parameters may be computed by inverting the gradient of , . This follows by the equivalence of mirror descent and natural gradient descent in continuous time. Concretely, the identity shows that is equivalent to . The auxiliary variable can then be identified with .
If the inverse gradient of is unknown but is chosen to be strongly convex, the contracting (Lohmiller & Slotine, 1998) dynamics with will converge to a ball around with radius set by where is the strong convexity parameter. By choosing so that this contracting dynamics is fast on the timescale of adaptation, will represent a good approximation of the instantaneous .
Our results highlight, through the equivalence of their continuous-time limits, that both mirror descent–like and natural gradient-like adaptive laws impose implicit regularization. This observation extends recent results on the implicit regularization of mirror descent (Azizan & Hassibi, 2019; Azizan et al., 2019) to natural gradient descent and, furthermore, applies to linearly parameterized and generalized linearly parameterized models in machine learning, not just in the context of adaptive control. This has previously been noted in Gunasekar et al. (2018a), where it was discussed that in discrete time, natural gradient descent only approximately imposes implicit regularization due to discretization errors.
Propositions 1 and 2 demonstrate for the first time the implicit bias of adaptive control algorithms. In doing so, they identify an additional design choice that may be exploited for the application of interest. Proposition 1 implies that the Slotine and Li controller, when initialized with the parameters at , finds the interpolating parameter vector of minimum norm. Other norms, such as the , , for arbitrary , or group norms will find alternative parameter vectors that may have desirable properties such as sparsity.5 The usual Euclidean geometry–based adaptive laws can be seen as a form of ridge regression, while imposed , and simultaneously, or regularization through the choice of can be seen as the adaptive control equivalents of LASSO (Tibshirani, 1996) or compressed sensing, elastic net, and bridge regression, respectively. In the context of adaptive control, this notion of implicit regularization is particularly interesting, as typical regularization terms such as and penalties cannot in general be added to the adaptation law directly without affecting the stability and performance of the algorithm.
Following remark 8, if is chosen as the th power of a norm, in practical applications it is necessary to include a matrix to ensure that has the same units as the tracking error component of the Lyapunov function. For example, if , then may be chosen as for consistency of units where is a gain matrix tuned for the standard adaptive law . In addition, admits a simple inversion formula for for any , as will be utilized in the simulations in section 8, although the corresponding inverse Hessian is nondiagonal for . For , the inverse Hessian is diagonal, but must then be calibrated independently from tuned for the standard law. Note that choosing to be an norm will impose sparsity on , so that should be taken to be diagonal to ensure sparsity in itself.
3.2 Non-Euclidean Measure of the Tracking Error
4 Adaptive Dynamics Prediction, Control, and Observer Design
In this section, we demonstrate how the new non-Euclidean adaptation laws of section 3.1 may be used for regularized dynamics prediction, regularized adaptive control, and regularized observer design.
4.1 Regularized Adaptive Dynamics Prediction
The above discussion demonstrates a separation theorem for adaptive dynamics prediction. If a dynamics predictor can be designed under the assumption that the true system dynamics is known (e.g., if bounds on are available), then the same dynamics predictor can be made adaptive by incorporating the skew-symmetric law, equation 4.1. Convergence properties then only depend on the nominal system with control feedback and are independent of the parameter estimator, as shown by the conditions for contraction.
In principle, these simple results could be made more general using the techniques developed in Lopez and Slotine (2021), or could be performed in a latent space computed via a nonlinear dimensionality-reduction technique such as an autoencoder (Champion, Lusch, Kutz, & Brunton, 2019) or more generally a hierarchical expansion (Chen, Paiton, & Olshausen, 2018). This could also extend to adaptive control, for example, in robot control applications where an adaptive controller could be designed in a latent space computed from raw pixels via a neural network.
4.2 Regularized Dynamics Prediction for Hamiltonian Systems
4.3 Regularized Adaptive Control for Lagrangian Systems
As in section 4.2, by using an approximation for , this approach may find sparse, interpretable models of the kinetic and potential energies. Estimating the potential energy directly may in some cases lead to simpler parameterizations than estimating the resulting forces.
If more structure in the inertia matrix is known, for example, that it depends only on a few unknown parameters, it may still be approximated using the usual Slotine and Li controller. The external forces can then be estimated by directly estimating the corresponding potential that generates them.
4.4 Regularized Adaptive Observer Design
As in section 4.1, this discussion demonstrates a separation theorem for adaptive observer design. If an observer can be designed for the true system with unknown parameters, then the same observer can be made adaptive by incorporating the adaptation law presented in this section. Convergence properties then depend only on the true system with feedback and are independent of the parameter estimator. The results of section 3.1 show that the choice of can be used to regularize the observer model while maintaining provable reconstruction of the full system state.
4.5 Regularized Dynamics Prediction for Recurrent Neural Networks
5 Velocity Gradient Algorithms and the Bregman Lagrangian
5.1 Velocity Gradient Algorithms
is nonnegative and radially unbounded, so that for all , and when . is uniformly continuous in whenever is bounded.
There exists an ideal set of control parameters such that the origin of the system 2.1 is globally asymptotically stable when the control is evaluated at . Furthermore, is a Lyapunov function for the system when the control is evaluated at . That is, there exists a strictly increasing function such that with .
The proof follows by consideration of the Lyapunov-like function .
If is chosen so that depends on only through and is linearly parameterized, then assumption 6 will immediately be satisfied by convexity of affine functions. Indeed, consider defining the goal functional for system 2.1 where depends on through . It is clear that this proposed goal functional satisfies assumptions 4 and 5 for bounded . Then , and equation 5.1 exactly recovers the Slotine and Li controller, equation 2.8.
An alternative perspective on velocity gradient algorithms can be found by using the expression . Assume that , and set where ensures that for . Then . This shows that the adaptation law transforms the gradient of with respect to by premultiplication by the regressor . This interpretation applies to the observers and dynamics predictors designed in section 4, as well as the adaptation law for contracting systems developed in Lopez and Slotine (2021). Conversely, this perspective shows that if a Lyapunov function is known for a nominal system , then the control input with adaptation law will return the perturbed system back to its nominal behavior.
is a nonnegative function and is uniformly continuous in for bounded and . Furthermore, is locally bounded in and uniformly in .
There exists an ideal set of controller parameters and a scalar function such that , , and for all .
The proof follows by consideration of the Lyapunov-like function .
Integral functionals allow the specification of a control goal that depends on all past data. is chosen so that it does not necessarily depend on the structure of the dynamics but depends explicitly on . Local functionals, on the other hand, result in adaptation laws that do have an explicit dependence on the dynamics through the appearance of the term in .
Integral functionals can be particularly useful if implies the desired control goal. In this work, we focus on the choice , which will require a PI form as described in section 2 in the context of Tyukin's algorithm.7 In particular, note that for this choice of , the result of proposition 4 implies that over the maximal interval of existence of . For some error models, this is enough to ensure that , and hence that and .8
Goal functionals can also be written as a sum of local and integral functionals with similar guarantees, and these approaches will lead to composite algorithms in the subsequent sections. (See Fradkov et al., 1999, chap. 3 for more detail.)
5.2 The Bregman Lagrangian and Accelerated Optimization Algorithms
The quantities , and in equation 5.4 are arbitrary time-dependent functions that will ultimately set the damping and learning rates in the second-order Euler Lagrange dynamics. To generate accelerated optimization algorithms, Wibisono, Wilson, and Jordan (2006) required two ideal scaling conditions: and . These conditions originate from the Euler Lagrange equations, where the second is used to eliminate an unwanted term, and a Lyapunov argument, where the first is used to ensure decrease of a chosen Lyapunov function.
Consider the higher-order adaptation algorithm, equation 5.6, with and . Then all trajectories remain bounded, , , and .
The proof follows by consideration of the Lyapunov-like function .
It is well known, for example from a passivity interpretation of the Lyapunov-like analysis (see, e.g., Slotine & Li, 1991), that the pure integrator in the standard Slotine and Li adaptation law, equation 2.8, can be replaced by any linear positive real transfer function containing a pure integrator. The higher-order algorithms presented in this work are distinct from this approach, as most clearly seen by the state-dependent damping term in equation 5.6.
In Wibisono et al. (2016), the suggested Lyapunov function in the Euclidean setting is , where is the global optimum and is the loss function. Noting that is the equivalent of in the adaptive control context (see remark 29), we see that the Lyapunov-like function used to prove stability of the adaptive law, equation 5.6, is similar to that used to prove convergence in the optimization context. The loss function term is replaced by , and it is necessary to add the term .
6 Adaptation Laws with Momentum
In this section, we develop several new adaptation laws for both linearly and nonlinearly parameterized systems. We begin by noting that the Bregman Lagrangian generates velocity gradient algorithms with momentum. We prove some general conditions under which these momentum algorithms will achieve tracking. By analogy with integral velocity gradient functionals, we then derive a proportional-integral scheme to implement a first-order composite adaptation law (Slotine & Li, 1991) driven directly by the function approximation error rather than its filtered version. We subsequently fuse the generating functional for the composite law with the Bregman Lagrangian to construct a composite algorithm with momentum.
We then employ a connection between recent developments in isotonic regression—the GLMTron of Kakade et al. (2011), along with extensions due to Goel and Klivans (2017) and Goel et al. (2018)---and Tyukin's algorithm, equation 2.12, to derive momentum algorithms for nonlinearly parameterized systems. These momentum algorithms can be seen as the adaptive control equivalent of the GLMTron with momentum.
We follow this development by discussing a new form of high-order algorithm inspired by the elastic averaging stochastic gradient descent (EASGD) algorithm (Boffi & Slotine, 2020; Zhang, Choromanska, & LeCun, 2014). We subsequently demonstrate the capability of using time-varying learning rates with our presented algorithms (Slotine & Li, 1991).
6.1 Velocity Gradient Algorithms with Momentum
With assumption 9 in hand, we can state the following proposition.
By radial unboundedness of in , equations 6.5 and 6.6 show that remains bounded. Similarly, radial unboundedness of in and show that and remain bounded. Integrating equation 6.6 shows that , so that . An identical argument shows that . Now, because and are bounded and because is locally bounded in and uniformly in by assumption, writing shows that is uniformly continuous in . Because is uniformly continuous in when is bounded, because is bounded, and because is continuous in , we conclude is uniformly continuous in and by Barbalat's lemma (lemma 71). This shows that .
and now require a modified version of assumption 9.
for some constant .
With assumption 10, we can state the following proposition.
As mentioned in section 5.1, we will be particularly interested in proposition 63 when , which will generate composite adaptation algorithms and algorithms applicable to nonlinearly parameterized systems. Proposition 63 then shows that over the interval of existence of . As shown by lemma 18, with our error model this is enough to show that always remains bounded and hence .
Classically, Lyapunov-like functions used in adaptive control consist of a sum of tracking and parameter estimation error terms, with chosen to cancel a term of unknown sign. Several Lyapunov functions in this work consist only of parameter estimation error terms, such as equation 6.10. From a mathematical point of view, all that matters is that is negative semidefinite and contains signals related to the tracking error. Integrating allows the application of tools from functional analysis to ensure that the control goal is accomplished. The lack of tracking error term in is the origin of the additional complication that must be shown to be bounded even after it is known that .
6.2 First- and Second-Order Composite Adaptation Laws
Here we consider the linearly parameterized setting , and derive new first- and second-order composite adaptation laws. Composite adaptation laws are driven by two sources of error: the tracking error itself, as summarized by in the Slotine and Li controller, and a prediction error. The prediction error term is generally obtained from an algebraic relation constructed by filtering the dynamics (Slotine & Li, 1991). We present a composite algorithm that does not require any explicit filtering of the dynamics but is instead driven simultaneously by and .
The term may also be obtained by following the I&I formalism (Astolfi & Ortega, 2003; Liu et al., 2010). To our knowledge, this discussion is the first that demonstrates the possibility of using a PI law in combination with a standard Lyapunov-stability motivated adaptation law to obtain a composite law.
More error signals may be used for additional terms in the adaptation law. For example, a prediction error obtained by filtering the dynamics may also be employed, leading to a three-term composite algorithm.
As mentioned in section 2.1, for clarity of presentation we have restricted our discussion to the th-order system 2.1. In general, the PI form 6.16 leads to undesired unknown terms contained in in addition to the desired unknown term. In this case, the desired unknown term is , while the undesired unknown term is . Indeed, the purpose of introducing the additional proportional term in equation 6.14 is to cancel this undesired unknown term. In general, cancellation of the undesired terms can be obtained by choosing to solve a PDE, and solutions to this PDE will only exist if the undesired term is the gradient of an auxiliary function. is then chosen to be exactly this auxiliary function. In some cases, the PDE can be avoided, such as through dynamic scaling techniques (Karagiannis, Sassano, & Astolfi, 2009) or the similar embedding technique of Tyukin (2011).
The properties of the adaptive law, equation 6.13, may be summarized with the following proposition.
Consider the adaptation algorithm 6.13 with a linearly parameterized unknown, . Then all trajectories remain bounded, , , , and .
The proof is given in section A.1.
Consider the higher-order composite adaptation algorithm, equation 6.19, for a linearly parameterized unknown, . Set and . Then all trajectories remain bounded, , , , , and .
The proof is given in section A.2.
By following the proof, the signal may be chosen alternatively to be matrix-valued as .
The term may be used in isolation, by considering the Lyapunov function .
A gain matrix of appropriate dimension may be placed in front of in . The quadratic parameter estimation error terms in the Lyapunov function should then be replaced by the weighted terms , and bounds on will be given in terms of .
6.3 A Momentum Algorithm for Nonlinearly Parameterized Adaptive Control
We now use the development in section 6.2 to present a new momentum algorithm applicable when the unknown parameters appear nonlinearly in the dynamics. We begin with an analogy to statistics.
Generalized linear model (GLM) regression is an extension of linear regression where the data are assumed to be generated by a function of the form for a known “link function” and unknown parameters . The first computationally and statistically efficient algorithm for this problem, the GLM-Tron of Kakade et al. (2011), assumes that is Lipschitz and monotonic, much like assumption 2.
The GLM-Tron algorithm was recently extended to the setting of kernel methods and was subsequently used to provably learn two hidden-layer neural networks by Goel and Klivans (2017); this extension is known as the Alphatron. In the kernel GLM setting handled by the Alphatron, the function to be approximated is assumed to be of the form , where is the kernel function for a reproducing kernel Hilbert space (RKHS) . is thus given by the RKHS inner product of a feature map , .
The proof is given in section A.3.
Equation 6.23 may be obtained via the Bregman Lagrangian, equation 6.18, for velocity gradient laws with momentum by choosing only the integral term. It is then necessary to modify the resulting Euler-Lagrange equations by setting to based on the monotonicity of as described in section 2.2. The following proposition summarizes the properties of equations 6.24 and 6.25.
The proof is given in section A.4.
As in remark 45, a gain matrix of appropriate dimension may be placed in front of in .
Predominantly inspired by deep learning, there has recently been strong interest in nonconvex models that are nevertheless amenable to gradient-based or gradient-inspired optimization. The development in this section suggests that machine learning models that can be provably optimized using gradient techniques represent a promising class of nonlinear parameterizations for adaptive control development.
6.4 The Elastic Modification
Boffi and Slotine (2020) observed that in the nondistributed case, equations 6.26 and 6.27 do not reduce to standard stochastic gradient descent, and that application of EASGD in this setting has different generalization properties from those of standard SGD when used to train deep neural networks. In a similar spirit, by construction of suitable Lyapunov functions, we now show that adding a center-like variable to the adaptive laws considered in previous sections maintains their stability. This immediately gives rise to a new class of higher-order adaptive control algorithms. Interestingly, these algorithms do not seem to admit an equivalent representation in terms of a single second-, third-, or fourth-order system for , but must be written as a system of first-order equations.
The algorithms considered in this subsection immediately extend to the case of cloud-based adaptation for networked robotic systems (Wensing & Slotine, 2018), where the center variable is allowed to have its own dynamics as in equation 6.27 rather than simply representing the instantaneous spatial average of the distributed parameters.
The proof is given in section A.5.
The proof is given in section A.6.
The proof is given in section A.9.
We have thus shown that all Euclidean adaptive control algorithms presented in this article,10 as well as the classic algorithm of Slotine and Li, can be modified to include feedback coupling to a low-pass filtered version of the adaptation variables. It is well known that iterate averaging for stochastic optimization algorithms such as stochastic gradient descent can improve convergence rates via variance reduction (Polyak & Juditsky, 1992). The elastic modification is similar in spirit but employs feedback rather than series coupling. This suggests that adding the elastic term may improve robustness of adaptation algorithms, and we leave a theoretical investigation of this conjecture for future work.
6.5 Exponential Forgetting Least Squares and Bounded Gain Forgetting
We focus on algorithms without the elastic modification of section 6.4; extension to the elastic modification is simple. We also focus on the bounded gain forgetting technique: proofs for the exponential forgetting least squares technique are identical, with the addition of an appropriate case statement in the time derivative of the Lyapunov function. For simplicity, we include only the time-dependent gain and set the scalar gains where applicable.
We begin with the first-order non-filtered composite, equation 6.13, with given by equation 6.42. In this case, the composite algorithm may be implemented via the PI form equations 6.14 to 6.17, where now .
The proof is given in section A.10.
The proof is given in section A.11.
Together with assumption 2, Assumption 6.48 states that lies between two linear functions. Given that the update, equation 6.42, is derived based on recursive linear least squares considerations, it is unsurprising that such an assumption is required in the nonlinearly parameterized setting. We are now in a position to state the following proposition.
The proof is given in section A.12.
The proof is given in section A.13.
7 Natural Momentum Algorithms
Consider the higher-order “natural” adaptation law equation 7.2. Assume that is -strongly convex so that globally. Take and . Then all trajectories remain bounded, , , and .
The proof is presented in section A.15.
For efficient implementation of the proposed natural momentum algorithms, as well as for their first-order equivalents, should be chosen so that is efficiently computable and ideally sparse. Alternatively, if the inverse function of the gradient is efficiently computable, or may be updated directly and subsequently inverted to arrive at the parameter values. Discretization of these algorithms is a subtle issue, and discretization of the and dynamics directly results in a natural gradient-like update (Amari, 1998), while discretization of the and dynamics leads to a mirror descent-like update (Beck & Teboulle, 2003; Nemirovski & Yudin, 1983); these discrete-time algorithms have the same continuous-time limit (Krichene, Bayen, & Bartlett, 2015).
The above natural adaptation laws with momentum may be generalized to composite algorithms, as well as to algorithms for nonlinearly parameterized adaptive control, by replacing Euclidean norms by Bregman divergences where appropriate in the proofs of the corresponding Euclidean algorithms (see, e.g., the proofs of propositions 21 and 22). Rather than derive this for each algorithm, we now show how the general results on velocity gradient algorithms with momentum (see propositions 6 and 7) can be extended to the non-Euclidean setting. We start with the case of a local functional, which requires the modification of assumption 9 to an equivalent non-Euclidean version.
For the integral variant, we require a non-Euclidean version of assumption 10.
With assumption 13, we can state the following proposition.
The general methodology captured by the proofs of propositions 23 and 24, in combination with the results of section 6.3, may be exploited to derive non-Euclidean variants of our nonfiltered composite algorithm and our momentum algorithm for nonlinearly parameterized adaptive control. Note that the strong convexity and smoothness requirements of propositions 21 and 22, in combination with a suitable choice of , are one way to satisfy the requirements of assumptions 12 and 13.
Our implicit regularization results in section 3 also extend to the higher-order setting captured by algorithm 7.2. The assumption that implies . As noted in section 5.2, , and we thus conclude that under this assumption . Because in equation 7.3 is identical to algorithm 3.2, the result follows. A formal statement of this fact is provided in section B.1.
In this section, we perform several numerical experiments demonstrating the validity of our theory and consider a number of applications of our non-Euclidean adaptive laws.
8.1 Convergence and Implicit Regularization of a Momentum Algorithm for Nonlinearly Parameterized Systems
Figure 3A shows the function approximation error for each algorithm along with a reference value for fixed . Each algorithm, as expected by our theory and seen by the low tracking error in Figure 1A, pushes to zero despite the different forms of regularization imposed on the parameter vectors. Figures 3B to 3F show the control input as a function of time along with the unique “ideal” control law valid when . All control inputs can be seen to converge to the ideal law, though the rate of convergence depends on the choice of algorithm. The control input is of reasonable magnitude throughout adaptation for each algorithm.
8.2 Learning to Control with Primitives
We now demonstrate that the mirror descent-like laws of section 3 can be used to learn convex combinations of control primitives. Our approach is analogous to the use of multiplicative weight updates in machine learning and respects the natural geometry over the probability simplex.
8.3 Dynamics Prediction for Hamiltonian Systems
We set , , and choose to identify basis functions relevant to the observed trajectory. We fix for all and initialize and to lock the system in an oscillatory mode. Past , we set and run the predictor open-loop, as well as perform shrinkage and set all coefficients with magnitude below formally equal to zero, leaving 13 remaining terms.
Results are shown in Figure 5. Figure 5A displays convergence of to zero with adaptation (solid) and demonstrates that adaptation is necessary for convergence (dashed). When switching to the open-loop predictor past , the system without adaptation sustains large errors, while the learned predictor maintains good performance. The inset displays a slow drift of the predictor trajectory from the true trajectory when run open-loop. Figure 5B displays the state trajectory (dotted), convergence of with adaptation (solid) to , and the incorrect behavior of without adaptation (dashed). The open-loop unlearned predictor tends to a fixed point, while the open-loop learned predictor maintains the correct oscillatory behavior. Figures 5C and 5D show parameter trajectories and asymptotically converged parameters, respectively. Together, the two panes demonstrate that the implicit bias of the algorithm ensures convergence to a sparse estimate of the system Hamiltonian.
8.4 Sparse Identification of Chemical Reaction Networks
Results are shown in Figure 6. In Figure 6A, we show convergence of the observer error to zero with adaptation (solid) and divergence away from zero without adaptation (dashed), demonstrating that adaptation is necessary for effective prediction. The inset displays a closer look at the asymptotic behavior of the open-loop dynamics predictor after shrinkage, which shows that the fixed point of the system is correctly learned. Figure 6B shows convergence of (solid) to (dashed). Figure 6C displays parameter trajectories as a function of time. Many parameters stay at or near zero as predicted by proposition 1. The inset displays a finer-grained view around zero of the parameter trajectories. Figure 6D shows a histogram of the final parameter values learned by the adaptive dynamics predictor, demonstrating that only a few relevant terms are identified.
9 Conclusion and Future Directions
It is somewhat unusual in nonlinear control to have a choice between a large variety of algorithms that can all be proven to globally converge. Nevertheless, in this article, we have presented a suite of new globally convergent adaptive control algorithms. The algorithms combine the velocity gradient methodology (Fradkov, 1980; Fradkov et al., 1999) with the Bregman Lagrangian (Betancourt et al., 2018; Wibisono et al., 2016) to systematically generate velocity gradient algorithms with momentum. Based on analogies between isotonic regression (Goel & Klivans, 2017; Goel et al., 2018; Kakade et al., 2011) and algorithms for nonlinearly parameterized adaptive control (Tyukin, 2011; Tyukin et al., 2007), we extended our higher-order velocity gradient algorithms to the nonlinearly parameterized setting of generalized linear models. Using a similar parallel to distributed stochastic gradient descent algorithms (Boffi & Slotine, 2020; Zhang et al., 2014), we developed a stable modification of all of our algorithms. We subsequently fused our developments with time-dependent learning rates based on the bounded gain forgetting formalism (Slotine & Li, 1991).
By consideration of the non-Euclidean Bregman Lagrangian, we derived natural gradient (Amari, 1998) and mirror descent (Beck & Teboulle, 2003; Nemirovski & Yudin, 1983)–like algorithms with momentum. Taking the infinite friction limit of these algorithms recovers a recent algorithm for adaptive robot control (Lee et al., 2018) that respects physical Riemannian constraints on the parameters throughout adaptation. By extending recent results on the implicit bias of optimization algorithms in machine learning (Azizan & Hassibi, 2019; Azizan et al., 2019) to the continuous-time setting, we proved that these mirror descent-like algorithms in the first-order, second-order, and nonlinearly parameterized settings impose implicit regularization on the parameter vectors found by adaptive control.
Throughout the article, for simplicity of exposition, we focused on the th order system, equation 2.1. As discussed in remark 2, our results extend to more general systems that have an error model similar to equation 2.6, in the sense that the proof technique summarized by lemma 72 is roughly preserved. The th order system structure makes the employed proportional-integral forms simple, as they can be written down explicitly as in equations 6.14 to 6.17. As summarized in remark 40, a PDE needs to be solved in the general case, and solutions to this PDE may not exist. Solution of the PDE can be avoided by the dynamic scaling technique of Karagiannis et al. (2009) or a similar embedding technique of Tyukin (2011).
A significant outstanding question is whether there is an empirical advantage to using our proposed momentum algorithms. In optimization, accelerated algorithms generated by the Bregman Lagrangian provide faster convergence when properly discretized, and it is thus likely that a careful discretization is necessary to obtain optimal performance of our momentum algorithms. However, we are not aware of any available convergence rates in adaptive control, and it would be necessary to prove such rates to understand analytically if there is an advantage. Similar higher-order algorithms have appeared in the literature for linear systems of relative degree greater than one (Fradkov et al., 1999; Morse, 1992), where first-order algorithms cannot control the system. Here we have focused on feedback linearizable systems, and perhaps there exist classes of nonlinear systems that cannot be adaptively controlled with a first-order algorithm but can with a momentum algorithm. We leave the investigation of these interesting and important questions to future work.
Appendix A: Omitted Proofs and Required Results
Barbalat's lemma is a classical technique in adaptive control theory, which is used in conjuction with a Lyapunov-like analysis to prove convergence of a given signal.
(Barbalat's Lemma (Slotine & Li, 1991)). Assume that . If is uniformly continuous, then .
Note that a sufficient condition for uniform continuity of is for to be bounded. Hence, for any signal with , we can apply lemma 71 to the signal and conclude that .
Assume that where is the maximal interval of existence of . Further assume that is bounded over , that both bounds are independent of , and that is locally bounded in and uniformly in . Then , , , and .