A remark on Uzawa's algorithm and an application to mean field games systems

In this paper, we present an extension of Uzawa's algorithm and apply it to build approximating sequences of mean field games systems. We prove that Uzawa's iterations can be used in a more general situation than the one in it is usually used. We then present some numerical results of those iterations on discrete mean field games systems of optimal stopping, impulse control and continuous control.


Introduction
Bibliographical comments 1.A remark on Uzawa's algorithm 1.1.Presentation of the standard algorithm 1.2.A generalization of Uzawa's algorithm 2. Application of Uzawa's iterations to mean field games 2.1.The case of optimal stopping 2.2.The case of impulse control 2.3.The case of continuous control 2.4.Other possible applications of Uzawa's iterations 3. Numerical framework and numerical results 3.1.Notations and presentation of the problem 3.2.A remark on the convergence of the discretized problems toward the continuous ones 3.3.Numerical results

Introduction
This paper is concerned with the study of an extension of Uzawa's algorithm.We show that the standard Uzawa's algorithm can be used to find solutions of systems similar to the ones characterizing saddle points of lagrangians, even though there is not a proper langrangian associated with this system.The second part of this paper is concerned with the application of this remark to build approximating sequences of solutions of Mean Field Games (MFG) systems.
Uzawa's algorithm was introduced to solve minimization problems with constraints.The main idea of this algorithm is to use a projected gradient descent on the dual problem.Because of its simplicity and efficiency, Uzawa's algorithm is often used in practical problems.We recall that the output of this algorithm is a sequence which converges toward the solution of the primal minimization problem.In the first part of this paper we prove that we can use the same algorithm to find solutions of a wider class of systems than the ones which characterize saddle points of lagrangians.
Next, we take full advantage of this remark to build approximating sequences for several MFG systems.MFG have been introduced by J.-M.Lasry and P.-L.Lions in [14] and independently and in a particular case by M. Huang, P. Caines and R. Malhamme in [13].The theory of MFG is concerned with Nash equilibria of differential games with infinitely mainly indiscernable players, who interacts only through mean field type terms.We refer to [16] for a detailed presentation of MFG and to [10] for a complete presentation of the probabilistic theory of MFG.In general, the study of a MFG requires to solve the so-called master equation, see [16,9], but in the case when there is no common noise, the problem reduces to a system of Partial Differential Equations (PDE).It is well known that in the so-called potential case, MFG systems can be interpreted as the optimality conditions for an optimal control problem of a PDE, see [14,8] for instance.Thus Uzawa's algorithm is a natural method we can apply to such optimal control problems.We show that, under monotonicity assumptions, we can apply an Uzawa's like algorithm to approximate solutions of MFG systems even in the non-potential case.In this paper we shall apply this algorithm to MFG systems of continuous control (i.e. as in [14]), optimal stopping (see [4]) and impulse control (see [5]).
The last part of this paper presents the results of the implementation of Uzawa's iterations to the discretized problems of MFG of optimal stopping, impulse control and continuous control.

Bibliographical comments.
We here give some details on the bibliographical context in which this article takes place.Concerning the literature regarding Uzawa's algorithm, there exist plenty of results on this well known algorithm.
Although, using this algorithm to find solutions of systems of inequalities seems to be used only in the case of linear system, as in [6,11] for instance.
Concerning the MFG literature, the first numerical methods for MFG systems have been developed by Y. Achdou and I. Capuzzo-Dolcetta in [1].Several other methods have been studied and some of them involved the optimal control interpretation in the potential case.Such methods are somehow similar to the one we present here because they are also the implementation of a search for saddle points.We refer to [7,3] for example.The main novelties of our work is to consider the non potential case and that we consider the cases of optimal stopping and impulse control.Furthermore, we mention the papers [12,2] of R. Ferreira, D. Gomes and al. in which the first order MFG system of continuous control is interpreted as a system of variational inequalities and solve numerically.The interpretation in terms of variational inequalities of the MFG system is central in the rest of this paper.
1.A remark on Uzawa's algorithm 1.1.Presentation of the standard algorithm.We present here the classical result of convergence of Uzawa's algorithm.Although we are going to present this algorithm from the point of view of the search of a saddle point, let us recall the well-known fact that given a convex minimization problem, Uzawa's algorithm is only the projected gradient ascent method applied on the dual problem.Let us take a lagrangian L defined by: where K 1 is a closed convex subset of the Hilbert space (H  We recall that a saddle point of L is a couple (x, y) We fix a real number δ > 0 and we denote by P A the orthogonal projection on the set A in H 3 .Uzawa's algorithm (with step δ) consists in building the sequence (x n , y n ) n∈N as follows : (3) where we recall that K2 = b(K 2 ).Before presenting a convergence result for those iterations, we introduce the following definition.An application f from the Hilbert space (H, < •, • >) into itself is said to be α monotone if for any x, y ∈ H, An application 0 monotone is simply called monotone and an application f is said to be strictly monotone if for any x, y ∈ H such that x = y the following holds A classical convergence result concerning the sequence (x n , y n ) n∈N is the following : Theorem 1.Let us assume that : • The application F is differentiable with differential f which is α monotone.
• The application a is C lipschitz for some constant C > 0. Then if δ < 2α C 2 , for any y 0 ∈ K 2 , the sequence (x n , y n ) n∈N defined by ( 3) is well defined and (x n ) n∈N converges toward x * in H 1 , where (x * , y * ) is the unique saddle point of L.
We briefly recall here the proof of this result.
Proof.Given the assumptions we made, we know that there exists a unique couple (x * , y * ) ∈ H 1 × H 2 verifying (2).It satisfies Moreover by construction, (x n ) n∈N satisfies for all n ∈ N : Because P K2 is a contraction, we obtain that We then deduce that C 2 we obtain that there exists 0 < β < 1 such that : Remark 1.The use of the application b and of the Hilbert space H 2 is somewhat artificial.We only use this formalism because it is closer to the set up needed for the applications of the next section.Moreover let us note that we do not state any convergence for the sequence (y n ) n≥0 .

1.2.
A generalization of Uzawa's algorithm.We now remark that instead of using (3) to define a sequence (x n , y n ) n∈N , we can use the following : Let us note that if F is a convex differentiable function, then (3) and ( 4) are equivalent, but the second one is more general in the sense that it allows us to build the sequence (x n , y n ) n∈N even in the case in which there is no function F for which f is the differential.Under the assumptions of theorem 1, the sequence (x n ) n∈N converges toward x * where (x * , y * ) is the saddle point of L.Here we are interested in approximating the couples (x * , y * ) ∈ H 1 × H 2 solutions of ( 5) We establish the following result : We assume that : • The application f is α monotone.
• The application a is C lipschitz for some constant C > 0 and differentiable.
• There exists a couple (x * , y * ) satisfying ( 5).Then, for any Proof.First let us remark that for any y 0 ∈ K 2 , the sequence (x n , y n ) n∈N is well defined.Indeed the second line of (4) defines a unique element x n ∈ H 1 for any n ∈ N.This comes from the fact that for any y ∈ K 2 , ǫ > 0, we can define the sequence (ξ p ) p∈N by : where P K 1 stands for the orthogonal projection (for H 1 ) onto K 1 .If ǫ is small enough, the sequence (ξ p ) p∈N is a Cauchy sequence whose limit ξ * satisfies Such an element is unique because of the α monotonicity of f .The rest of the proof follows the same argument as in the proof of theorem 1.
Remark 2. The existence of a couple (x * , y * ) satisfying ( 5) can be obtained directly under some assumptions on the applications a and b via a Kakutani's type fixed point theorem.

Application of Uzawa's iterations to mean field games
We now present how we can use the previous results to approximate some MFG systems.We shall apply this remark on Uzawa's algorithm to three different MFG systems.The first one is a system modeling a MFG of optimal stopping as in [4].The second one is a MFG system modeling an impulse control problem, following [5] and we finish with the classical case of continuous control as in [14].To simplify notations, we present the following results in a stationary setting in which the state space is the d dimensional torus T d .
2.1.The case of optimal stopping.We are here interested in approximating the solution of the following system of unknown (u, m) : where f is a continuous application from L 2 (T d ) into itself, ν, λ > 0 are two parameters of the model and ρ ∈ H −1 (T d ) is the entry rate of the players.The exit cost of the MFG is here 0. The first variational inequality of this system arises from the obstacle problem satisfied by the value function u of a generic player.The second variational inequality and the integral relation arise from the "Fokker-Planck equation" satisfied by the density of players m.Let us remark that we have here abused the name variational inequality as we only refer to a variational formulation which is an inequality and not to the famous concept introduced in [15] by Lions and Stampacchia.This system models Nash equilibria in mixed strategies of a MFG of optimal stopping, we refer to [4] for more details on this system.From [4] we know that there exists a unique solution (u, m) ∈ H 2 (T d ) × H 1 (T d ) of (6) under the assumption that f is strictly monotone, i. e. that it satisfies for all m, m ′ ∈ L 2 (T d ): Let us remark that (u, m) also satisfies ( 7) In the case when f is strictly monotone, (7) ) which is the unique solution of (6).The system (7) falls under the scope of application of the previous section.Thus we define for δ > 0 the following Uzawa's iterations : where L is the linear operator and P A stands for the orthogonal projection in L 2 (T d ) onto the set A. Let us note that from classical results on variational inequalities (see [15] for instance), (u n ) n≥0 is a well defined sequence of Recalling the results of the previous section, (m n ) n≥0 is well defined under some monotonicity assumptions on f .We have the following result : Theorem 3. Assume that f is α monotone from L 2 (T d ) into itself for some α > 0 and that δ < 2α, then for any u 0 ∈ H 2 (T d ), the sequence (m n ) n≥0 defined by (8) converges toward m in L 2 (T d ), where (u, m) is the only solution of (6).
Proof.This result is a direct application of theorem 2.
Let us remark that the projection involved in ( 8) is similar to the resolution of a bi-laplacian obstacle problem.Indeed, given u n , m n , we are looking for u n+1 such that : Let us also make a remark on the potential case.The potential case is the case when there exists F : L 2 (T d ) → R such that for every m, m ′ ∈ L 2 (T d ) : In such a situation, if f is strictly monotone, following the result of [4], the unique solution (u, m) of ( 6) is also the saddle point of the lagrangian L defined on {µ ∈ The iterations (8) are then the result of the classical Uzawa's algorithm on L.
2.2.The case of impulse control.In this section we are interested in building approximations of solutions of the following system : (10) where f is a continuous application from L 2 (T d ) into itself, bounded uniformly from below on the positive elements of L 2 (T d ), ν, λ > 0 are two parameters of the model, ρ ∈ H −1 (T d ) is the entry rate of players and M is the operator defined by where J is a finite set of T d and k is a smooth non-negative function.
The system (10) models Nash equilibria of MFG of impulse control in which the players face the running cost f (m) and have to pay k(x, ξ) if they are in x to jump ξ further.The density of players is m and u represents the value function of a generic player.We refer to [5] for more details on this problem and for the following result.If f is strictly monotone and k satisfies (10), moreover, this couple (u, m) satisfies : Thus we define, as in the case of optimal stopping, the following Uzawa's iterations for δ > 0 : where L still denotes the linear operator L = −ν∆ + λId, the closed convex set K ′ is defined by and P A stands for the orthogonal projection onto A in L 2 (T d ).We have the following result of convergence : Theorem 4. Assume that f is α monotone from L 2 (T d ) into itself for some α > 0 and that δ < 2α, then for any u 0 ∈ H 2 (T d ), the sequence (m n ) n≥0 defined by (8) converges toward m in L 2 (T d ), where (u, m) is the only solution of (6).
Proof.This result is once again a direct application of theorem 2.
Let us remark that although the Hamilton-Jacobi-Bellmann equation in ( 10) is a quasi-variational inequality, the equation we have to solve at each iteration in (12) to update the lagrange multiplier u n is a variational inequality, which is in principle easier to solve than a quasi-variational inequality.
Moreover, in the potential case, i. e. when there exists F satisfying (9), if f is strictly monotone, the solution (u, m) of ( 10) is the saddle point of L defined on {m ∈ L 2 (T d ), m ≥ 0} × {u ∈ H 2 (T d ), u ≤ M u} by : This results can be found in [5].The iterations (12) are then the ones from the use of the classical Uzawa's algorithm on L.

2.3.
The case of continuous control.We end this list of applications of Uzawa's iterations with the construction of approximating sequences for the following MFG system : where f is the running cost of the players and the hamiltonian H(x, p) is assumed to be convex in its second variable and uniformly lispchitz.We refer the reader to [14,16] for a full presentation and results on the system (13).
) is a solution of ( 13) (with m being a weak solution of the Fokker-Planck equation), then it is also a solution of : ( 14) Under the assumption that f is strictly monotone, there exists at most one solution (u, m) ∈ H 2 (T d ) × L 2 (T d ) of ( 14).Although this system does not allow a direct application of theorem 2, the convexity of the hamiltonian allows us to prove a result of convergence for Uzawa's like iterations.Given a sequence of non-negative real numbers (δ n ) n≥0 , we define Uzawa's iteration in this case by : (15) ; where for all v ∈ H 1 (T d ), L v is the operator defined by : We now establish the following result : Theorem 5. Assume that there exists a solution (u, m) ∈ H 2 (T d ) × L 2 (T d ) of ( 13) and that f is α monotone.Then there exists a sequence of non-negative real number (δ n ) n≥0 such that the iterations (u n , m n ) n≥0 defined by (15) are such that Proof.We denote by (u, m) ∈ H 2 (T d )×L 2 (T d ) the unique solution of (14).We take a sequence (δ n ) n≥0 , δ n > 0 for all n ≥ 0 and we consider the iterations (u n , m n ) n≥0 given by ( 15) for a fixed u 0 ∈ H 2 (T d ).We introduce the notation Let us remark that for all n ≥ 0: We now make some calculations around the third term of the right hand side of the previous equality.We compute : Because m is the solution of L * u m = ρ, we deduce from the convexity of H : On the other hand : By the maximum principle, L * −1 un ρ ≥ 0, thus we deduce from the convexity of the hamiltonian that : This inequality, together with the previous one implies that : Using the equation satisfied by u and the definition of m n , we obtain that : The α convexity of f yields finally : Using this inequality in (16) we obtain that: We assume in a first time that for all n ∈ N : (17) Let us observe that in this situation the sequence (||HJB(u n ) − HJB(u)|| L 2 ) n≥0 is decreasing and thus it has a limit and (u n ) n≥0 is a bounded sequence of H 2 (T d ).We also remark that we deduce from the convergence of (||HJB Because (||HJB(u n )|| L 2 ) n≥0 is bounded and f is α monotone, we deduce that (m n ) n≥0 is bounded in L 2 and thus that : To complete the proof of the theorem, let us remark that if (17) is not satisfied for n * ∈ N, then m n * = m and the convergence is also proved.
Remark 3. Let us remark that because of the α monotonicity of f , there are obvious estimates in L 2 (T d ) for (m n ) n≥0 , thus the sequence (δ n ) n≥0 can be chosen to be an explicit constant.
In the potential case, when there exists F satisfying (9), and when f is strictly monotone, the solution (u, m) of ( 13) is also the saddle point of the lagrangian L defined on {µ ∈ L 2 (T d ), µ ≥ 0} × H 2 (T d ) by : Even though the optimal control interpretation presented in [14] is not exactly written in this form, it can be easily checked that the formulations are equivalent, at least formally.The iterations (15) are in this case the result of Uzawa's algorithm on L, in the sense that they are formally the result of a gradient ascent method on the dual problem : sup Remark 4. In the three cases mentioned above (optimal stopping, impulse control and continuous control), the sequence (u n ) n≥0 defined by the Uzawa's iterations is always a bounded sequence of H 2 (T d ).Therefore, up to a subsequence, (u n ) n≥0 converges in H 1 (T d ) toward u ∈ H 2 (T d ).This function u is in fact such that (u, m) is the solution of the MFG system and thus the whole sequence (u n ) n≥0 converges toward u.

2.4.
Other possible applications of Uzawa's iterations.We give here some immediate applications of Uzawa's iterations.First let us note that the operator −ν∆ + λId involved in the three problems above can be replaced by more general elliptic linear operators.Let us also mention that this method is also valid in more general domains than T d .This method can also be applied in time dependent situations.
Another important remark is that Uzawa's iterations can also be applied in the optimal control of PDE governed by inequalities, such that inf where A is a partial differential operator.Such a class of problem is of some importance.For instance we refer to [4] for a heuristic argument on why (6) can be interpreted as the optimality conditions for the relaxation of an optimal shape problem.The relaxation is then of the form just mentioned above.

Numerical framework and numerical results
We present here the discrete versions of the three problems mentioned in the previous section (optimal stopping, impulse control and continuous control MFG systems).We also present numerical results of the implementation of Uzawa's iteration for those problems.
3.1.Notations and presentation of the problem.We give here the notations we are going to use to present the discretized problem we are interested in.We fix a non-negative integer d and we define h > 0 by h = d −1 .We work here on a grid G d = {(i, j), 1 ≤ i, j ≤ d} which we interpret as a discretization of the 2 dimensional torus.Let f d : R d 2 → R d 2 be a continuous application.We fix ξ ∈ G d and k 0 > 0 a real number.We then define for all v ∈ R d 2 M v by : We denote by g : (p 1 , p 2 , p 3 , p 4 ) → g(p 1 , p 2 , p 3 , p 4 ) a discretization of the hamiltonian H : R 2 → R defined by H(p) = 1 + |p| 2 .Thus g is such that for p 1 , p 2 ∈ R : g(p 1 , p 1 , p 2 , p 2 ) = 1 + (p 1 ) 2 + (p 2 ) 2 ; and g is non decreasing with respect to p 1 and p 3 and non decreasing with respect to p 2 and p 4 .We denote by ∇ p g the gradient of g.We also define the vector of derivatives D h v of a vector v ∈ R d 2 by : For ν, λ > 0, we also define the discrete operator A : R d 2 → R d 2 by : where we use periodic boundary condition on G d .For v ∈ R p for some p ≥ 0, we use the notation v ≥ 0 when for all 1 ≤ i ≤ p, v i ≥ 0. We take an element For the rest of this section, R d 2 is endowed with the scalar product : In this section, we present the results of the implementation of Uzawa's iterations to approximate the solutions of the following three problems (each time the unknown is the couple (u, m)) : Those problems are the discretized version of respectively ( 6), (10) and (13).

3.2.
A remark on the convergence of the discretized problems toward the continuous ones.Although the convergence of (18), ( 19) and (20) toward their continuous version is not the objective of this article, we explain here briefly why such a convergence is expected.We give some results on the case of (18).We refer to [1] for results on (20).

We begin by detailing in which sense
We assume that if ( md ) d≥1 converges toward m in L 2 (T 2 ), then ( fd (m d )) d≥1 converges toward f (m) in L 2 (T d ).We also assume, using the same notations, that (ρ d ) d≥1 converges toward ρ in L 2 (T 2 ).We now start by proving a lemma which gives the main idea for the convergence of the finite problem.
Lemma 1.Let us assume that f d : R d 2 → R d 2 is α monotone and that (u, m) is the only solution of (18).For any ǫ 1 , ǫ 2 > 0 and (v, µ) such that Therefore, using the fact that (u, m) is the solution of (18), we deduce that Thus we obtain that Using the α monotonicity of f d the result is proved.
Remark 5.If (21) is not satisfied for all µ ′ , v ′ but only for m, u, the results of the lemma still holds.
We now show an exemple of result of convergence.
Proposition 1.Let us assume that f : L 2 (T 2 ) → L 2 (T 2 ) is α monotone and let us assume that there exists (u * , m * ), unique solution of (6).We also assume that for every d ∈ N, d ≥ 1, f d : R d 2 → R d 2 is α monotone and that there exists a unique solution (u d , m d ) of (18).Then the following holds  In figure 3 we give the density m and its Lagrange multiplier u obtained after 3000 Uzawa's iterations.We use a standard Newton method on the finite differences scheme at each step to solve the Hamilton-Jacobi-Bellmann equation which updates the Lagrange multiplier.To compute

( a )
Contours of m (b) Contours of u (c) Graph of m (d) Graph of u