数值计算之 梯度向量和梯度矩阵，雅可比矩阵，海森矩阵

数值计算之 梯度向量和梯度矩阵，雅可比矩阵，海森矩阵

数值计算之 梯度向量，雅可比矩阵，海森矩阵

• 前言
• 梯度向量
• 梯度矩阵
• 雅可比矩阵
• 海森矩阵
• 总结

前言

非线性最小二乘中的函数求导内容，主要涉及梯度向量、雅可比矩阵和海森矩阵。因此提前做一个辨析。实际上之前在矩阵求导中已经提到过这些内容。

梯度向量

对于实值向量函数 f ( x ) ∈ R , x = ( x 1 , x 2 , … , x n ) T f(x)in R,x=(x_1,x_2,dots,x_n)^T f(x)∈R,x=(x1​,x2​,…,xn​)T，其梯度向量可表示为：
∇ f ( x ) = ∂ f ( x ) ∂ x = [ ∂ f ∂ x 1 ∂ f ∂ x 2 … ∂ f ∂ x n ] nabla f(x)=frac {partial f(x)}{partial x}= begin{bmatrix} frac {partial f}{partial x_1} \ frac {partial f}{partial x_2} \ dots \ frac {partial f}{partial x_n} \ end{bmatrix} ∇f(x)=∂x∂f(x)​=⎣⎢⎢⎢⎡​∂x1​∂f​∂x2​∂f​…∂xn​∂f​​⎦⎥⎥⎥⎤​
梯度向量的布局与分母的布局相同，也就是分母布局。

梯度矩阵

对于实值矩阵函数 f ( X ) ∈ R f(X)in R f(X)∈R，其梯度矩阵 ∇ f ( X ) nabla f(X) ∇f(X)可表示为：
∇ f ( X ) = ∂ f T ( X ) ∂ X = [ ∂ f ∂ x 11 ∂ f ∂ x 12 … ∂ f ∂ x 1 n ∂ f ∂ x 21 ∂ f ∂ x 22 … ∂ f ∂ x 2 n … … … … ∂ f ∂ x n 1 ∂ f ∂ x n 2 … ∂ f ∂ x n n ] nabla f(X)=frac {partial f^T(X)}{partial X}= begin{bmatrix} frac {partial f}{partial x_{11}} & frac {partial f}{partial x_{12}} & dots & frac {partial f}{partial x_{1n}} \ frac {partial f}{partial x_{21}} & frac {partial f}{partial x_{22}} & dots & frac {partial f}{partial x_{2n}} \ dots & dots & dots & dots\ frac {partial f}{partial x_{n1}} & frac {partial f}{partial x_{n2}} & dots & frac {partial f}{partial x_{nn}} \ end{bmatrix} ∇f(X)=∂X∂fT(X)​=⎣⎢⎢⎢⎡​∂x11​∂f​∂x21​∂f​…∂xn1​∂f​​∂x12​∂f​∂x22​∂f​…∂xn2​∂f​​…………​∂x1n​∂f​∂x2n​∂f​…∂xnn​∂f​​⎦⎥⎥⎥⎤​
对于实值向量函数 f i ( x ) ∈ R , x = ( x 1 , x 2 , … , x n ) T f_i(x)in R,x=(x_1,x_2,dots,x_n)^T fi​(x)∈R,x=(x1​,x2​,…,xn​)T组成的向量 ( f 1 ( x ) , f 2 ( x ) , … , f n ( x ) ) T (f_1(x),f_2(x),dots,f_n(x))^T (f1​(x),f2​(x),…,fn​(x))T，其梯度矩阵可表示为：
∇ f ( x ) = ∂ f T ( x ) ∂ x = [ ∂ f 1 ∂ x 1 ∂ f 2 ∂ x 1 … ∂ f n ∂ x 1 ∂ f 1 ∂ x 2 ∂ f 2 ∂ x 2 … ∂ f n ∂ x 2 … … … … ∂ f 1 ∂ x n ∂ f 2 ∂ x n … ∂ f n ∂ x n ] nabla f(x)=frac {partial f^T(x)}{partial x} =begin{bmatrix} frac {partial f_1}{partial x_{1}} & frac {partial f_2}{partial x_{1}} & dots & frac {partial f_n}{partial x_{1}} \ frac {partial f_1}{partial x_{2}} & frac {partial f_2}{partial x_{2}} & dots & frac {partial f_n}{partial x_{2}} \ dots & dots & dots & dots\ frac {partial f_1}{partial x_{n}} & frac {partial f_2}{partial x_{n}} & dots & frac {partial f_n}{partial x_{n}} \ end{bmatrix} ∇f(x)=∂x∂fT(x)​=⎣⎢⎢⎢⎡​∂x1​∂f1​​∂x2​∂f1​​…∂xn​∂f1​​​∂x1​∂f2​​∂x2​∂f2​​…∂xn​∂f2​​​…………​∂x1​∂fn​​∂x2​∂fn​​…∂xn​∂fn​​​⎦⎥⎥⎥⎤​
梯度矩阵需要保持分母的布局不变，也就是分母布局。

雅可比矩阵

对于实值矩阵函数 f ( X ) ∈ R f(X)in R f(X)∈R，其雅可比矩阵 J ( X ) J(X) J(X)可表示为：
J ( X ) = ∂ f ( X ) ∂ X T = [ ∂ f ∂ x 11 ∂ f ∂ x 21 … ∂ f ∂ x n 1 ∂ f ∂ x 12 ∂ f ∂ x 22 … ∂ f ∂ x n 2 … … … … ∂ f ∂ x 1 n ∂ f ∂ x 2 n … ∂ f ∂ x n n ] J(X)=frac {partial f(X)}{partial X^T}= begin{bmatrix} frac {partial f}{partial x_{11}} & frac {partial f}{partial x_{21}} & dots & frac {partial f}{partial x_{n1}} \ frac {partial f}{partial x_{12}} & frac {partial f}{partial x_{22}} & dots & frac {partial f}{partial x_{n2}} \ dots & dots & dots & dots\ frac {partial f}{partial x_{1n}} & frac {partial f}{partial x_{2n}} & dots & frac {partial f}{partial x_{nn}} \ end{bmatrix} J(X)=∂XT∂f(X)​=⎣⎢⎢⎢⎡​∂x11​∂f​∂x12​∂f​…∂x1n​∂f​​∂x21​∂f​∂x22​∂f​…∂x2n​∂f​​…………​∂xn1​∂f​∂xn2​∂f​…∂xnn​∂f​​⎦⎥⎥⎥⎤​

对于由实值向量函数 f i ( x ) ∈ R , x = ( x 1 , x 2 , … , x n ) T f_i(x)in R,x=(x_1,x_2,dots,x_n)^T fi​(x)∈R,x=(x1​,x2​,…,xn​)T组成的向量 ( f 1 ( x ) , f 2 ( x ) , … , f n ( x ) ) T (f_1(x),f_2(x),dots,f_n(x))^T (f1​(x),f2​(x),…,fn​(x))T，其雅可比矩阵可表示为：
J ( x ) = ∂ f ( x ) ∂ x T = [ ∂ f 1 ∂ x 1 ∂ f 1 ∂ x 2 … ∂ f 1 ∂ x n ∂ f 2 ∂ x 1 ∂ f 2 ∂ x 2 … ∂ f 2 ∂ x n … … … … ∂ f n ∂ x 1 ∂ f n ∂ x 2 … ∂ f n ∂ x n ] J(x)=frac {partial f(x)}{partial x^T}=begin{bmatrix} frac {partial f_1}{partial x_1} & frac {partial f_1}{partial x_2} & dots & frac {partial f_1}{partial x_n} \ frac {partial f_2}{partial x_1} & frac {partial f_2}{partial x_2} & dots & frac {partial f_2}{partial x_n} \ dots & dots & dots & dots \ frac {partial f_n}{partial x_1} & frac {partial f_n}{partial x_2} & dots & frac {partial f_n}{partial x_n} \ end{bmatrix} J(x)=∂xT∂f(x)​=⎣⎢⎢⎢⎡​∂x1​∂f1​​∂x1​∂f2​​…∂x1​∂fn​​​∂x2​∂f1​​∂x2​∂f2​​…∂x2​∂fn​​​…………​∂xn​∂f1​​∂xn​∂f2​​…∂xn​∂fn​​​⎦⎥⎥⎥⎤​
雅可比矩阵需要保持分子的布局不变，也就是分子布局。

海森矩阵

对于实值向量函数 f ( x ) ∈ R , x = ( x 1 , x 2 , … , x n ) T f(x)in R,x=(x_1,x_2,dots,x_n)^T f(x)∈R,x=(x1​,x2​,…,xn​)T，其海森矩阵 H ( x ) H(x) H(x)是函数的二阶导，实际上就是 f ( x ) f(x) f(x)的梯度向量对 x x x的雅可比矩阵：

H ( x ) = J ( ∇ f ( x ) ) = ∂ ∂ f ( x ) ∂ x ∂ x T = [ ∂ 2 f ∂ x 1 ∂ x 1 ∂ 2 f ∂ x 1 ∂ x 2 … ∂ 2 f ∂ x 1 ∂ x n ∂ 2 f ∂ x 2 ∂ x 1 ∂ 2 f ∂ x 2 ∂ x 2 … ∂ 2 f ∂ x 2 ∂ x n … … … … ∂ 2 f ∂ x n ∂ x 1 ∂ 2 f ∂ x n ∂ x 2 … ∂ 2 f ∂ x n ∂ x n ] H(x)=J(nabla f(x))=frac{partial frac {partial f(x)}{partial x}}{partial x^T} = begin{bmatrix} frac {partial^2f}{partial x_1partial x_1} & frac {partial^2f}{partial x_1partial x_2} & dots& frac {partial^2f}{partial x_1partial x_n} \ frac {partial^2f}{partial x_2partial x_1} & frac {partial^2f}{partial x_2partial x_2} & dots& frac {partial^2f}{partial x_2partial x_n} \ dots & dots & dots & dots \ frac {partial^2f}{partial x_npartial x_1} & frac {partial^2f}{partial x_npartial x_2} & dots& frac {partial^2f}{partial x_npartial x_n} \ end{bmatrix} H(x)=J(∇f(x))=∂xT∂∂x∂f(x)​​=⎣⎢⎢⎢⎡​∂x1​∂x1​∂2f​∂x2​∂x1​∂2f​…∂xn​∂x1​∂2f​​∂x1​∂x2​∂2f​∂x2​∂x2​∂2f​…∂xn​∂x2​∂2f​​…………​∂x1​∂xn​∂2f​∂x2​∂xn​∂2f​…∂xn​∂xn​∂2f​​⎦⎥⎥⎥⎤​

总结

梯度向量和梯度矩阵保持分母布局不变；

雅可比矩阵保持分子布局不变；

海森矩阵其实是实值向量函数的梯度向量对自变量求雅可比矩阵。