本文主要是介绍斯坦福CS229（吴恩达授）学习笔记（5），希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！
CS229-notes3
- 说明
- 正文
- Problem Set #2: Kernels, SVMs, and Theory
- 1. Kernel ridge regression
- 2. ℓ 2 \ell _2 ℓ2 norm soft margin SVMs
- 3. SVM with Gaussian kernel
- 4. Naive Bayes and SVMs for Spam Classification
说明
此笔记 是cs229-notes3讲义中的学习内容，与B站上的“07 最优间隔分类器问题；08 顺序最小优化算法”视频对应，主要是该部分对应的习题解答，解答过程中可能会用到cs229-notes1中的部分内容。
课程相关视频、讲义等资料可参照《斯坦福CS229（吴恩达授）学习笔记（1）》 获取。
正文
Problem Set #2: Kernels, SVMs, and Theory
1. Kernel ridge regression
解答：
（a）
J ( θ ) = 1 2 ∑ i = 1 m ( θ T x ( i ) − y ( i ) ) 2 + λ 2 ∣ ∣ θ ∣ ∣ 2 = 1 2 ∑ i = 1 m ( θ T x ( i ) − y ( i ) ) 2 + λ 2 θ T θ J(\theta)=\frac{1}{2}\sum_{i=1}^m(\theta^{T}x^{(i)}-y^{(i)})^2+\frac{\lambda}{2}||\theta||^2=\frac{1}{2}\sum_{i=1}^m(\theta^{T}x^{(i)}-y^{(i)})^2+\frac{\lambda}{2}\theta^T\theta J(θ)=21i=1∑m(θTx(i)−y(i))2+2λ∣∣θ∣∣2=21i=1∑m(θTx(i)−y(i))2+2λθTθ
所以
∂ ∂ θ J ( θ ) = ∂ ∂ θ 1 2 ∑ i = 1 m ( θ T x ( i ) − y ( i ) ) 2 + λ 2 θ T θ = 1 2 ∑ i = 1 m ∂ ∂ θ ( θ T x ( i ) − y ( i ) ) 2 + λ θ = 1 2 ∑ i = 1 m ⋅ 2 ⋅ ( θ T x ( i ) − y ( i ) ) ⋅ x ( i ) + λ θ = ∑ i = 1 m ( θ T x ( i ) − y ( i ) ) x ( i ) + λ θ \begin{aligned} \frac{\partial}{\partial\theta}J(\theta) =&\frac{\partial}{\partial\theta}\frac{1}{2}\sum_{i=1}^m(\theta^{T}x^{(i)}-y^{(i)})^2+\frac{\lambda}{2}\theta^T\theta \\ =&\frac{1}{2}\sum_{i=1}^m\frac{\partial}{\partial\theta}(\theta^{T}x^{(i)}-y^{(i)})^2+\lambda\theta\\ =&\frac{1}{2}\sum_{i=1}^m·2·(\theta^{T}x^{(i)}-y^{(i)})·x^{(i)}+\lambda\theta\\ =&\sum_{i=1}^m(\theta^{T}x^{(i)}-y^{(i)})x^{(i)}+\lambda\theta\\ \end{aligned} ∂θ∂J(θ)====∂θ∂21i=1∑m(θTx(i)−y(i))2+2λθTθ21i=1∑m∂θ∂(θTx(i)−y(i))2+λθ21i=1∑m⋅2⋅(θTx(i)−y(i))⋅x(i)+λθi=1∑m(θTx(i)−y(i))x(i)+λθ
令
X = [ − ( x ( 1 ) ) T − − ( x ( 2 ) ) T − ⋮ − ( x ( m ) ) T − ] y ⃗ = [ y ( 1 ) y ( 2 ) ⋮ y ( m ) ] \begin{aligned} X=&\left[ \begin{matrix} -&(x^{(1)})^T& -&\\ -&(x^{(2)})^T &-\\ &\vdots\\ -&(x^{(m)})^T& - \end{matrix} \right]\\ \vec y=&\left[ \begin{matrix} y^{(1)}\\ y^{(2)}\\ \vdots\\ y^{(m)} \end{matrix} \right]\\ \end{aligned} X=y=⎣⎢⎢⎢⎡−−−(x(1))T(x(2))T⋮(x(m))T−−−⎦⎥⎥⎥⎤⎣⎢⎢⎢⎡y(1)y(2)⋮y(m)⎦⎥⎥⎥⎤
则
∂ ∂ θ J ( θ ) = ∑ i = 1 m ( θ T x ( i ) − y ( i ) ) x ( i ) + λ θ = X T ( X θ − y ⃗ ) + λ θ = ( X T X + λ I ) θ − X T y ⃗ \begin{aligned} \frac{\partial}{\partial\theta}J(\theta) =&\sum_{i=1}^m(\theta^{T}x^{(i)}-y^{(i)})x^{(i)}+\lambda\theta\\ =&X^T(X\theta-\vec y)+\lambda\theta\\ =&(X^TX+\lambda I)\theta-X^T\vec y \end{aligned} ∂θ∂J(θ)===i=1∑m(θTx(i)−y(i))x(i)+λθXT(Xθ−y)+λθ(XTX+λI)θ−XTy
令 ∂ ∂ θ J ( θ ) = 0 \frac{\partial}{\partial\theta}J(\theta) =0 ∂θ∂J(θ)=0，可得 θ = ( X T X + λ I ) − 1 X T y ⃗ \theta=(X^TX+\lambda I)^{-1}X^T\vec y θ=(XTX+λI)−1XTy。
（b）
首先可以证明
( λ I + B A ) − 1 B = ( λ I + B A ) − 1 ( B − 1 ) − 1 = ( B − 1 ( λ I + B A ) ) − 1 = ( λ B − 1 + A ) − 1 = B B − 1 ( λ B − 1 + A ) − 1 = B ( λ B − 1 B + A B ) − 1 = B ( λ I + A B ) − 1 \begin{aligned} (\lambda I+BA)^{-1}B=&(\lambda I+BA)^{-1}(B^{-1})^{-1}\\ =&(B^{-1}(\lambda I+BA))^{-1}\\ =&(\lambda B^{-1}+A)^{-1}\\ =&BB^{-1}(\lambda B^{-1}+A)^{-1}\\ =&B(\lambda B^{-1}B+AB)^{-1}\\ =&B(\lambda I+AB)^{-1} \end{aligned} (λI+BA)−1B======(λI+BA)−1(B−1)−1(B−1(λI+BA))−1(λB−1+A)−1BB−1(λB−1+A)−1B(λB−1B+AB)−1B(λI+AB)−1
如果把 x ( i ) x^{(i)} x(i)替换成 ϕ ( x ( i ) ) \phi(x^{(i)}) ϕ(x(i))，则 θ = ( ϕ ( X ) T ϕ ( X ) + λ I ) − 1 ϕ ( X ) T y ⃗ \theta=(\phi(X)^T\phi(X)+\lambda I)^{-1}\phi(X)^T\vec y θ=(ϕ(X)Tϕ(X)+λI)−1ϕ(X)
这篇关于斯坦福CS229（吴恩达授）学习笔记（5）的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！
