x x x y y y f θ ( x ) f_{\theta}(x) f θ ( x ) θ \theta θ
x x x 32 × 32 × 3 32\times 32\times 3 32 × 32 × 3 f f f
O ( 1 ) O(1) O ( 1 ) O ( N ) O(N) O ( N ) N N N
d 1 ( I 1 , I 2 ) = ∑ p ∣ I 1 p − I 2 p ∣ d_{1}(I_{1},I_{2})=\sum_{p}\mid I_{1}^p - I_{2}^p \mid d 1 ( I 1 , I 2 ) = ∑ p ∣ I 1 p − I 2 p ∣ d 2 ( I 1 , I 2 ) = ∑ p ( I 1 p − I 2 p ) 2 d2(I_1,I_2)=\sqrt{ \sum_{p}(I_{1}^p−I_{2}^p)^2 } d 2 ( I 1 , I 2 ) = ∑ p ( I 1 p − I 2 p ) 2
x x x W W W b b b
f ( x , W ) = W x + b f(x,W) = Wx+b f ( x , W ) = W x + b
W W W W x + b = 0 Wx+b=0 W x + b = 0 W W W W W W
P ( Y = k ∣ X = x i ) = e S y i ∑ j e S j P(Y=k \mid X=x_{i}) = \frac{e^{S_{y_{i}}}}{\sum_{j}e^{S_{j}}} P ( Y = k ∣ X = x i ) = ∑ j e S j e S y i
L i = − l o g p L_{i}=-log \ p L i = − l o g p
1 C \frac{1}{C} C 1 L i = − log ( 1 C ) = l o g C L_{i}=-\log\left( \frac{1}{C} \right) =log C L i = − log ( C 1 ) = l o g C log 10 ≈ 2.3 \log10 ≈ 2.3 log 10 ≈ 2.3 log C \log C log C
L i = ∑ j ≠ y i { 0 if s y i ≥ s j + 1 s j − s y i + 1 otherwise = ∑ j ≠ y i max ( 0 , s j − s y i + 1 ) L_i = \sum_{j \ne y_i}
\begin{cases}
0 & \text{if } s_{y_i} \ge s_j + 1 \\
s_j - s_{y_i} + 1 & \text{otherwise}
\end{cases}
= \sum_{j \ne y_i} \max(0,\, s_j - s_{y_i} + 1) L i = j = y i ∑ { 0 s j − s y i + 1 if s y i ≥ s j + 1 otherwise = j = y i ∑ max ( 0 , s j − s y i + 1 )
L ( W ) = 1 N ∑ i = 1 N L i ( f ( x i , W ) , y i ) ⏟ Data Loss + λ R ( W ) ⏟ Regularization Loss L(W) = \underbrace{\frac{1}{N} \sum_{i=1}^{N} L_{i}(f(x_{i}, W), y_{i})}_{\text{Data Loss}} + \underbrace{\lambda R(W)}_{\text{Regularization Loss}} L ( W ) = Data Loss N 1 i = 1 ∑ N L i ( f ( x i , W ) , y i ) + Regularization Loss λ R ( W )
R ( W ) = ∑ k ∑ l W k , l 2 R(W)=\sum_{k}\sum_{l}W^2_{k,l} R ( W ) = k ∑ l ∑ W k , l 2 W W W [ 0.25 , 0.25 , 0.25 , 0.25 ] [0.25,0.25,0.25,0.25] [ 0.25 , 0.25 , 0.25 , 0.25 ] [ 1 , 0 , 0 , 0 ] [1,0,0,0] [ 1 , 0 , 0 , 0 ]
R ( W ) = ∑ k ∑ l ∣ W k , l ∣ R(W)=\sum_{k}\sum_{l}\mid W_{k,l} \mid R ( W ) = k ∑ l ∑ ∣ W k , l ∣
min w 1 2 ( w − a ) 2 + λ ∣ w ∣ \min_{w} \frac{1}{2}(w-a)^2+\lambda\mid w \mid w min 2 1 ( w − a ) 2 + λ ∣ w ∣ w ∗ = s i g n ( a ) ⋅ m a x ( ∣ a ∣ − λ , 0 ) w^*=sign(a)\cdot max(\mid a \mid -\lambda,0) w ∗ = s i g n ( a ) ⋅ ma x ( ∣ a ∣ − λ , 0 ) ∣ a ∣ ≤ λ \mid a\mid \leq \lambda ∣ a ∣≤ λ w ∗ = 0 w^*=0 w ∗ = 0
W W W L ( W ) L(W) L ( W )
d f ( x ) d x = lim h → 0 f ( x + h ) − f ( x ) h \frac{df(x)}{dx} = \lim_{h \to 0} \frac{f(x+h)-f(x)}{h}
d x df ( x ) = h → 0 lim h f ( x + h ) − f ( x ) ∇ W L \nabla_{W}L ∇ W L
d f ( x ) d x = lim h → 0 f ( x + h ) − f ( x ) h \frac{df(x)}{dx} = \lim_{h \to 0} \frac{f(x+h)-f(x)}{h} d x df ( x ) = lim h → 0 h f ( x + h ) − f ( x ) x ∈ R D x\in\mathbb{R}^D x ∈ R D W ∈ R C × D W\in\mathbb{R}^{C\times D} W ∈ R C × D s = W x s = Wx s = W x p = s o f t m a x ( s ) p=\mathrm{softmax}(s) p = softmax ( s ) p k = e s k ∑ j e s j p_k=\frac{e^{s_k}}{\sum_j e^{s_j}} p k = ∑ j e s j e s k L = − log p y L=-\log p_y L = − log p y y y y L L L W W W W → s → p → L W\rightarrow s\rightarrow p\rightarrow L W → s → p → L ∂ L ∂ W \frac{\partial L}{\partial W} ∂ W ∂ L ∂ L ∂ W = ∂ L ∂ s ⋅ ∂ s ∂ W \frac{\partial L}{\partial W}=\frac{\partial L}{\partial s}\cdot\frac{\partial s}{\partial W} ∂ W ∂ L = ∂ s ∂ L ⋅ ∂ W ∂ s ∂ L ∂ s \frac{\partial L}{\partial s} ∂ s ∂ L y oh ∈ R C y_{\text{oh}}\in\mathbb{R}^C y oh ∈ R C ∂ L ∂ s = p − y oh \frac{\partial L}{\partial s}=p-y_{\text{oh}} ∂ s ∂ L = p − y oh s = W x s=Wx s = W x ∂ L ∂ W k j = ( p k − ( y oh ) k ) x j \frac{\partial L}{\partial W_{kj}}=(p_k-(y_{\text{oh}})_k)x_j ∂ W kj ∂ L = ( p k − ( y oh ) k ) x j ∂ L ∂ W = ( p − y oh ) x T \frac{\partial L}{\partial W}=(p-y_{\text{oh}})x^T ∂ W ∂ L = ( p − y oh ) x T
W t + 1 = W t − η ⋅ ∇ W L ( W t ) W_{t+1}=W_{t}-\eta \cdot \nabla_{W}L(W_{t}) W t + 1 = W t − η ⋅ ∇ W L ( W t ) η \eta η
λ \lambda λ λ m a x \lambda_{max} λ ma x λ m i n \lambda_{min} λ min c o n d ( H ) = ∣ λ m a x ∣ ∣ λ m i n ∣ cond(H)= \frac{\mid \lambda_{max}\mid}{\mid \lambda_{min \mid}} co n d ( H ) = ∣ λ min ∣ ∣ λ ma x ∣ c o n d ( H ) cond(H) co n d ( H ) D D D D D D
v t + 1 = ρ v t + ∇ L ( W t ) W t + 1 = W t − α v t + 1 \begin{aligned}
v_{t+1} &= \rho v_{t} + \nabla L(W_{t}) \\
W_{t+1} &= W_{t} - \alpha v_{t+1}
\end{aligned} v t + 1 W t + 1 = ρ v t + ∇ L ( W t ) = W t − α v t + 1 ρ \rho ρ
η \eta η w i w_{i} w i
c a c h e = d e c a y ⋅ c a c h e + ( 1 − d e c a y ) ⋅ ( ∇ W ) 2 cache=decay \cdot cache + (1-decay) \cdot (\nabla W)^2 c a c h e = d ec a y ⋅ c a c h e + ( 1 − d ec a y ) ⋅ ( ∇ W ) 2 W = W − η c a c h e + ϵ ∇ W W=W-\frac{\eta}{\sqrt{ cache + \epsilon }}\nabla W W = W − c a c h e + ϵ η ∇ W
m m m v v v
m m m v v v β 1 \beta_1 β 1 β 2 \beta_2 β 2 m m m v v v
m ^ t = m t 1 − β 1 t v ^ t = v t 1 − β 2 t \begin{aligned}
\hat{m}_{t} &= \frac{m_{t}}{1-\beta_{1}^{t}} \\
\hat{v}_{t} &= \frac{v_{t}}{1-\beta_{2}^{t}}
\end{aligned} m ^ t v ^ t = 1 − β 1 t m t = 1 − β 2 t v t t t t β t ≈ 0 \beta^t ≈0 β t ≈ 0 t t t β t \beta^t β t m m m v v v
w t + 1 = w t − η ⋅ m ^ t v ^ t + ϵ w_{t+1}=w_{t}-\eta \cdot \frac{\hat{m}_{t}}{\sqrt{ \hat{v}_{t}} + \epsilon} w t + 1 = w t − η ⋅ v ^ t + ϵ m ^ t
N × N N\times N N × N
