What the Step Size Really Means
Series: Stochastic Approximation from First Principles · Part 4 of 22 In the update $$ w_{k+1}=w_k+\alpha_k(\text{target}_k-w_k), $$what does $\alpha_k$ actually represent? Separate Direction from Distance Treat the update as a vector movement: $$ w_{k+1} =w_k+ \underbrace{\alpha_k\|\text{target}_k-w_k\|}_{\text{actual distance moved}} \cdot \underbrace{ \frac{\text{target}_k-w_k} {\|\text{target}_k-w_k\|} }_{\text{unit direction}}. $$The vector $\text{target}_k-w_k$ contains both a direction and a distance: $\|\text{target}_k-w_k\|$ is the distance suggested by the latest observation. $\alpha_k\|\text{target}_k-w_k\|$ is the distance the update actually moves. $\alpha_k$ is the fraction of the suggested movement that the update accepts. In the river analogy, you walk an $\alpha_k$ fraction of the estimated distance toward the location implied by the current sound. The step size is therefore both a movement ratio and a degree of trust in the latest estimate. ...