ML Prep

Full gradient versus stochastic gradient

Problem

For losses ℓ1(w)=(w−1)2\ell_1(w)=(w-1)^2 and ℓ2(w)=(w+1)2\ell_2(w)=(w+1)^2, compute the full empirical gradient at w=2w=2 and each per-example gradient. Explain why an unbiased uniformly sampled stochastic gradient can still have variance.

Reveal answer or reference solution

The per-example gradients are 22 and 66. The mean, or full, gradient is 44. A uniformly sampled per-example gradient has expectation 44 but takes values 22 or 66, so it has nonzero variance.

Local history

Loading attempts saved in this browser…

    Use with your agent

    Share this URL and your attempt. Ask the agent to start with a clarifying question or the smallest useful hint.

    Tutor me on https://mlprep.iwase.dev/item/original-ml-sgd/. If window.mlPrepAgent is available, read attempts for item original-ml-sgd before tutoring. Inspect my attempt, keep the item ID, and do not reveal the full answer first. After a real attempt, append its record with recorded_by agent, include agent_session_id when available, and read it back.

    Appears in