Gradient Descent & Backprop

What You'll Cover Today

Day 5 of Calculus for AI in 5 Days brings everything together. You'll synthesize what you've built across the week into a complete, working implementation. This is the hardest day — and the most satisfying.

SGD

Understanding SGD is the core goal of Day 5. The concept is straightforward once you see it in practice — most confusion comes from skipping the mental model and jumping straight to implementation. Start with the model, then write the code.

# SGD — Working Example
# Study this pattern carefully before writing your own version

class SGDExample:
    """
    Demonstrates core SGD concepts.
    Replace placeholder values with your real implementation.
    """
    
    def __init__(self, config: dict):
        self.config = config
        self._validate()
    
    def _validate(self):
        required = ['name', 'type']
        for field in required:
            if field not in self.config:
                raise ValueError(f"Missing required field: {field}")
    
    def process(self) -> dict:
        # Core logic goes here
        result = {
            'status': 'success',
            'topic': 'SGD',
            'data': self.config
        }
        return result


# Usage
example = SGDExample({
    'name': 'my-implementation',
    'type': 'sgd'
})
output = example.process()
print(output)

Adam

Adam is the practical application of SGD in real projects. Once you understand the underlying model, Adam becomes the natural next step.

backpropagation

backpropagation rounds out today's lesson. It connects SGD and Adam into a complete picture. You'll use all three concepts together in the exercise below.

Common Mistakes on Day 5

• Skipping the fundamentals — SGD requires understanding the underlying model before you can apply it correctly. Read the section twice if needed.
• Ignoring error messages — error messages for Adam are usually precise. Read them carefully before searching online.
• Hard-coding values — anything that might change between environments belongs in configuration, not in your source code.
• Not testing edge cases — the happy path is not enough. What happens with empty input? With unexpected types? With network failures?