Computational Graphs

• Can represent any function
  • node = step of computation
  • no matter how complex the function is

Advantage

• Can apply backpropagation
  • recursively use chain rule to compute the gradient with respect to every variable
• Especially useful when using complex models
  • ex) Convolutional Network

Backpropagation

How it's done

1. Forward pass computation first
   • Give all intermediate variable names
     • q : $x + y$
     • f : $qz$
2. From the back apply chain rule backwards to all paths
3. Calculate gradient of all variables
   • $\frac{\partial f}{\partial x},\frac{\partial f}{\partial y},\frac{\partial f}{\partial z}$

• What's exactly happening
  • Each node have two inputs and an output
    • • local gradient
  • during backpropagation:
    • each nodes get upstream gradients coming back
    • compute only the gradient of two inputs
    • give computed gradients to previous node(s)
• when a node is outputted to multiple nodes
  • gradients are added
• If x, y, z are vecotrs
  • works the same
  • local gradients are represented as Jacobian matrix
    • Jacobian matrix = diagonal matrix
  • gradient of a vector is always same size as original vector

Advantage

• Simpler than using complex calculation to compute gradients

Tips

• backpropagation of
  • addition node will end up multiplying one
    • no change
  • multiplication node will end up multiplying other input to the upstream value
  • max gate
    • greater input: upstream gradient value
    • smaller input: 0