Theoretical Knowledge Points of "Improving Deep Neural Networks"

### Practical Deep Learning and Optimization - **Dataset Splitting**: A common split ratio is 98% for training, 1% for validation, and 1% for testing. Increasing data volume or applying regularization can improve model performance. Validation and test sets should be from the same distribution. Adjusting regularization parameters helps reduce overfitting. - **Optimization Algorithms**: Mini-batch gradient descent is faster than full batch processing; the ideal mini-batch size ranges between 1 and m. Exponential weighted averages are used to track data changes; learning rate decay techniques like \(0.95^t \alpha_0\) and \(\frac{\alpha_0}{\sqrt{t}}\) are effective. Adam combines the advantages of RMSProp with momentum. ### Hyper