Manas Narkar provides some advice on optimizing neural network models:
Adding Dropout
Dropout is considered as one of the most effective regularization methods. Dropout is basically randomly zero-ing or dropping out features from your layer during the training process, or introducing some noise in the samples. The key thing to note is that this is only applied at training time. At test time, no values are dropped out. Instead, they are scaled. The typical dropout rate is between 0.2 to 0.5.
Click through for a demo on dropout, as well as coverage of several other techniques.