Automatic optical inspection technology (AOI) is a visual inspection technology that has developed rapidly in recent years. The high speed and accuracy of AOI can greatly enhance the efficiency of modern industrial production. However, when this technology is applied to optical components inspection, it encounters a challenge that the specular highlight induces over or under exposure during the imaging process, and further results in a low imaging contrast and unclear defect details of the targets. To solve this problem, a dark-field polarization imaging setup based on a division-of-focal-plane polarization camera was adopted to achieve high contrast defect images. Meanwhile, algorithms based on the improved LeNet-5 convolutional neural network were developed to recognize the defects. An accuracy above 99.5% was obtained for the distinction of defective and non-defective samples, and an accuracy of 94.4% was reached for the various defect classification. Our work demonstrated an effective application of polarization imaging and machine learning in AOI of optical components manufacturing.