Aiming at the subtle differences between fabric defects and complex texture background and the problem of small size defects, we propose a fabric surface defect recognition method based on multibranch residual network. First, the convolution kernel is decomposed and the multibranch shared feature extraction module is constructed using the residual connection to mine the feature information at different scales. At the same time, squeeze-and-excitation network (SE-Net) with a reduction ratio of 16 is added to the module to suppress the interference of complex texture background. Second, the channel attention mechanism is embedded in the residual block of the network backbone to realize the compensation for the lost feature information. Finally, the swish activation function is used to enhance the accuracy and robustness of the deep network, and transfer learning is used to further improve the network accuracy. The experimental results show that the proposed model is superior to the existing model in terms of average recognition accuracy. The proposed model has the highest recognition accuracy of 96.17% for eight defect categories, which proves the effectiveness of fabric surface defect class recognition.