Automatic detection of fabric defects is an important process in the textile industry, which is required to locate and classify microdefects from a large fabric image. We propose a learning-based system for automatic detection of microfabric defects. A segmentation algorithm based on fractal and gray features is applied to extract microdefect regions. Gabor fractal network is designed to further improve identification ability of this approach. The proposed network achieves superior performance in terms of detection accuracy with a much smaller model size. The best testing accuracy rates on dark line, hole, broken yarn, and dirt are 96.9%, 98.0%, 92.9%, and 98.8%, respectively. Experimental results demonstrate the effectiveness of the proposed scheme in defect detection for microfabrics. The proposed system has great potential for automatic detection of microfabric defects.