Most existing arbitrary shape text detection methods employ connected components and text center lines for grouping text instances, which assume that texts in adjacent positions belong to the same instance. However, many hard-to-group scene texts are too complex to be effectively processed in this way. To address this challenge, we propose a novel scene text-spotting method that utilizes feature-based clustering inspired by human cognitive principles of text perception. Our approach involves first utilizing a character spotter to obtain the location and the transcription information of the characters. Then, a lightweight recognition network extracts the visual features of the characters by their locations. These visual features are then grouped into instances through a K-means-fuzzy-net, which explicitly model visual feature similarity to effectively group the nested text, the large-margin text, the continuous text, and the one with overlapping characters. Finally, the recognition results of text instances are processed by a word correction module to improve the overall accuracy and reduce the vulnerability of individual character detection. Additionally, we have contributed a hard-to-group text dataset. Experiments demonstrate the state-of-the-art performance of our method in addressing scenarios. Hard-to-group text dataset is available at: https://github.com/baggio321/Hard-to-Group-Text-Dataset.