A hybrid shaping (HS) scheme based on geometric shaping (GS) and probabilistic shaping (PS) in a coherent optical communication system is proposed. A particle swarm optimization algorithm and Maxwell-Boltzmann distribution are employed to sequentially implement GS and PS. The results demonstrate that hybrid shaped 8/12-ary quadrature amplitude modulation (HS-8/12QAM) is superior to regular-8/12QAM (R-8/12QAM) in terms of reducing the bit error rate (BER) and increasing the generalized mutual information (GMI). HS-8QAM achieves a 2 dB optical signal-to-noise ratio (OSNR) gain and 0.45 bits / symbol GMI gain compared with R-8QAM. Meanwhile, HS-12QAM achieves 1.9 dB OSNR gain and 0.68 bits/symbol GMI gain compared with R-12QAM. In addition, HS-8/12QAM is better than R-8/12QAM in terms of transmission distance and data rate.