The optimization and uncertainty analysis of laser-based optical sensors in the design phase is a challenging task due to the presence of stochastic laser speckle effects. We present an accurate, efficient, and versatile simulation framework for the design of optical sensor assemblies, capable of handling objective as well as subjective speckle effects. The framework integrates the stochastic nature of laser speckle with the deterministic properties of ray-tracing simulations, enabling the simulation of sensitivities to translational as well as rotational target motion and reliable performance estimation, even for more complex optical assemblies. To validate the simulation results for translation and rotation, they are compared against the experimental data of four speckle-based optical sensor assemblies as well as against analytical relations for speckle pattern motion. The accuracy of the developed framework is demonstrated by simulation errors for correlation peak shift of the speckle pattern of less than 2  μm rms and 2.4  μm rms for translation and rotation, respectively. For the center of gravity shift as additional simulation output for an integrated laser sensor for sensing translations in all three degrees of freedom, a simulation error of 2.6  μm rms was obtained, which also lies well below the resolution of the designed optical sensor assemblies.