Ground penetrating radar is an effective method for detecting masonry retaining walls. At present, the common method for numerical simulation of masonry retaining walls is to treat the wall as a uniform material and establish a uniform model or establish a heterogeneous model of regular blocks. Most research on the image features of internal anomalies in the retaining wall focuses on the images generated by the discontinuous surface between the anomaly and the wall, with less research on the background noise generated by the retaining wall. In response to these issues, this study proposes a modeling method for heterogeneous mortar stone retaining wall numerical models based on the Voronoi diagram algorithm. Through field experiments and the open source software gprMax, radar detection and numerical simulation of complete retaining wall and anomalies such as voids, loose areas, aquifer regions, and rebar in the retaining wall are carried out to study the electromagnetic response characteristics of mortar stone retaining walls, the main factors affecting radar image, and the image features of internal anomalies in the retaining wall. The experimental and simulation results show that the simulation results of the proposed non-uniform barrier numerical model are basically consistent with the measured radar response characteristics, which can provide some guidance for the interpretation of retaining wall radar images. The main factors affecting the radar image in the wall are the size and shape of the stones, the thickness and relative permittivity of the mortar, and the antenna frequency. The same phase axis of reflection wave about the stone in the background wave tends to be consistent with the shape of the stone. When the average area of the stone is larger, it is necessary to distinguish between the background wave and some anomaly reflection waves. The larger the difference in thickness and relative permittivity of the mortar, the higher the amplitude of the background wave. The higher the antenna frequency, the better the resolution, but the shallower the detection depth. The radar images of voids, aquifer regions, and rebar are continuous curve waveforms on the same phase axis, and their amplitude is higher than the background wave of the surrounding wall. The radar image of the loose area is a chaotic lump waveform on the same phase axis, and its amplitude is close to the background wave of the surrounding wall.