Rayleigh wave inversion is a key method for obtaining the shear wave velocity structure. However, the current traditional inversion method has problems such as relying on the initial model, easy to fall into the local extreme value, and difficult to calculate the Hessian matrix. At the same time, the difference between the Rayleigh wave phase velocity and polarization rate dispersion curves on the sensitivity of the velocity structure parameters limits the accuracy of a single inversion result. Due to the certain characteristics of orderliness and ergodicity of shear wave velocities in various strata, the Markov decision algorithm is thus adopted to simulate all underground structural situations, and the velocity structure is expressed precisely through the adaptive layering method proposed in this paper. On this basis, finally, the dispersion curves of phase velocities and polarizability derived from the forward modeling of the stratum structure are fused in characteristic segments to construct a sample data set; Considering that the dispersion curves of phase velocity and polarizability corresponding to the formation velocity structure possess certain temporal characteristics, a convolutional neural network concatenated with long short-term memory recurrent neural network was built as the backbone network model for joint inversion; After the completion of the model construction, comprehensive and systematic supervised training was carried out on this network. Eventually, a model that can meet the actual requirements and has a relatively good inversion effect was obtained.The model test shows that the relative errors of the joint inversion results are all reduced compared with the single inversion results; the addition of noise in some sample data not only effectively improves the model inversion accuracy but alsoincreases the generalization performance of the model. In practical applications, the method proposed in this paper is applied to the measured data of the 2008 Wenchuan earthquake in Hongkou Shenxigou, Dujiangyan City, Sichuan Province, and good inversion results are obtained, which provide scientific evidence for the localization effect of "co-seismic deformation".