As a key geological factor affecting coal mine safety production, the accurate identification of fault structure is crucial for hazard prevention and efficient mining during coal exploration and development. With the enhancement of exploration and development of deep coalbed methane, the requirements for fault identification in underground coal-bearing strata are getting higher and higher. Therefore, this paper carries out research on edge detection technology for seismic faults. Based on the analysis of previous research results, a novel edge detection method using an LBP/VAR composite operator is proposed, which combines rotation invariant Local Binary Pattern (LBP) and Rotation Invariant Variance (VAR). Firstly, a theoretical geological model with continuous variation of vertical fault displacement is established, and different operators are used to test the result of forward modeling, which proves that the recognition accuracy of LBP/VAR operator is better. then, different intensities of noise are introduced to simulate the real seismic acquisition environment. Then, the noise with different signal-to-noise ratios is introduced. By comparing with the rotation invariant local binary pattern and the traditional edge detection operators such as Canny and Roberts, the stability of the LBP/VAR operator under noise interference is verified. Subsequently, the proposed method is applied to the real coalfield seismic data, and compared with the detection results of conventional seismic attributes, traditional edge detection operators and rotation invariant local binary pattern. The results show that the LBP/VAR operator has the best recognition effect, which is basically consistent with the fault edge in the original image. The recognition effect of the Canny operator is better than that of the Roberts operator. The experimental result show that the LBP/VAR operator shows significant advantages in fault recognition accuracy and noise immunity through the collaborative analysis of texture features and contrast. Especially under complex geological conditions with low signal-to-noise ratio of the seismic data, this method can still maintain high accuracy for fault boundary location. The study result can provide reference for the qualitative identification of coal seam faults, and have certain guiding significance for future exploration and development in coal seam.