In the process of reconstructing the Digital Elevation Model (DEM) based on the TerraSAR-X/TanDEM-X bistatic system, there are a large number of spatially distributed random noises in the interferograms due to the influence of geometric distortion and incoherence factors. The existing methods exhibit filtering waves or weak filtering phenomena, which limits the reconstruction accuracy of TerraSAR/TanDEM-X DEM. Based on this, this paper proposes a PDV-Goldstein filtering method that combines the phase derivative variance and the Goldstein filtering method. The improved filtering method calculates the phase derivative variance of the interferogram. It uses the prior filtering factor to adaptively filter the regions with different intensities in the interferogram, so as to improve the filtering effect of the interferogram and the quality of subsequent DEM reconstruction. In this paper, simulated and real data are used to compare experiments with various filtering methods. The results show that the filtered interference pattern obtained by the PDV-Goldstein filtering method can significantly maintain the phase detail information. The residual point reduction rate is 87.83%, which is 13.92% higher than that of the traditional Goldstein filtering method. It can be proved that the improved PDV-Goldstein filtering method has superior performance in noise suppression and phase detail preservation, which can support the subsequent DEM reconstruction work.