The contact damping between rough surfaces has an important influence on the wear, vibration, contact fatigue, and energy dissipation between interfaces. In this paper, based on contact theory, a tangential damping mathematical model of rough surfaces is established from the point of view of viscous contact damping energy dissipation mechanism of asperities and considering the fractal characteristics of three-dimensional topography of rough surfaces. Through the combination of micro-contact modeling and macro dynamic testing of composite beams, the analysis results show that there are important evolution rules between tangential damping and surface fractal parameters and material parameters. The nonlinear relations between them are as follows: tangential contact damping is positively correlated with normal load, load ratio, and maximum contact area of asperity, and negatively correlated with fractal roughness; tangential contact damping increases first and then decreases with the increase of three-dimensional fractal dimension. The results of computational and experimental modal analysis show that the established mathematical model is feasible for predicting tangential damping. The study of tangential contact damping between surfaces can lay a foundation for improving the performance of assembly interfaces.