We derive the phonon dynamics of magnetic metals in the presence of strong spin-orbit coupling. We show that both a dissipationless viscosity and a dissipative viscosity arise in the dynamics. While the dissipationless viscosity splits the dispersion of left-handed and right-handed circularly polarized phonons, the dissipative viscosity damps them differently, inducing circular phonon dichroism. The effect offers a new degree of manipulation of phonons, i.e., the control of the phonon polarization. We investigate the effect in Weyl semimetals. We find that there exists strong circular phonon dichroism in Weyl semimetals breaking both the time-reversal and the inversion symmetry, making them potential materials for realizing the acoustic circular polarizer.