Dissecting the Failure of Invariant Learning on Graphs