The Dynamical Mordell-Lang Conjecture for skew-linear self-maps. Appendix by Michael Wibmer