A Tale of Two Portals: Testing Light, Hidden New Physics at Future $e^+ e^-$ Colliders