Theory of nonlinear Landau-Zener tunneling