Ultra-small all-optical switches are of importance in highly integrated optical communication and computing networks. However, the weak nonlinear light-matter interactions in natural materials present an enormous challenge to realize efficiently switching for the ultra-short interaction lengths. Here, we experimentally demonstrate a submicron bidirectional all-optical plasmonic switch with an asymmetric T-shape single slit. Sharp asymmetric spectra as well as significant field enhancements (about 18 times that in the conventional slit case) occur in the symmetry-breaking structure. Consequently, both of the surface plasmon polaritons propagating in the opposite directions on the metal surface are all-optically controlled inversely at the same time with the on/off switching ratios of >6 dB for the device lateral dimension of <1 mu m. Moreover, in such a submicron structure, the coupling of free-space light and the on-chip bidirectional switching are integrated together. This submicron bidirectional all-optical switch may find important applications in the highly integrated plasmonic circuits.