Test-time adaptation (TTA) aims to adapt the model trained on source domain to unseen target domain using a few unlabeled images during inference, which holds great value for the deployment of models in the clinical practice. In this setting, the model can only access online unlabeled test samples and pre-trained model on the source domain. Because unlabeled test samples may arrive sequentially, the model needs to adjust online for the cross-domain distribution shift from different medical institutions, the scale of which would change concurrently and continually over time. However, unstable optimization and abnormal distribution will lead to error accumulation and catastrophic forgetting. Considering the role of brain extracellular space in balancing neural homeostasis and signal transmission, we recognize that the existing TTA methods lack a dedicated component to ensure the stability and accuracy of the model. In this paper, we propose a robust TTA approach for cross-domain segmentation as MemTTA. Specifically, firstly, we introduce transductive batch normalization to ensure stability, which calculates the mean and the variance from the source domain and current test batch. Secondly, we propose a memorized spatial pixel-level clustering strategy to represent each category with multiple and anisotropic prototypes for feature alignment, which can be associated with the parametric classifier. During test time, we adapt the segmentation model to each test batch with self-supervision augmentation consistency learning to improve the inference performance. MemTTA needs only one epoch training on each test batch, and then is comparable to standard models as the traditional inference pipeline. The proposed method is extensively evaluated on neuron, brain metastases, cardiac, and abdominal organ image segmentation. The experimental results demonstrate that our proposed MemTTA can effectively mitigate test-time domain shift and catastrophic forgetting, and is superior to existing state-of-the-art approaches.