China is an agricultural country with the largest population in the world. However, intensification of droughts and floods has substantial impacts on agricultural production. For effective agricultural disaster management, it is significant to understand and quantify the influence of droughts and floods on crop production. Compared with droughts, the influence of floods on crop production and a comprehensive evaluation of effects of droughts and floods are given relatively less attention. The impact of droughts and floods on crop production is therefore investigated in this study, considering spatial heterogeneity with disaster and yield datasets for 1949-2015 in China mainland. The empirical relationships between drought and flood intensity and yield fluctuation for grain, rice, wheat, maize and soybean are identified using a Bayesian hierarchical model. They are then used to explore what social-economic factors influenced the grain sensitivity to droughts and floods by the Pearson's coefficient and locally weighted regression (LOSEE) plots. The modeling results indicate that: (a) droughts significantly reduce grain yields in 28 of 31 provinces and obvious spatial variability in drought sensitivity exists, with Loess Plateau having highest probability of crop failure caused by droughts; (b) floods significantly reduce grain yield in 20 provinces, while show positive effect in the northwestern and southwestern China; (c) the spatial patterns of influence direction of droughts and floods on rice, maize and soybean are consistent with the grain's results; and (d) promoting capital investments and improving access to technical inputs (fertilizer, pesticide, and irrigation) can help effectively buffer grain yield lose from droughts.