It is increasingly acknowledged that climate change is influencing terrestrial ecosystems by increased drought and rainfall intensities. Soil microbes are key drivers of many processes in terrestrial systems and rely on water in soil pores to fulfill their life cycles and functions. However, little is known on how drought and rainfall fluctuations, which affect the composition and structure of microbial communities, persist once original moisture conditions have been restored. Here, we study how simulated short-term drying and re-wetting events shape the community composition of soil fungi and prokaryotes. In a mesocosm experiment, soil was exposed to an extreme drought, then re-wetted to optimal moisture (50% WHC, water holding capacity) or to saturation level (100% WHC). Composition, community structure and diversity of microbes were measured by sequencing ITS and 16S rRNA gene amplicons 3 weeks after original moisture content had been restored. Drying and extreme re-wetting decreased richness of microbial communities, but not evenness. Abundance changes were observed in only 8% of prokaryote OTUs, and 25% of fungal OTUs, whereas all other OTUs did not differ between drying and re-wetting treatments. Two specific legacy response groups (LRGs) were observed for both prokaryotes and fungi. OTUs belonging to the first LRG decreased in relative abundance in soil with a history of drought, whereas OTUs that increased in soil with a history of drought formed a second LRG. These microbial responses were spread among different phyla. Drought appeared to be more important for the microbial community composition than the following extreme re-wetting. 16S profiles were correlated with both inorganic N concentration and basal respiration and ITS profiles correlated with fungal biomass. We conclude that a drying and/or an extreme re-wetting history can persist in soil microbial communities via specific response groups composed of members with broad phylogenetic origins, with possible functional consequences on soil processes and plant species. As a large fraction of OTUs responding to drying and re-wetting belonged to the rare biosphere, our results suggest that low abundant microbial species are potentially important for ecosystem responses to extreme weather events.