Despite the toxicity of mercury (Hg) for organisms in the environment, little is known on its impact on the soil microbiome, especially its chronic effect. Here, we assessed the effects of a long-term contamination of Hg in soils on the bacterial and fungal communities along a gradient of contamination from no to high contamination. Short-term reactions (30 days) of the microbial communities in these soils having different levels of historic Hg contamination were further evaluated in microcosm experiments where soils were either spiked with dissolved HgCl2 or not. Results show a clear impact of a long-term Hg contamination on both bacterial and fungal community structures and diversity but only a weak effect was observed on their activities (basal respiration and growth rates). No short-term effects of Hg were observed on the microbial community structures and activities. Taxa from the Chthoniobacteraceae (bacteria) and Trichosporon sp. (fungi) were associated with high Hg contaminated soils, implying they possess capabilities to tolerate Hg in soils. Abundance of mercury reductase (merA) gene copies increased with higher Hg concentrations in soils both during short and long-term exposure to Hg pointing to potential mechanisms within microbial cells to tolerate higher amounts of Hg in soils.