We've just released a new feature: maintenance windows! You can now place your site in maintenance which will mute all notifications during that period.