Bluebirds are meticulous when it comes to keeping a clean nest. They remove all of the shell and take it away from the nest cavity/box so as not to attract predators. They will also remove the fecal sacks that the chicks will produce to keep the nest clean.
I'm going to let them be for a couple of days until all the eggs hatch. We're expecting rain tomorrow and much colder temps on Sunday and I don't want to spook Mrs. Blue into leaving them unprotected and cold. I'd worry too much.