Were seriously spooked by the deer, one or more will do. I've seen different interactions between deer and cats, some cats hiss at the deer and the freak and run off or the cat(s) run to a safe location and observe. A deer will chase a small animal if it decides to either play chase or seriously neutralizing a threat. I have seen these things all take place between deer and cats.
They will all come out of hiding, some will recognize the security your place affords them and keep that info on their minds.
Your going out and acting normal helps them recover from their fears. You were probably subconsciously sensing that the cats were stressing, they may have been telepathically trying to get your attention and that's why your were anxious when you awoke. They are dialed in to you and your role as protector. I always know when one of my cats is in a situation where they are in possible danger, sentient beings share that trait, it's a natural phenomenon.
The kitties have learned a lesson about bigger animals but you might do best by just going about things as usual so they can see that everything is okay and they can go about growing and being cute. Later in the day they will all have checked in.