Men make estrogen by aromatizing testosterone. If your body thinks its getting to much testosterone it can aromatize more as a way of limiting testosterone. For men it varies the levels though. Men with high levels of testosterone tend to have healthier hair and nails (estrogen receptive parts of body). In fact one reason men lose hair is because testosterone drops and less is available to be aromatized.
Estrogen is even very anabolic. Young women tend to have nice muscle tone because of high estrogen not low estrogen. It is also lipogenic though and adds fat. Many body builders don't take estrogen aromitizing prohibitors because they dont' want to lose the anabolic affects of the estrogen. Plus it plays a lot of vital roles in our bodies. One time for a week I took an anti estrogen aromitizing supplement and I had a lot of side effects from it. Like dandruff, when I'd never had dandruff before. I quit it and everything went back to normal, but thats when I really started appreciating my seemingly very healthy estrogen levels.
Estrogen is pretty healthy for you, I doubt it would cause anxiety. Acer may be right though that yohimbe can give an increased likelyhood of anxiety. As with most things it might affect some men really badly, and other not at all. If you are noticing increased anxiety while taking them it probably is them. However if you were scared you would be anxious if you took them, then did get anxious when you took them it most likely is psycho-somatic.