It's tricky to do unguided. You're Likely to leave out a lot of needed nutrients and end up malnourished in the long term. If you can't work with either a very knowledgeable person who has been guided by a licensed nutritionist, or a licensed nutritionist, I would not suggest it.
And, no it will be more likely to lower your testosterone. Plants like soy contain phytoestrogen.