I have thought about this before sometimes. The two strongest forces in evolution are preference and death. While environmental factors may play a smaller role, I would sooner hypothesize that the evolution of the penis it comes down to
choosing power (preference).
What I mean by choosing power is simply who has the final say in who fucks whom. In a matriarchal society, women have the choosing power. In a patriarchal society, men 'take' the women. There are many varieties of this, but I see the strongest correlation of penis size to geography tied up in social systems like this.
In China and India there is a long history of arranged marriages, so the women have no power to express their preference. Asia is fairly homogenously patriarchal through history, with a few exeptions: Samoa, Hawaii and some other Pacific Islands are known to have matriarchies. Guess what? There have been cases of huge dicks from there! I'm sure Red will back me up here.
Arranged marriages occur in Europe too, but Europe is more of a mix of social structures if you look at the details of history over the past 2000 years or so. Britain used to be an entirely matriarchal society during the druidic times (think Stone Henge). The mixed and 'medium' size of dicks from Europe reflects the history of social structures there.
In Africa there is a huge range of tribal societies, many matriarchal, many patriarchal. There is a range of dick sizes coming out of Africa, and a few monster sized ones which I think gives them the reputation. I think this is because the tribal matriarchies there are very old, and so the women there have gone through many many generations choosing the largest cocks. I think the heat there has more to do with the clothing than with how low the dick hangs as opa suggests. If that were the case, Africans growing up in Minnesota would have small dicks (not the case). Clothing hides or reveals the body. Walking around naked in Africa with your dick hanging out takes a lot of guesswork out of the woman's job, so dick size is more of an initial decision factor, rather than a later factor as it is in our society (due to clothes). In this way, a society that doesn't wear clothes allows women to more quickly find her preferred dick size in a group of men.
So I think social structure has more to do with dick size than the immediate environment, but the environment can have it's own effects on social structure. So from that perspective I would say that wherever you go in the world where women have had a lot of choosing power through history, you will likely find big dicks.