There is anecdotal evidence that men who wear boxer shorts may father more children than those who wear briefs. The lowered fertility of briefs-wearers is termed the "Jockey Shorts Effect." The cause is thought to be a decreased rate of spermatogenesis when the testes are held too tightly against the body by briefs. They're too close to the fire, so to speak, and the temperature rises above optimum. In recent years this idea has come into some disrepute, and it's not one of those issues that burns to be settled. The chances of someone landing a big NIH grant for a long-term epidemiological study of the relative fertility of the two groups of men is slim. It's possible the Jockey Shorts Effect is just another Urban Legend.
Among land animals, the elephant retains its testes intra-abdominally, so obviously elephant spermatogenesis has a different temperature optimum than that of the body core for elephants. Many seasonal breeding small mammals have intra-abdominal testes in the "off season" but the testes descend for the breeding season. I'm unaware of any other land mammals than the elephant that normally have intra-abdominal testes, but of course that's the case in marine mammals.