There have been exemplar clarinet players (Lazarus, for one) who insisted that their synthetic body (ebonite, in his case - a mixture of rubber, sulphur, carbon black and elemental lead) clarinets were superior to any others that they had played. So, you are not alone in your opinion, and there is some weight behind it as well.
I strongly suspect that the quality of manufacture is a more significant factor, and that the wooden clarinets have more attention paid to all factors during the course of their fabrication, mostly because it is expected of a professional quality instrument. Push comes to shove, the wooden body only make up a small (mind you, significant, but still small) portion of the price of the finished instrument. Labor accounts for most of the price of what you end up buying.
It is quite possible to get a bummer of a wood clarinet, but I have yet to encounter one in the lines that I have purchased (Selmer and Buffet).
I've not kept up on things in this area, but I was told once that Bundy student horns were moulded to their final internal dimensions, while Vito horns were reamed (with a long, slightly conical, machine steel cutter) to theirs. And, that might have accounted for the following:
I have owned a plastic clarinet (a Vito, I am ashamed to say) that I bought for about $20.00 with the intention of turning it into a lamp. However, it had the sweetest tone of any horn that I have ever owned. Other than some intonation issues here and there (and the fact that it was the product of Leblanc, the great Satan of clarinet manufacturing), it would have made an excellent everyday horn. I ended up selling it to a broke sax player who used to let me use his soprano whenever it was time to play When I'm Sixty-Four with a community orchestra. I sold it to him for what I paid for it, nice guy that I am.