I grew up on the Atari 5200 controller and had a shitty Apple II C+ joystick, and I honestly adapted to them quite well. To me the 5200 controller bashing is just a fad, I honestly don't think it is that bad.
Honestly, the two controllers that confuse the hell out of me are:
1.) Virtual Boy - The two d-pad set up always messed me up. When I use the controller, somehow my hands go all over the place, and I just can't find any comfort in it.
2.) Wii - The remote only controller for the Wii works very well, both for motion games and also for horizontally controlled classic style gaming. Then some games ask for you to attach the nunchuk, and this is when it all goes to hell. I am too uncoordinated to move both of my arms in specific directions. When I play Zelda and have to do a spin attack, that movement just isn't natural to me. As much as I enjoy the new Zelda game, I just struggle so much when I have to roll stuff and get my arms moving in various ways. It reminds me of the short time period I tried to play drums as a kid, and it was just horribly difficult and I gave up.
**EDIT**
As soon as I hit submit, it came to me that I really dislike another controller. The Sega Master System d-pad is a pile of garbage. Since there is no definition in the directions on the pad, I always end up hitting diagonal when I just want to go straight. In games where holding up is required, this can be a big issue.
Also, although I like the actual feel of the controller, special mention needs to go out to the Vectrex, as the sticks on them were incredibly fragile.