The d-pad and lack of backlight do kinda suck- but you get almost forty hours of playtime on 1 AA battery. Funny thing is you can pick up a Swancrystal new for $20-30 shipped, but some of the best games are $50+( Judgement Silversword, Dicing Knight, Run=Dim, Makimira. For good games in the $10-30 range I would recommend Gundam UC, Gundam Seed, Final Lap Special, Star Hearts, and Guilty Gear. I've also heard good things about Front Mission and both Rockman titles.
I've heard bad stuff about the GP2X- cheaply made, technical issues. I would love to have a handheld that can do what it does with emulators, but there's a lot of horror stories out there.