I almost forgot: Last Gladiators Digital Pinball! Cheap and awesome.
\m/
Most of the essential stuff has been mentioned.
Digital Pinball Necronomicon is also wicked awesome but costs a bit more. I think it's better, but if you are only buying one and are trying to keep the budget right, Last Gladiators still should be less than $10 shipped for the JP version.
Some of my top favourites, and I know I am mostly just reiterating games already listed in this thread.
Magical Drop II/III
Puzzle Bobble 2/3
Magic Knight Rayearth (BURN IT, as others have said)
Dragon Force (this is also pretty much burn territory these days I think)
Dungeons & Dragons
Sega Worldwide Soccer 98 (if you like sport games this one is pretty fun and simple, lots of modes iirc)
Sega Rally
Daytona
Saturn Bomberman
There are a lot of great shooters and NoPepper's post is actually a great starting point. I've played just about all of them and certainly DDP, Silvergun and Garegga are in the top 5 on Saturn. Layer Section is excellent too. I'm also really high on Soukyugurentai. Make sure you get the fixed version. Those are all pretty much true Saturn shooters. The Saturn also has a crap ton of compilations of older shooters.
Game that sucks don't play it
Albert Odyssey