With DC, VGA (480p) is worth it for 3D games for sure. Using low-res RGB on DC is a bit weird, as some games do not have a 15khz RGB mode. Meaning they output in Comp, S-Vid, VGA, but NOT RGB.
As for the X68K, depends a lot on your monitor. If you have an analog CRT computer monitor, it looks quite nice to me even in 1600x1200. If you are using the digital output, I think most people prefer to output in 640x480 and throw a scanline generator on the chain. If you want to be a purist, the 15khz games should have scanlines, while the 24khz and 31khz shouldn't. But in the end, just play around until you get a picture you like. Matter of preference really.