RealFlight does not support antialiasing via internal settings. You'll need to use the NVidia control panel to turn it on (and it sounds like you've gotten started there). Once you've done that correctly it does work.
The main thing to know is that you'll need to tell it to override the application settings for certain things. The basic antialiasing and anisotropic filtering, for example, are set to "application-controlled" by default.
You can either do that at the global level, which will affect everything, or by adding RealFlight to the list of programs and tweaking the settings just for it.
In the attached screenshot I have forced antialiasing and anisotropic filtering on as well as cranked up a couple other settings.
Oh, and make sure to actually apply the settings changes!