That's not an easy question to answer, but most of us here who dug out our vintage RC cars from dusty old boxes and wanted to rejuvenate them went through the same stage. On one end of the scale, you have minimal cost; on the other end, you have optimal performance & low maintenance. Somewhere in between is logical, and that depends on your wallet.
I think it helps to discuss what's different between old electronics and all the new stuff, so you know the pros & cons of modernizing/keeping the old stuff.
1. Batteries: NiCad & NiMH batteries lose their ability to take a proper charge as they age. NiMH batteries--even brand-new ones--lose their charge just sitting around. I've had fully-charged NiMH packs go almost completely dead after sitting for a month or two. LiPo battery technology is far more advanced; as long as they're stored with a partial charge, they will almost never lose their capacity to take a full charge, even years later. They also don't "slow down" as they are discharged, like NiCad & NiMH batteries do; you get consistent performance throughout the range of charge. Of course, going LiPo means you also need a new battery charger.
2. Motors: Brushed motors require maintenance and cleaning, brushes & springs wear, comms get carboned up, and need capacitors to avoid glitching due to electromagnetic interference. New brushless motors are practically maintenance-free. Especially if you buy a motor & ESC combo, glitchy performance is practically a thing of the past.
3. ESC's: You can still buy modern ESC's that will handle brushed AND brushless motors, but most of them are either/or. Choosing a brushless motor & ESC combo is the best way to upgrade with the least likelihood for problems. You can power modern ESC's with old NiMH batteries, but may need to disable low voltage cut-off options as those batteries ideally need to discharge completely, unlike LiPo batteries, which should never discharge below 7.4V (3.7V per cell).
4. Radios: Old AM/FM radios are often glitchy with EMI, have much shorter range and you need to match frequency crystals--and worry about other people who might run the same frequencies. New 2.4 GHz digital radios have a huge range advantage, tiny receivers and short antennae, and all you do is bind a receiver to a transmitter and you're done. No worries about interference from other transmitters or motor/receiver glitching. Also, most modern transmitters can be used to control several different cars just by binding each car's receiver to the transmitter.
So, doing it "the optimal way" is definitely not on the "cheap" side of the scale. I think if you were going to do it in stages, I'd probably suggest going with a modern brushless motor & ESC combo first. You could still use the rest of your old gear. Good combos are all over eBay; Leopard Toro, EZRun, Turnigy, HobbyWing and dozens of clones...full combos can be had for $60-ish and up, which would include the motor, ESC & programming card. I don't think you'd have issues using your old batteries as long as you disabled the low-voltage cutoff feature of the ESC. A 30A to 60A ESC with motor in the range of 13.5T to 17.5T would be a decent upgrade that wouldn't blow up most gearboxes; you can decrease the amount of "punch"--acceleration--by programming the ESC.
Next, I'd probably buy some LiPo batteries and do away with the old NiCad & NiMH batteries. Buy a decent LiPo battery charger while you're at it. You'd have to enable the low-voltage cutoff in the ESC, of course, to protect the batteries from discharging too much and potentially causing a fire. A couple batteries and a charger could probably be had for just over $100-ish.
Last, I'd get rid of the glitchy, fickle old radio gear and make the move to 2.4 GHz. Radio & receiver packages start at about $40-ish (FlySky GT3B is a common example), and have way more features to adjust than old radios. I personally think adjustable endpoints is fantastic, to be able to decrease the amount of servo throw is quite useful.
I hope this helps, and of course there will be other opinions. I agree like you that it's a pretty expensive venture to upgrade everything all at once.