I think of crystallised fruit, of the Nutcracker, of the Child Catcher in the film of Chitty Chitty Bang Bang. Of a doting grandfather, with a crumpled paper bag in his jacket pocket.
My further researches tell me that a sugar plum is nothing to do with fruit but is another term for a dragée - a hard sweet made merely of sugar in a small round or oval shape. Again, the word dragée conjures up the antique: grandparents' sweets. Perhaps the "sugar plum" simply became the more child-friendly term as opposed to the more draconian-sounding expression.