A different interpretation of the same story:
Brooke Astor 'congratulated' Camilla Parker Bowles
Either way, the future Duchess of Cornwall reportedly responded politely, according to the New York Post:
"She laughed," at Astor's "mistress business" remark, [witness Vartan Gregorian] remembered. "Gracious."
(Warning: When I looked at the New York Post article it contained a pop-up ad that disabled my browser, but here's a link if you want to try it yourself.)