J.C. Flügel called it "the great masculine renunciation", that is, the early nineteenth century abandonment of colour and ornament in men's dress. That's not to say that men's dress couldn't be stylish —as depicted in these fashion plates from La Mode below.
At first I thought both these gentlemen were wearing dressing gowns, but then I noticed the hat on the mantelpiece, and the buttons on the garment worn by the gentleman on the left, I realised that he was wearing a form of long overcoat ("reddingote" in the French caption, from the English "riding coat"). He's clearly a man-about-town dropping in to visit his friend in the velvet dressing gown on the divan at the right. The man on the right is also wearing some kind of cap, possibly a smoking cap used to keep the smell of tobacco from permeating the wearer's hair.
All the models have neatly trimmed and styled facial hair, an intermediate step between the clean-shaven faces of the previous generation, and the wild whiskers of the mid-Victorians!
(Images from the Bunka Gakuen University Library.)