There's Always Tomorrow (1956)

Clifford Groves, toy manufacturer, is in full charge at the factory but feels left out and taken for granted by his wife and children at home. Alone and depressed, he meets old flame Norma, and one thing leads to another. While their relationship is still fairly innocent, his son Vinnie sees them together and suspects the worst. It's time for tortured souls behind rain-streaming windows...