Richards was given his marching orders after he appeared to aim a punch at Aaron Downes as the pair clashed four minutes before the break.
And it was centre-back Downes who landed the decisive blow nine minutes after the interval when he powered home a header from Darren Currie's corner.
There was no shortage of entertainment in the very open early stages and Jamie Ward could have given Chesterfield the lead inside 30 seconds.
Ward scampered clear from Danny Hall's clearance but scooped a weak effort straight at Vale reserve keeper Chris Martin, who stood in for the injured Joe Anyon.
Martin was then more seriously extended to tip around shots from Gregor Robertson and Jack Lester before Vale hit back in the 20th minute.
Gareth Owen's punt up the field sent Richards clear and the Vale skipper drove a left-foot shot which crashed against the crossbar and rebounded clear.
The action continued to sway from end to end until Richards gave the visitors a helping hand through his moment of madness.
Chesterfield piled on the pressure at the start of the second-half with Martin making a tremendous save from Lester after the striker exchanged passes with Jamie Lowry.
But Martin had no chance when Downes landed the decisive blow.
Robertson should have put the visitors in the comfort zone three minutes later when he blazed over from eight yards with the goal at his mercy.
And Chesterfield had to survive a spirited recovery from Vale, who almost snatched a last minute equaliser when substitute Kyle Perry headed straight at Tommy Lee.