Rigidity Results For Group Von Neumann Algebras With Diffuse Center