2002 Pontiac Grand Prix Climate Control Probelm?

I have a 2002 Grand Prix GT 3.8...My problem is with the fan/ blower of the car. Fan speeds 1-4 do not work at all and fan speed 5 is intermittent at best. Sometimes the whole systems works correctly however that has become a rare occurrence. I have checked the fuses that is fine. I changed the fan relays and the problem still persists and seems to be getting worse. What else should I be looking at in order to solve this problem? Climate control unit? Any help would be appreciated.