Weird HVAC problem

So to make this short, I started having a problem where the fan wouldn't kick on at all unless I banged on the fan housing. The compressor would kick on no problem but not the fan. So I replaced the blower motor itself since it wouldn't work on any speed. Then when I turned it on it only worked on high. So I replaced the resistor and all was well for about a week now. Then today it didn't turn on the fan again... So I played a bit and once I changed the orientation (ie changing from face to defrost) and magically the fan turned back on. So I'm at a loss. What could cause my fan not to work unless I change the orientation? This is beyond my intermediate expertise with this van. Any insight is appreciated. Thank you in advance!

Check electrical connections to the defrost switch. Just follow the wires and check for problems.
Usually a bad ground can cause problems.. Fuses are okay though?

make sure the blower switch is good too... the terminals tend to melt due to poor design