I allow the physical armor, with some of the aforementioned penalites for traveling, sleeping, entering town with armor, but I put in this 
rule:

The protection is not cumulative.  I.e., you can only use the highest value, either the spiritual or physical, for protection in combat.