Experts say confusion, fear and politicians using gender diverse people as scapegoats could be behind the shifting attitudes.