I very much understand the anger of having to keep that much reserve in there and having them charge (annually) a 6% fee (25 a month over 12 months is 300).
With that said, lets say you accelerate your pay down and you don't put in 30K for 3 years in retirement accounts.
Assume 7% interest for those retirement savings and you don't tap them until 15 years from now.
30K at 7% for 15 years = 82,770.95
35K at 7% for 12 years (monies are redirected to house for 3 years and the 5K frees up for investing at year 3 ) = 78,826.71
So this scenario you will pay a $4000 penalty which will be a loss of just under $3000 in purchasing power at the future point because of inflation.
Depending on how mad you are that might be a worthwhile purchase but its a lot for your future self to pay for the experience. I am not saying don't do it - just noting how much it would cost. Or you could reframe it mentally and note that you are saving $300 annually in fees by super saving powers and keeping the 5000 there. A 6% return. Nice!
Going out 30 years the difference is 228,367.65 if you keep with your retirement plan or 217,485.37 if you take a 3 year break on the account. The longer it goes the more expensive the decision become. Admittedly a good dilemma to have either way so kudos for your financial discipline for even having this choice!