I understand where you are coming from about worrying about missing out, I felt the same way. However, now that we've been rid of them, we've really been able to take advantage of lifestyle changes that we wouldn't otherwise be able to. About 3 & 1/2 months ago I was actually able to become a stay at home dad since we were able to make the financials work. This is huge since I get to spend an awesome amount of time with my 2 girls now and now I get to spend all my creative energy writing and working towards my passions!
You may feel that it will take 2-3 years, but you'd be amazed at how fast it goes when you get intensely motivated towards a goal and put all your energy at it!
Either way you go, you are building wealth and aren't blowing the money, so you're definitely in the winners circle regardless!