I haven't reached FI or ER... but I can provide the opposite perspective. My husband and I are currently living in a high cost of living area, paying for our own benefits (including time off and health insurance), and generally making it difficult (though not impossible!) for ourselves to save for FI/RE... all to work jobs we love. So, we've made that choice, already.
I will say one thing about loving your job: it ruins you for working jobs you don't love! I had the dream job, and then I got laid off from it 3 years ago. A lot of people did. That company was so much fun to work for, and I loved what I did there - I would have stayed till they kicked me out. (Oh wait, I did!) Then, those of us who worked there? We had so much trouble adjusting to other work. We were spoiled. Transitions were really, really hard. Some people from that company just can't work for anyone else now. (Thankfully, some of them banded together and started their own business together, creating a work environment that they love - that's the cure for that ailment!)
Sometimes I think... what if I could go back in time and never work for the Dream Job company? What if I could go back and get a really, really lucrative career, instead, and never spoil myself in the first place? I could be retiring so much earlier! Would I?
Nah. Working where I work now, loving what I do, is a form of retirement to me (from those jobs I didn't love in the interim!).