I would only be concerned about financing the home till you freed up the equity in your existing home. I've seen too many "that won't happen" scenarios happen and financially hurt/crush people. In your situation, only play real estate without using debt and never enter retirement with a mortgage). Only you and your wife know your limitations with physical labor and the stress of a lifestyle change. I wouldn't view it lightly, but by no means crawl up and let time pass you.
I'm a young man and would love to own your dream. In fact, could walk to that park and enter the Mammouth Sniper Challenge each winter.