We make time to go to work, do the laundry, run errands, and pay bills because we have to. These things are necessary responsibilities of life that can't be avoided. But what do you do when the necessary things of life become overwhelming?
Thats when you need to stop and re-evaluate your priorities. If you don't, those closest and most important to you begin to feel alienated. They feel less and less valuable to you as they watch you invest the majority of your time and your best energies into things other than your relationship with them.
I believe that a lot of Christian marriages have broken up because of this very thing. Especially concerning pastoral couples. No matter how much you love the Lord, or how pure your motives, your spouse and children are not going to be okay with you leaving them on the back burner so that you can make yourself available to 'minister' to strangers. And I don't believe the Lord ever intended for a person's spouse and children to be the lowest on the spiritual totem pole as far as their priorities go. In fact, if it is true that God meant for the family to be a picture of His love for the church, then what does a setup like that imply about God?
If you want to serve the Lord and minister to those around you, that is an honorable and lovely thing. Just make sure that your ministry is just as warm and loving inside your home as it is outside your doors.