Priorities

When my b/f and I first started dating I was the commander of my American Legion post.  I was super busy and had very little time to devote to a relationship.  Slowly things changed and the more serious we got the more he became a priority.  Now the shoe seems to be on the other foot.  He's busy with everything it seems.  Work, kids, family, etc., and whenever I bring up the fact that we seem to spend very little "alone" time together it starts a fight with him saying that I just don't understand what it feels like to be pulled in every direction and that he doesn't need this right now.  So lately we've been talking about moving in together when he gets orders back down to So. Cal.  I have many concerns about this, but one of them is can I handle not being a priority in his life.  I understand that kids come first.  My son comes first in my life.  But everything comes before me, or at least it seems like that.  Six months ago I asked that we have "date" night one night a month.  Just us, no kids.  Started out great....it's happened twice in six months.  So I am at a loss right now...