The short answer is that passports should be considered during the initial separation of you and your spouse. This may involve a conversation with your spouse about which parent will hold onto your children’s passports, and how passports will be provided to the other parent for any travel.
If your children are still infants, and do not have passports, you may need to consider adding provisions into your separation agreement or court order about who will obtain and hold on to your children’s passports. Issues often arise in this area, and settling the passport issue is especially important if you and your spouse do not get along well.