I need some advice on my 19 week old baby and his dad.?

My 19 week old baby cries hysterically everytime his dad holds him, The father and i don't live together but was around throughout the whole pregnancy and up to now, but recently he have not see him on a regular, so now he sees him every other day. it breaks my heart to hear him cry and have no clue what can cause him to cry like that with his father. Any suggestions would be helpful,