Lots of parents I've spoken to found it hard going from 1-2 children, due to worrying about how they will split their time and love for more than one baby.
Also some children can have a tough time transitioning to having a new baby in the home. This can be challenging for parents to deal with.
For me personally, going from 3 to 4 was definitely the toughest!