So I was playing with a male sim and his cat and I bred her (with a stray, the whole story is she got out and came back with a little more than before )

But when the kittens aged up, I couldn't breed them. I even got a new male cat to breed with the female in the litter, but nothing. No "try for kitten" menu came up.

The same thing happened when I adopted a kitten from the pet adoption service (I think it was Cymmi?) and tried to breed him when he aged up, but no "try for kitten" again. Is there something about a pet being a kitten/puppy when you get them that means they can't breed?

Has anyone else had this problem?

Check their relationships panels. Pets born in-game can certainly breed - I have several 3rd-gen Crittur dogs. However, the pet you are instructing to try for puppy/kitten must have a good relationship with both the Sim giving the instruction and the pet they are supposed to be breeding with. It sounds as though you've been trying too soon after getting the new pet, before they've had time to develop a friendship with the existing one.

Maybe there's stuck pet pregnancy controller/token? Try clearing it with batbox or Bo - no sim loaded

Too full house, maybe? Pets with large litters would fill up the household pretty quickly, unless you have mods to counteract the household limit (default is up to 6 pets, up to 8 sims, 10 sims/pets in all - CAS is still 8 sims/pets). If you're unsure, check if "try for baby" comes up for the human sims. If not, you're probably at the limit and don't have a mod to fix this.

Could be a mod compatibility problem if anything. When my game glitched it was usually because of a mod.

Good tips already, I just want to add that they also need a pet house. I hardly ever play with pets, but I remember that being necessary. Usual pet beds don't work.