Are there any specifics I should watch out for in the marriage quest? It's got to a point where I should ask the betrothed for hand in marriage, but they keep rejecting my Sim. They feel like my Sim is extremely irresistible, have their relationship meter maxed out, and I've tried to have them married both at the beach and on my Sim's home lot. The rejection doesn't even affect the extremely irresistible level (unless I spam it in which case they start finding me odd which is to be expected).

Could this be a glitch, as I don't think there's anything else I should be doing?... They're already courting and the ask for hand in marriage interaction shows up normally, it's the rejection part that keeps happening - which is a different issue from all the similar threads I've come across when searching before posting this.

Not sure if I fully understand, are they rejecting the marriage or the marriage proposal?

There is definitely a glitch when trying to start marriage ceremony. They keep waving hands as if there is no room for the marriage to take place.
Usually, it is very annoying, and I have to try multiple times but it eventually happens.

I don't think I ever encountered actual rejection.

I believe the interaction is called "Ask for hand in marriage." The interaction is rejected by the proposee - my hero gets down on one knee, and the other person giggles and waves their hand no (they however don't wave that there's no room to route, though).

Could it be because both Sims are male? But I'm pretty sure both of them are courting each other, i.e. don't have another marriage going on with someone else or anything like that.

If you are seeing the "Ask for hand in marriage" prompt then they should be courting but no engaged yet. If you check the relationship panel it should say Boyfriend.

Both sims being male should not be an issue.

Is the other person in a good mood? Maybe they are hungry or sick? Even if they are flitry/irresistible, if they have negative buffs they will refuse big romantic things.
Make sure you do this in the morning so they are rested or maybe in a well decorated room that gives environment buffs?
Serve food so they are not hungry. If they have any visual effects above their head they might be sick, so need to be treated.

If this won't help, then I have no idea. Glitch or a broken mod maybe?

Good thinking! I did get a neutral "exposed to sickness" buff. It could be that the proposee is actually really sick, while my hero hasn't got the sickness to develop yet.