In Act 3, Scene 1, what does Mercutio think is the reason Romeo refuses to fight?

Here's what Romeo says:

I do protest I never injur'd thee,
but love thee better than thou canst devise.
And so, dear Capulet, whose name I tender
As dearly as my own, be satisfied.

Mercutio is the next to speak:

O calm, dishonourable, vile submission.
Alla stoccata carries it away.

Draws.
Tybalt, you ratcatcher, will you walk?

You're right to assert as you do that Mercutio does not know that Romeo's love for the Capulet's name is anything to do with Juliet: in fact, he dies never knowing about Romeo and Juliet's relationship.

Mercutio simply thinks that Romeo's refusal to fight is because of cowardice, and because he is scared of Tybalt: and his submission to Tybalt is "calm, dishonourable, vile". As his friend won't fight Tybalt, he draws his sword to fight him himself. And the rest, as they say, is history.