Greylisting: shouldn't it be server-wide?

So I’m pretty confused right now.

Mailbox A receives mail from sender X for the first time. Gets stuck for a few minutes because of greylisting, no issues.
The next mail from X to A goes through without delay because the sender is now known.
But now, mailbox B also receives mail from sender X.
What I would expect: the mail gets delivered instantly because sender X is already recognized by the server through the mail sent to mailbox A.
What actually happens: Greylisting triggers again for mailbox B.

What do you think?

Greylist maintains a matching pair of from and to email.