Reported speech is when we tell someone what another person said. To do this, we can use direct speech or indirect speech. From the statement above, X made the speech while Y is reporting the speech. In reported speech, we do not have to quote the exact statement but report it in its past form and the auxiliary verb must always be changed. The correct reported speech by Y would be;

Y: They asked him whether he wanted to be paid in cash or by cheque.

The auxiliary verb 'be' was dropped and replaced with 'whether or if' as the case may be.