In most cases, yes—you will lose progress. Points and eligibility are tied to the specific X account you used when you registered, and points generally cannot be migrated between X accounts.
If you disconnected an X account from a wallet and then connected that same X account to a new wallet, your points from that X account will reflect in the new wallet.