She was speaking at a garden party at Buckingham Palace on Tuesday, the same day that Prime Minister David Cameron was filmed making undiplomatic remarks to her about corruption in Nigeria and Afghanistan.
The queen's remarks may not be helpful to the British government's determined efforts to boost trade ties with China.
Under her constitutional role, the 90-year-old monarch never makes any politically or diplomatically sensitive comments in public, and it is rare for the content of her private conversations to be revealed.
In Beijing, Chinese Foreign Ministry spokesman Lu Kang said Xi's visit to Britain last year had been "extremely successful" and both countries' officials had made great efforts to that end.
In footage broadcast by the BBC, the queen is seen meeting senior police officer Lucy D'Orsi, who is introduced by an official as having been in charge of security during Xi's visit in October.
"Oh, bad luck," the queen says in response.
D'Orsi then describes her dealings with Chinese officials as "quite a testing time" and recounts that at one point they had walked out of a meeting and told her "the trip was off".
The queen says: "They were very rude to the ambassador."
Lu said he had not heard anything about Xi's visit possibly being called off at any point, adding that the visit had opened a new "golden age" in relations, brought about by both countries.
CENSORED
The BBC reported that in China, items about the queen's remarks were censored from its news bulletins.
The Chinese authorities often censor items they object to from foreign news bulletins, which can only be seen by very few people in China as foreign TV channels are only allowed in high-end hotels and a tiny number of select apartment buildings.
A spokeswoman for the queen said: "We do not comment on the queen's private conversations. However, the Chinese state visit was extremely successful and all parties worked closely to ensure it proceeded smoothly."
Xi's visit was full of pomp and ceremony, with Cameron and his finance minister George Osborne keen to impress the Chinese leader and present Britain as China's firmest friend in Europe.
The queen has been careful to keep her views to herself during her 64-year reign, but several other members of Britain's royal family have made undiplomatic comments about China in the past.
Her eldest son, heir-to-the-throne Prince Charles, has skipped two state banquets for Chinese guests in Britain, and described some Chinese officials in a journal that was leaked to the media as "appalling old waxworks".
Relations between London and Beijing have been complicated by the weight of history, particularly the 1860 Opium War when British and French troops stole piles of plunder from the Summer Palace in the Chinese capital, then burned it to the ground.
In 2010, Cameron and a delegation of ministers caused offence during a visit to Beijing by wearing poppies - a symbol of remembrance of fallen troops for Britons, but in China a reminder of the opium trade that helped trigger the conflict.