The influential Tory backbencher, who chairs the hardcore European Research Group, cautioned people could soon get “fed up” with Parliament’s upper house.
He added the public will think it has “very little legitimacy” if peers keeping making amendments to the Government’s flagship Brexit laws.
The comments come a day after Theresa May suffered three defeats on the EU (Withdrawal) Bill in the Lords.
Many peers who oppose Theresa May’s plans say they are simply trying to secure the best outcome possible after Brexit and performing their democratic duty to scrutinise and improve legislation.
Speaking at an Open Europe event on Tuesday, Mr Rees-Mogg said: “I think we are in a position of peers against the people, I think it’s deeply unattractive.
“It’s the weakest position for the House of Lords to be in.
“There is a problem with the House of Lords, which is it is very condescending towards the democratic vote. It seems to think that they know better than 17.4 million people.”
Mr Rees-Mogg added the Lords had a “noble history as a revising chamber” but that all its attempts to change the Brexit Bill were to stop Britain leaving the EU.
The only amendment he said was a genuine attempt to improve the legislation was restricting so-called Henry VIII powers to allow ministers to change laws without requiring a vote.
But the North East Somerset MP warned: “When it challenges the democratic will, as it is doing now, then we get fed up with it and think it has very little legitimacy and needs to be challenged.
“Their lordships are playing with fire and it would be a shame to burn down a historic House…”
“They may have to decide whether they love ermine or the EU more.”
Responding, Labour’s leader in the Lords, Baroness Smith of Basildon, said: “I think Jacob is getting a little bit over-excited.
“All that has happened is that the House of Lords has voted to give the House of Commons an opportunity to further look at these issues.
“Why is he is so worried about that?”
Mr Rees-Mogg, who has been tipped as a potential future Tory leader, also on Tuesday rubbished a vote called on the UK staying in a customs union due to take place on Thursday.
The result would be “about as useful as the wallpaper in this room,” he said.
And Mr Rees-Mogg compared selling out fishing communities by trading water access for a free-trade deal to Windrush families.
He said they were both examples of “socialism” where the Government would “put the collective before the individual”.