The Fed should increase interest rates "in the coming months" if the economy picks up as expected and jobs continue to be generated, Yellen said on Friday, bolstering the case for a rate hike in June or July.
Gold was down 0.9 percent at $1,201.41 per ounce as of 0634 GMT, after earlier touching $1,199.60, its lowest since Feb. 17.
U.S. gold was 1 percent lower at $1,201.40.
The dollar on Monday rose above 111.00 yen for the first time since late April, with the move underpinned by comments from Yellen at the end of last week.
"After $1,200, gold will try to break $1,160," said William Wong, assistant head of dealing at Wing Fung Precious Metals in Hong Kong.
"I think the market will be quieter today as U.S. and the UK have a holiday. So, after the Asian markets, the prices won't change much," Wong added, however.
Bullion has been under pressure since the prospect of an imminent rate hike was indicated by the Fed's April meeting minutes that were released earlier this month, with key central bank officials consistently supporting an increase.
An increase in rates raises the opportunity cost of holding gold, which does not earn interest.
"Much will depend on what the dollar will do on Tuesday when all markets resume trading," INTL FCStone analyst Edward Meir said in a research note.
Spot gold may drop to $1,175 per ounce, as it has broken a support at $1,206 per ounce, Reuters technical analyst Wang Tao said. And a break below $1,175 could open the way towards $1,145, he said.
Among other precious metals, spot silver and spot platinum touched seven-week lows.
Speculators’ Net Long gold position in COMEX decreased 26% between 17-24 May while gold prices dipped 3.6% in the period.
Reference: Reuters, CFT