Bank stocks languished after Bank of Japan Governor Haruhiko Kuroda said it was theoretically possible for the central bank to cut interest rates to around minus 0.5 percent, driving the banking sector to the biggest drop on the board.
The Nikkei share average fell 0.8 percent to 16,974.45.
Hong Kong shares finished down on Wednesday as investors sold shares on the final day of the annual meeting of China's parliament.
Analysts said market participants might be taking profits protectively ahead of the closure of the National People's Congress, although mainland shares finished up.
The Hang Seng index fell 0.2 percent, to 20,257.70, while the China Enterprises Index lost 0.4 percent, to 8,571.36 points.
China stocks closed up on Wednesday as the country's annual parliamentary session came to a close.
The CSI300 index of the largest listed companies in Shanghai and Shenzhen rose 0.5 percent, to3,090.03, while the Shanghai Composite Index gained 0.2 percent, to 2,870.43 points.
Reference: Reuters