Answer by Professional Japanese Teacher
Sumimasen. Nihongo ga wakarimasen.
Sorry, I don’t understand Japanese.
The direct translation of the above expression is “I don’t understand Japanese” but this is the most appropriate expression in this case. “Wakarimasu” is a verb meaning “to understand” and “ga” is a particle to indicate the object. Try answering with the above expression if you are spoken to in Japanese.
You can also ask others if they can speak English as below.
Q: 英語（えいご）がわかりますか。Eego ga wakarimasuka?
Do you understand English?
A: いいえ、わかりません。 Iie, wakarimasen. No, I don’t understand.
Since you’re in Japan, do take the opportunity to try communicating in Japanese.
You can refer to the following Essential Greetings for a start.