Even if you run out of coins, you will never see a game over screen in Super Mario Odyssey. The news comes from Nintendo Japan, who confirm as much in a tweet: 

So, while you will lose ten coins and become more poor (which is important because items help you buy stuff), the game will never say Game Over in regards to your playthrough. The mechanic sounds like a welcome change considering there were already plenty of ways to game the system in past titles to ensure you never got a game over anyway. Are you happy with this decision? Are you excited to play Super Mario Odyssey later this year? 

Source: Eurogamer