yes, to do so you need to go into the survey template that you are using, make a copy and then modify it. You can delete the code for this button or you can put it into comments which will hide it.
How this is done, depends very much on the template you are using. In Bootswatch you'll probably find it in clearall_links.twig, if I am not mistaken.
Help us to help you!
Provide your LS version and where it is installed (own server, uni/employer, SaaS hosting, etc.).
Always provide a LSS file (not LSQ or LSG).
Note: I answer at this forum in my spare time, I'm not a LimeSurvey GmbH employee.