When search is empty, pressing back once should take me out of search

Right now, I sometimes enter search up swiping up accidentally, but then I have to press back twice to first close the keyboard and then go out of search. This has been very annoying.