Well you don't have to go into great detail all the time. The journey could actually just be a paragraph long.
At this point of the story, he's suppose to drift off to the tavern tired from his rough day.
He's alone and drifting off? Surely, an ideal time for him to do some thinking and possibly add some background to your character. Why has he had a rough day? Is it a regular thing? What are his plans now?
Or use an event to make him think of something else. He's walking to the tavern and he sees something and this makes him think of something else, something that again builds on the character or the story.
It is quite difficult to think of at first, but just see how your thoughts flow normally as you walk around, and then you can apply that to your