Yes, speed is one factor. Also, if you can't get on plane, you'll be plowing all the time and it's going to get miserable and wet quick in even slightly rough water. That means you'll have to stay away from any big water bodies unless conditions are perfect. I had a highsider for years and I wouldn't even consider less than a 8hp.Why would there a travel limitation? Speed?
I have a 13' HS, at the moment not caring about speed just want to be able to go out and fish later on id be interested in throwing a bigger motor.