The idiom is agricultural — you hoe a row of crops, not a road. American rural origin, 19th century. 'Road to hoe' makes no sense: you don't hoe pavement.

Row, not road. Hoes are for rows of crops. A tough row to hoe = hard work on your patch.