These aren't private roads, they're public roads. They're government owned and anyone can drive on them, even if you don't have business in the port. 

I understand that you go through a gate with security, but they're not there really checking to see if you have business on the port or not. I can have a McDonald's bag and say "I'm dropping off lunch to my wife/gf/whomever" and they'll let me on without verifying anything.