Even if a contact is not in the same team as you it can be routed if it has been assigned to you (i.e. contact ownership).

There is an option to control this on the admin side under Field Customization ->  Teams.  Check the box "Route owned contacts only when Caller and Contact are in the same team" if you want Team to override Contact Owner.