You can follow/unfollow anything you choose by clicking the Edit button when you are using the ‘Following’ selection on the filter panel.
There is an automated follow function in your settings page that you can turn off – this is probably responsible for the ‘unexpected follows’:
