In the config file you mentioned there is a setting called Unicorn.MaxItemsToQueue
<!--
Max number of items that will be processed individually by the publishing code.
If number of changed items in a sync operation goes > MaxItemsToQueue, a system Smart Publish will be executed instead
Set this value to 0 if you always want Smart Publish to happen. Set it to something silly (like 1000000) if you never want Smart Publish to happen
-->
<setting name="Unicorn.MaxItemsToQueue" value="1000" />
It means that if you sync more items than value of the setting, it will trigger smart publish of the whole site instead of publishing only those items which were synced.
Just change the setting to 1000000 and it will stop auto-publishing the whole site and instead it will only publish what is synced.
And it you don't want auto publish to happen at all, you can disable the file, empty the file and leave only <sitecore>
node in it or remove TriggerAutoPublishSyncedItems
processor - whatever is the easiest for you.