Why am I getting notifications for records that aren't mine?

Check whether you are looking at All Tasks rather than My Tasks, and whether tasks are assigned to you. Task reminders follow the Assigned To field, and Virtual Assistant nudges follow the topics set to Proactive.