Sometimes knocks to the doors. They know that you steal supplies. They want 1 carrot and 1 potato for keep this in silence. If you didn't give them what they want few times they will force attackers to come to you more frequently.