Removing mud / leaves in the drivetrain will help the bike run trouble free. Removing some mud on the down tube will not.
After each ride I remove the dirt on the fork stanchions, shock and dropper then I lubricate it. Same goes for the for the drivetrain.
That being said, my wife found that I wash my bike more often than our car… which is true.