Clean up nullptr handling for functors

Right now optional is used to wrap pointers as well, while that should not be necessary.

See 1b5772d6 (comment 9904096)