Document pod labels and annotations

We have a range of labels and annotations that are set or can be set by the end user on the pods. These labels and annotations are used and get checked in the code but we have not defined and documented any of them.

We should keep a list of labels and annotations we know and use and what they are used for.