A bit of philosophy and explanation

Extend the olive branch with one hand and hold the pen in the other, while carrying the sword by the side for when it needs to be drawn and as a reminder of one's duty to try all they can to prevent its use.

Extending the olive branch is the offer of peace.
The pen is the means to secure that peace.

Carrying the sword represents war and drawing it is the declaration of it and it being a reminder for it to not get drawn is due to how costly war is.