Plummer was born on December 13, 1929 in Toronto, Ontario. He is the only child of Isabella Mary (née Abbott), an artist who was secretary to the Dean of Sciences at McGill University, and John Orme Plummer, who sold stocks and securities. His great-grandfather on his mother's side was Canadian Prime Minister Sir John Abbott. His father's uncle was patent lawyer and agent F. B. Fetherstonhaugh. Plummer is a second cousin of British actor Nigel Bruce, known as Doctor Watson to Basil Rathbone's Sherlock Holmes. His parents divorced shortly after his birth, and he was brought up in the Abbott family home in Senneville, Quebec outside of Montreal. He speaks both English and French fluently.