Crystal was born at Doctors Hospital on the Upper East Side of Manhattan, and initially raised in The Bronx. As a toddler, he moved with his family to 549 East Park Avenue in Long Beach, New York, on Long Island. He and his older brothers Joel and Richard, nicknamed Rip, were the sons of Helen (née Gabler), a housewife, and Jack Crystal, who owned and operated the Commodore Music Store, founded by Helen's father, Julius Gabler. Jack was also a jazz promoter, a producer, and an executive for an affiliated jazz record label, Commodore Records, founded by Helen's brother, musician and songwriter Milt Gabler. Crystal is Jewish (his family emigrated from Austria and Russia), and he grew up attending Temple Emanu-El (Long Beach, New York) where he was Bar Mitzvahed. The three young brothers would entertain by reprising comedy routines from the likes of Bob Newhart, Rich Little and Sid Caesar records their father would bring home. Jazz artists such as Arvell Shaw, Pee Wee Russell, Eddie Condon, and Billie Holiday were often guests in the home. With the decline of Dixieland jazz and the rise of discount record stores, in 1963 Crystal's father lost his business and died later that year at the age of 54 after suffering a heart attack while bowling. His mother, Helen Crystal, died in 2001.