Crosby was born in Los Angeles, California, and grew up in Humboldt and Sonoma counties in Northern California, raised by a single mother from an upper-class family who owned a record store in Fortuna, California. His great-grandfather was John C. Crosby, a congressman from Massachusetts and his ancestry on his mother's side is English and Native American. It is speculated that his father, who was a lounge guitarist whom he never met, may have been partially Japanese.