Born in Fresno, California, Lee’s family moved to the Los Angeles area early in his childhood. When Los Angeles hosted the Summer Olympics in 1932, a 12-year-old Lee became inspired by all of the banners, souvenirs, and other paraphernalia around the city in honor of the Games. Later that summer, he discovered that he was able to do somersaults better than his friends and so he made it his goal to become an Olympic diving champion.

This was easier said than done. Like his friend, fellow diver and history maker, Vicki Manalo Draves, his local pool only allowed non-white community members in once a week. These “international days” were Wednesdays, the day before the pool’s weekly draining and cleaning, according to the United States Olympic & Paralympic Museum. A local coach, Jim Ryan, took Lee on as a student, and to make up for Lee’s limited pool time, Ryan dug a sandpit in his backyard and taught Lee how to dive into the pit.

When Lee enrolled at Occidental College in Los Angeles, he became a student athlete and Ryan continued to coach him. While at Occidental, Lee won the 3-meter springboard and 10-meter platform diving events at the 1942 Amateur Athletic Union of the United States (AAU) National Championships. He graduated the next year and enrolled in medical school at the University of Southern California (USC). While at USC, Lee was in an army student training program and served in the U.S. Army Reserve, which paid for his tuition. He graduated in 1947.

Lee kept diving throughout his education and qualified for his first Olympic Games in 1948 in London (the 1940 and 1944 Olympics had been canceled because of World War II). There, he won bronze in the 3-meter springboard event and gold in the 10-meter platform event. The latter cemented him in history as the first Asian American man to win Olympic gold—but not the first Asian American to win gold, as Draves had done that only two days prior.

Over the next four years following London, Lee didn’t compete much. Instead, he focused on his military service, reaching the rank of major in the U.S. Army Medical Corps. Although he qualified for the 1952 Summer Olympics in Helsinki, Finland, Lee considered skipping the Games because of a “sense of duty to the army,” according to the United States Olympic & Paralympic Museum.

His military superiors felt differently however, and that, in this case, Lee could serve his country in a different capacity. According to his obituary in the Los Angeles Times, he was told “Major Lee, we’ve only got one doctor who can win an Olympic gold medal. We’ve got hundreds of doctors who can repair the wounded. You can go, but you better win.”

And that’s exactly what Lee did. The army granted him one month’s leave to train and compete and he came home with a second gold in the 10-meter platform event, becoming the first diver ever to win back-to-back Olympics in the same event. After the Games, Lee returned to the military and went on to serve in South Korea during the Korean War from 1953-’55.

When he came back home, Lee, with his celebrity status and ability to charm fans, was tapped by the U.S. Department of State to be a goodwill ambassador to Asia and Europe and represent the country as a “democratic land of opportunity and equality in countries where communist propaganda was spreading,” according to the National Museum of American Diplomacy.

Lee spent most of his post-Olympic life focused on his medical career, practicing as an ear, nose and throat specialist for 35 years. But he didn’t completely leave the competitive diving world. Lee coached the 1960 U.S. Olympic team, as well as the 1964 Japanese and Korean Olympic teams. Lee also coached American divers Bob Webster and Greg Louganis, both of whom went on to win multiple Olympic gold medals.

Lee died in 2016 at the age of 96 in Newport Beach, California, from complications of pneumonia.