Chasten Buttigieg/instagram

Pete Buttigieg's twins have already visited a major landmark before turning one year old.

On Wednesday, the U.S. Transportation Secretary, 40, caught up with PEOPLE at this year's South by Southwest festival (SXSW), where he chatted about his 7-month-old twins Joseph "Gus" August and Penelope Rose.

Buttigieg, who shares his kids with his husband Chasten, revealed that the couple loves to take their twins out around Washington, D.C., and have already made a stop at the White House.

"I took them to the White House. Chasten and I took them to the Christmas tree lighting at the White House," he says. "That was a fun performance, although that was a little much for them. They definitely respond to lights and to music."

The politician says he and Chasten also "love taking them on walks, especially now that it's warming up again."

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.

RELATED: Pete Buttigieg's Baby Boy Goes Home After '3 Weeks In and Out of Hospitals': He's 'Doing Great'

"They love it. They're always more relaxed when they're in the stroller, [it's] something about the vibrations," he adds.

He also notes that he's "excited about exposing them to music and performances," though that may be "a little bit in the future."

For now, Buttigieg says he and Chasten are "just getting started" on reading bedtime stories to their kids.

"It is amazing how many books are in some way transportation themed. Maybe I'm more attuned to that. Trains and planes, cars," he says. "It will be fun to get into some of that stuff."

Pete Buttigieg/Twitter

In September, Buttigieg shared the first family photo with his husband and their twins.

"Chasten and I are beyond thankful for all the kind wishes since first sharing the news that we're becoming parents. We are delighted to welcome Penelope Rose and Joseph August Buttigieg to our family," he tweeted at the time.

The politician previously announced that he and Chasten were parents on Twitter in August. "For some time, Chasten and I have wanted to grow our family. We're overjoyed to share that we've become parents!" he wrote.