Milik opened the scoring from the spot in the 11th minute, sending Frosinone keeper Michele Cerofolini the wrong way after a penalty was awarded for a foul on midfielder Fabio Miretti.
The Poland striker doubled the lead from close range in the 38th, having controlled a brilliant cross from Weston McKennie, before adding a third after Manuel Locatelli profited on Frosinone's defence error to set the Pole up following halftime.
Milik thought he scored his fourth goal of the night just before the hour mark but his effort was ruled out for offside.
However, Kenan Yildiz made it 4-0 minutes later, sending another expert McKennie's pass in with his first touch.
"I thank the boys, who played a good match and entertained the crowd," Juventus coach Massimiliano Allegri told Mediaset.
"The important thing is to understand that you have to stay in the game, play hard and defend well. Today it's a good thing we didn't concede goals, we had conceded too many lately.
"We must rejoice at the victory, but from Saturday we must think about Sassuolo, who beat us in the first leg (in Serie A)."
Juventus, who are the team that has reached the last four the most times in the competition's history - 36 in its 77 editions - will next face Lazio in a two-legged semi-final in April.
Maurizio Sarri's side advanced on Wednesday following an ill-tempered 1-0 victory over AS Roma.
Fiorentina will play Atalanta in the other semi-final.
The final will be played on May 15th at the Stadio Olimpico.