Inside the gymnasium of Madison High School a patriotic tableau is being given; Mayor Shinn follows with an address. Harold Hill demands the attention of the audience and starts once again to expound his ideas about a boys' band until the kids become infected with his enthusiasm ("Seventy-Six Trombones"). Later the same evening, Harold Hill goes to the library to win Marian over to his cause. He tries to impress her by telling her he is a "professor" of music, a graduate of the Gary, Indiana University, in the gold medal class of 1905. But Marian tells him in no uncertain terms that he cannot mesmerize or hoodwink her the way he had done the rest of the townspeople. Thus brushed off unceremoniously, Harold consoles himself with the idea that, after all, he has no possible interest in a girl as prim and conventional as Marian. ("The Sadder but Wiser Girl"). Nevertheless, he makes another effort to win over Marian, by sneaking up to her desk at the library and trying to convince her that he is infatuated with her ("Marian the Librarian").

But all the while, Harold keeps alive the enthusiasm for his band project, and goes about town signing applicants. Cuttingly, Marian asks him why he does not use his gift to greater advantage at a carnival. She knows he is a fraud, and thus will have no traffic with him, even though she is not the kind of girl who waits for a knight in shining armor ("My White Knight"). The reason Marian knows he is a fraud is because she has consulted the Indiana State Educational Journal and has discovered that Gary, Indiana University, had not even been founded in 1905. She is about to bring this information to the leading citizens of the town when Wells Fargo arrives with the musical instruments and uniforms, creating so much enthusiasm and excitement among her neighbors that she simply does not have the heart to disillusion them about Harold Hill. Indeed, now that Harold has, indeed, delivered the instruments and the uniforms without absconding with the money she is much more sympathetic to him; and there is no question in her mind that he is a man of considerable charm. Her resistance to him is finally broken: She comes to tell Harold she is in love with him ("Till There Was You"). Now Harold Hill gives himself up completely to making a success of the boys' band. In a stirring finale the boys appear in full regalia and sound their first raucous notes as members of the town band.