In the first half of the poem, where is the port?

I think what you are asking is "where is the poet" in this poem. In order to answer that, we need to clarify the idea that the poet is not necessarily the actual narrator of the poem. We have no idea where the poet "is" in the poem. However, Whitman gives us a number of clear details to allow us to determine where the narrator of the poem is located.
In line four, the narrator specifically states that they are "sitting" in a "lecture room" of some sort. There is also clear indication that the narrator is not alone in the lecture room, as there is also the "learn'd astronomer," one of the central focuses of the poem, but also "much applause" based on what the astronomer is saying. The first three lines also make it clear that the narrator is listening to some sort of specific lecture, as "proofs," "figures," and "charts and diagrams" are all "ranged out" and "shown" by the astronomer. Finally, Whitman's narrator begins to feel "tired and sick" and leaves the lecture hall to enter into "the mystical moist night-air" late in the poem, which gives the reader the sense that the lecture hall was very stuffy. While it is difficult to tell where the poet, Whitman himself, was when he wrote the first half of the poem, his narrator is clearly situated in a crowded, stuffy lecture room, being subjected to the demystification of the world. The narrator eventually leaves the lecture to return to the natural world and enjoy it for what it is.