I do not know exactly where the Manhattan Wharf was located in 1867 but it is likely to have been within the vicinity of the New York Passenger Ship Terminal.  The current terminal replaced a facility named the Chelsea Piers, said to have operated from 1910 to the 1930's.

The Quaker City departed Manhattan Wharf June 8, 1867 at 2:00pm and returned to New York November 19 at 10:00am.

Mark Twain Project: Quaker City Itinerary

On May 18, 1895 the Clemens family arrived in New York from Paris.  This is the beginning of Twains tour around the world as reported in his book "Following the Equator". (MTDBD)