It is unclear when the original house was built but records suggest that it was in the last half of the C18th for the Pointer family. It could possibly have replaced an earlier building. It passed by marriage to the Standleys then sold to Lord Overstone, and later passed on to Lord Lord Esmé Gordon (born 1853) who extended it some time in the 1880s. The main house was refurbished / re-clad, while a large service wing was built in a mixed Victorian manner with elements of the Italianate and Jacobean styles. This more than doubled the size of the house and created a new north wing, behind which there were further service buildings and a stable block. Lord Esmé Gordon also owned 71 Princes gate Kensington in London and appears to have outstretched himself, having been declared bankrupt on 17 August 1891. He died in 1900 just 47 years old.
Its use changed from home to school to hospital in the C20th. One of the later owners, a Lady Wantage is recorded as having died in 1920, after which Paxton Park was sold to Mr.Harold Boardman, who used it as a boys' school. It was a Christian Science boarding school in the 20s (a photo of 1927 suggests all girls). It then became the Cambridge district's maternity hospital during WW2 until 1955. Following usage as a hospital the hall was finally demolished in 1959. The site of the C18th house today is from 38 Park Avenue to the road nearby to the NE.