Plans to create a riverside garden in this area date from 1893 though the possibilities must have been discussed for some years prior to this. Two years later, in 1895, land had been purchased, and by 1902 the gardens were sufficiently complete to permit the planting of a London Plane tree in August 1902 to celebrate the Coronation of Edward VII and his Queen, Alexandra.

A bandstand was erected and the band of the Royal Horse Guards played there on Wednesdays from 3pm to 5pm.