A safety team decided that conditions outside the harbour walls were too rough, and have postponed the races until tomorrow, when they will re-assess the situation at 9.30am. Eleven races had been scheduled.
The raft race within the harbour is still expected to go ahead at 3.45pm today.
Other events, including the greasy pole competition, are unaffected.
The annual event had already suffered a blow when it was revealed that the Battle of Britain Memorial Flight's planned flypast over the town on Sunday had been cancelled after several of the World War Two-era aircraft were grounded over safety fears.