The ruins of the medieval castle walls in Wallingford have been damaged by recent bad weather and the town council has issued a safety warning.