Police were called to reports of a sudden death involving a mechanical digger at the site at 5.30pm on Wednesday, according to Demolition News.
The man is said to have died instantly after apparently plunging around 100ft while operating a mini excavator.
Safety concerns meant his body was not recovered until Thursday when emergency crews returned to the scene.
The Health and Safety Executive is now investigating. A spokesman for Avon and Somerset Police said they were "keeping an open mind" about the circumstances of the incident.
The factory was shut down in March after Cadbury was taken over by US firm Kraft.
Got a story? Email news@theconstructionindex.co.uk
