I heard once, although I cann't say for sure, that when the missiles approched the 40mm opened up on the approaching missile, after all the missile was quite large, but they didn't hit it.
Wikipedia says the same thing. One thing about the Styx is that, although a large subsonic missile by today's standards, it's a very small target compared to what AA gunners would've typically expected. Also it's a very
fast target. If it was a battleship with the massive AA arsenal typical of a vessel that size, maybe, but not a small ship like this. Destroyers had a hard time shooting down kamikazies, which were slower and larger targets.