If you have been to the island and only ate fudge, you are missing out!! Don't be put off by the thousands of bikes on the island, because I rode 10-12 miles of the "fun trails" without coming across another biker on a busy summer weekend.
There seems to be a pretty amazing amount of trails (single track, two tracks, and paved bicycle only trails) throughout the center of the island. Some is easy, some is very difficult. The Tranquil Bluff Trail is my favorite and varies from easy on the north end to some pretty wild stuff back at the south closer to town (that trail turns into the Manitou Trail between Arch Rock and Robinson's Folly). A lot of those trail are right along a ridge (seems like it's several hundred feet high) overlooking Lake Huron and the main paved trail that surrounds the island. There are quite a few spots that you would certainly not want to fall off (or you could easy not ever be able to ride, or breath, again if you know what I mean).
Oh yah, bring your own bike to the island if you want to ride the fun trails as all the rentals are for touring around the island on the flat paved trails (and I probably wouldn't trust them to even do that). Most of the ferries charge an additional $8 (round trip) for a taking your bike to the island and back.
Watch out for road apples!!