Casting from object[] in C#

Can anyone explane this?

ArrayList list = new ArrayList();
list.Add(new Banana());
object[] areTheseBananas = list.ToArray();
Banana[] bananas = (Banana[]) areTheseBananas; //  causes runtime casting exception!

Why doesn't the language allow this?

I know you can do...

ArrayList list = new ArrayList();
list.Add(new Banana());
Banana[] bananas = (Banana[]) list.ToArray(typeof(Banana));

...but in the code I am working with I don't know the type of the array at the point that ToArray is being called.