Because the writing system is ridiculous
The beauty of the characters is indisputable, but these ideograms weren't too practical for daily use. It is absolutely true that Chinese is hard because of the huge number of characters one has to learn. And Chinese is not very phonetic, which means that often you just completely forget how to write a character, if there is no obvious semantic clue in the radical, and no helpful phonetic component somewhere in the character.
Because the language doesn't have the common sense to use an alphabet
The English is so easy because of the skills needed to master the writing system are 26 letters and they are written from left to right ,horizontally, across the page, with spaces to indicate word boundaries. In comparison, Chinese has nothing that corresponds to an alphabet, though there are recurring components that make up the characters. It can be said that the components of Chinese characters are arrayed in two dimensions, rather than in the neat one-dimensional rows of alphabetic writing.
Because even looking up a word in the dictionary is complicated
One of the most unreasonably difficult things about learning Chinese is that merely learning how to look up a word in the dictionary is about the equivalent of an entire semester of secretarial school. Chinese must also be one of the most dictionary-intensive languages on earth for there are various dictionaries.
Because there's classical Chinese (wenyanwen)
Whereas modern Mandarin is merely perversely hard, classical Chinese is deliberately impossible. Classical Chinese really consists of several centuries of esoteric anecdotes and in-jokes written in a kind of terse, miserly code for dissemination among a small, elite group of intellectually-inbred bookworms who already knew the whole literature backwards and forwards.
Because there are too many romanization methods and they all suck
Perhaps that's too harsh, but it is true that there are too many of them, and most of them were designed either by committee or by linguists, or -- even worse -- by a committee of linguists. It is, of course, a very tricky task to devise a romanization method; some are better than others, but all involve plenty of counterintuitive spellings.
Because tonal languages are weird
It's one of the most common complaints about learning Chinese, and it's also one of the aspects of the language that westerners are notoriously bad at. As non-native speakers, you must memorize along with the vowels and consonants. The real difficulty comes in when you start to really use Chinese to express yourself. Intonation and stress habits are incredibly ingrained and second-nature.
Because there is culture difference
One of the main reasons Chinese is so difficult for westerners is that the culture between the East and the West has been isolated for so long. China has had extensive contact with the West in the last few decades, but there is still a vast sea of knowledge and ideas that is not shared by both cultures. When westerners and Chinese get together, there is often not just a language barrier, but an immense cultural barrier as well.