Keep it all. But keep it
neat.
Nobody else has mentioned that you should find an e-mail program that suits your needs and system. I personally love Gmail and am infinitely thankful that my uni uses it; if you use a Gmail account, start getting familiar with stars, labels and folders. All will help manage it, and make finding useful things far quicker and easier. Outlook (the program) is similarly excellent, but few people seem to have it these days. I assume licences are expensive... Outlook.com (note the difference; web-based and free) is about as much fun as shooting yourself in the foot. And about as effective.
Find a program you like, create a system you trust and
follow it religiously. It doesn't have to be super-complicated; anything from "Production e-mails" through to MatthewShiner's folders is suitable as long as you can navigate it efficiently and can file stuff quickly and accurately. David Allen makes a great point in his book "Getting Things Done" that a system has to be simple and even kinda fun to use, and should be flexible enough to accommodate anything you could have to file; aim for this rather than a strictly-regimented system whereby you have 8,000 folders (example only) and go slightly neurotic filing e-mails into it. The biggest key to success isn't whether it's a complicated system or not, but whether you trust it to keep all of your data in check. If you can trust it to the point where you aren't wondering "Now, where did the e-mail from the PM re bump-in go...?", it's perfect.
There's no reason to delete substantive emails. Storage is incredibly cheap and becoming cheaper by the day; I'm fairly certain the amount of storage I have on my personal gmail is outpacing the rate at which I'm using it....
Same situation for me. I sometimes wonder what it would take for me to collect data faster than Google expands my mailbox capacity...