"But I have never seen anyone who would put a reference down who would speak badly of said person."
I have been on both sides of this.
Have called a reference who, with in all legalities, basically said the person was a huge screw up, and didn't think they could handle the job, and was upset they were used as a reference.
I have also been put down as a reference for a person I would consider a DNH (Do not Hire).