Roman, I think there's a problem with mashing the two ideas of rehabilitation and punishment in prisons together -- American prisons don't rehabilitate because they aren't really designed to because enough people still think they ought to be designed as and for punishment. As far as I'm concerned, a jail term means "you belong to the state for X period of time," and the state needs to take that time to figure out what it needs to do to you to not commit the crime again.
Just the idea of prisons is relatively new; the word murder comes from the amount of money a murderer was supposed to pay to the victim's family. There was still a graded system of punishment where different offences cost different amounts and I can see how once we started using prisons there would be a range of sentences up to the maximum of 'the rest of your life' even if it doesn't make much sense from a prison as paying-a-debt-to-society standpoint.
The state can try to make you atone for your crimes by sticking you with other criminals and let you find ways to pass the time, or the state can try to give some sort of positive action to actually convince you not to break the law. It can be education, therapy or hypnosis as far as I'm concerned, but a society which can actually implement this positive action is going to be much better off than one which can't.
Paragraph 1. I absolutely agree. The problem is that a jury in sentencing is supposed to do that in death penalty cases. The irony is that the 'penitentary' system in the US was designed to do just that - rehabilitate. By the 1930's it was evident that rehabilitation was a farce. Anyone who ends up 'rehabilitated' usually didn't need to end up in prison in the first place for any number of reasons.
The biggest flaw in criminal sentencing is that there is too much elapsed time between someone being charged and someone being punished if and when they are convicted. This means that there is a diminished mental association in the criminal's mind between the crime committed and the punishment. This leads to the belief amongst most criminals that they are being punished not because they committed a crime but because they got caught. In their minds, getting caught is the crime, not the criminal act that landed them in trouble in the first place.
Another biggie flaw in the US justice system is that while it is intended that one is innocent until proved guilty the fact is that it is just the reverse; one is in all practical terms guilty until proven otherwise. It's actually not a justice problem but a law enforcement problem, but that's another issue.
The level of recidivism is clear evidence that there is no rehabilitation and than prisons are not effective at either punishment or rehabilitation. In the mean time, every attempt at both has been tried to no avail.
One of the problems we have is that there are too many laws to be violated, most of which are totally inane laws. I do not condone the consumption of illegal drugs, but I have issue with punishing users instead of going after the distributors. The theory currently is that if you deny the distributers customers the market dies. Applying market theory to crime is absurd. If you catch a drug dealer and shoot him, the supply rapidly dries up regardless of the demand. Imperial China largely solved the opium problem in a flash by simply executing convicted opium dealers. Brutal, but effective.
Paragraph 2. Yes, in the scheme of things, prisons, or rather the idea of confinement as a punishment, is a recent idea. The Romans used 'carcers' or holding places for felons until they were brutally punished for whatever crime they committed. The Romans punished. We coddle.
Crime is an ever existing problem and will never be eliminated ('Marginal 10% theory' and such). The murder rate (and the capital punishment rate for that matter) has gone up since the elimination of public executions. I think public hangings go a long way in deterring potential murderers and rapists. Hanging a murderer or rapist assures everyone that the executed person never commits that crime (or any crime) again.
Paragraph 3. The idea you seem to being driving at is to study criminals so as to gain knowledge of how to deter crime. I agree, but criminals should be studied in the same way we study cancer; with the intent to eliminating criminals and crime. There are some people whose crimes are so heinous that the only appropriate sentence is death. Hitler, Mussolini, Saddam Hussein, Ossama Bin Laden and a whole parade of other loonies. Could you imagine trying to rehabilitate a Hitler?