In the Return of the King, how come Denethor wouldn't believe Pippin and not try to burn Faramir? And why did he also want to kill himself?

Denethor had a palantir, a seeing stone, similar to the one Saruman had in Orthanc. But Denethor's one (unbeknownst to him) was controlled by the Dark Lord, who showed Denethor lots of images of how Mordor's troops were winning the war. Denethor thought there was no hope for West. The clincher was when he saw (in the stone) the ships of the Corsairs (old enemies of Gondor) coming up the river. He didn't know that Aragorn had already defeated the Corsairs and was in control of the ships.

Why did he still try to kill Farmamir, and why did he to want to die?

Because he felt he was going to come under the sway of Sauron (which he actually was without realising it), and didn't want his remaining son to fall either. He was a very proud man, and couldn't face defeat, and couldn't really accept any possibility of anyone else doing what he had failed to do.