ALBANY, N.Y. — Gov. Eliot Spitzer says the chairman of the state panel that rules on judges' conduct should resign after co-authoring a humor book that said the accused are likely guilty.

Manhattan attorney Raoul Felder, the unpaid chairman of the state Commission on Judicial Conduct, wrote a book with comic Jackie Mason titled Schmucks! Our Favorite Fakes, Frauds, Lowlifes, Liars, the Armed and Dangerous, and Good Guys Gone Bad.

Felder's colleagues on the commission have already called for him to step down, citing several passages in the book they considered offensive, including: "anytime you hear the word 'allegedly,' you can bet it's true," and "nothing in our country is more insidious than affirmative action."

Spitzer weighed in yesterday on Felder, who said his free speech is protected by the First Amendment.

"He does not have a First Amendment right to be chair of the commission on judicial conduct," Spitzer said yesterday. "By virtue of having that position, he accepts certain limits upon his own behavior, just as judges do. Just as other elected or governmental individuals do, whether appointed or elected."

Asked if he would seek to remove Felder if he didn't resign as chairman, Spitzer said: "Let us see how this plays out."

In an April 13 interview, shortly after the commission announced its vote of no confidence, Felder said he would defend his First Amendment rights in court.

"I would feel better as an American if they could point to something, some official act I have done, where I haven't acted appropriately or somebody feels their rights were infringed by anything I've done," Felder said. "And they can't do that."

The panel said the book was "crude, biased, vulgar and otherwise demeaning" and "repeatedly invokes racial, ethnic and religious invective."

Panel members said the affirmative-action statement "raises a reasonable perception that the speaker could not pass fair judgment on anyone he perceived to have benefited from affirmative action."

The commission rules on complaints against judges including conflicts of interest and inappropriate behavior. The commission is seeking to remove Felder as chairman, but not as a commissioner. Felder's term ends next year. He was appointed by former Gov. George Pataki.