Neuroendocrine (NE) differentiation in metastatic castration-resistant prostate cancer (mCRPC) is an increasingly common clinical feature arising from cellular plasticity. We recently characterized two mCRPC phenotypes with NE features: androgen receptor (AR)-positive NE-positive amphicrine prostate cancer (AMPC) and AR-negative small cell or neuroendocrine prostate cancer (SCNPC). Here, we interrogated the regulation of RE1-silencing transcription factor (REST), a transcriptional repressor of neuronal genes, and elucidated molecular programs driving AMPC and SCNPC biology. Analysis of prostate cancer cell lines, mCRPC specimens, and LuCaP patient-derived xenograft models detected alternative splicing of REST to REST4 and attenuated REST repressor activity in AMPC and SCNPC. The REST locus was also hypermethylated and REST expression was reduced in SCNPC. While serine/arginine repetitive matrix protein 4 (SRRM4) was previously implicated in alternative splicing of REST in mCRPC, we detected SRRM3 expression in REST4-positive, SRRM4-negative AMPC, and SCNPC. In CRPC cell lines, SRRM3 induced alternative splicing of REST to REST4 and exacerbated the expression of REST-repressed genes. Furthermore, SRRM3 and SRRM4 expression defined molecular subsets of AMPC and SCNPC across species and tumor types. Two AMPC phenotypes and three SCNPC phenotypes were characterized, denoted either by REST attenuation and ASCL1 activity or by progressive activation of neuronal transcription factor programs, respectively. These results nominate SRRM3 as the principal REST splicing factor expressed in early NE differentiation and provide a framework to molecularly classify diverse NE phenotypes in mCRPC. SIGNIFICANCE: This study identifies SRRM3 as a key inducer of cellular plasticity in prostate cancer with neuroendocrine features and delineates distinct neuroendocrine phenotypes to inform therapeutic development and precision medicine applications.