REM sleep behavior disorder (RBD) is a parasomnia with dream enactment and presence of REM sleep without atonia (RSWA). RBD diagnosed manually via polysomnography (PSG) scoring, which is time intensive. Isolated RBD (iRBD) is also associated with a high probability of conversion to Parkinson's disease. Diagnosis of iRBD is largely based on clinical evaluation and subjective PSG ratings of REM sleep without atonia. Here we show the first application of a novel spectral vision transformer (SViT) to PSG signals for detection of RBD and compare the results to the more conventional convolutional neural network architecture. The vision-based deep learning models were applied to scalograms (30 or 300 second windows) of the PSG data (EEG, EMG and EOG) and the predictions interpreted. A total of 153 RBD (96 iRBD and 57 RBD with PD) and 190 controls were included in the study and 5-fold bagged ensemble was used. Model outputs were analyzed per-patient (averaged), with regards to sleep stage, and the SViT was interpreted using integrated gradients. Models had a similar per-epoch test F1 score. However, the vision transformer had the best per-patient performance, with an F1 score 0.87. Training the SViT on channel subsets, it achieved an F1 score of 0.93 on a combination of EEG and EOG. EMG is thought to have the highest diagnostic yield, but interpretation of our model showed that high relevance was placed on EEG and EOG, indicating these channels could be included for diagnosing RBD.