Hyperspectral image classification is a challenging problem as obtaining complete and representative training sets is costly, pixels can belong to unknown classes, and it is generally an ill-posed problem. The need to achieve high classification accuracy may surpass the need to classify the entire image. To account for this scenario, we use classification with rejection by providing the classifier with an option not to classify a pixel and consequently reject it. We present and analyze two approaches for supervised hyperspectral image classification that combine the use of contextual priors with classification with rejection: 1) by jointly computing context and rejection and 2) by sequentially computing context and rejection. In the joint approach, rejection is introduced as an extra class that models the probability of classifier failure. In the sequential approach, rejection results from the hidden field associated with a marginal maximum a posteriori classification of the image. We validate both approaches on real hyperspectral data.