SpectralFormer: Rethinking Hyperspectral Image Classification with Transformers
release_iw67o2iwhjafbhhrwogcswyk7u
by
Danfeng Hong and Zhu Han and Jing Yao and Lianru Gao and Bing Zhang and Antonio Plaza and Jocelyn Chanussot
2021
Abstract
Hyperspectral (HS) images are characterized by approximately contiguous
spectral information, enabling the fine identification of materials by
capturing subtle spectral discrepancies. Owing to their excellent locally
contextual modeling ability, convolutional neural networks (CNNs) have been
proven to be a powerful feature extractor in HS image classification. However,
CNNs fail to mine and represent the sequence attributes of spectral signatures
well due to the limitations of their inherent network backbone. To solve this
issue, we rethink HS image classification from a sequential perspective with
transformers, and propose a novel backbone network called SpectralFormer.
Beyond band-wise representations in classic transformers, SpectralFormer is
capable of learning spectrally local sequence information from neighboring
bands of HS images, yielding group-wise spectral embeddings. More
significantly, to reduce the possibility of losing valuable information in the
layer-wise propagation process, we devise a cross-layer skip connection to
convey memory-like components from shallow to deep layers by adaptively
learning to fuse "soft" residuals across layers. It is worth noting that the
proposed SpectralFormer is a highly flexible backbone network, which can be
applicable to both pixel- and patch-wise inputs. We evaluate the classification
performance of the proposed SpectralFormer on three HS datasets by conducting
extensive experiments, showing the superiority over classic transformers and
achieving a significant improvement in comparison with state-of-the-art
backbone networks. The codes of this work will be available at
https://github.com/danfenghong/IEEE_TGRS_SpectralFormer for the sake of
reproducibility.
In text/plain
format
Archived Content
There are no accessible files associated with this release. You could check other releases for this work for an accessible version.
Know of a fulltext copy of on the public web? Submit a URL and we will archive it
2107.02988v2
access all versions, variants, and formats of this works (eg, pre-prints)