VNƯ Jo u rn al o f Science, E a rth Sciences 28 (2012) 264-275
Assessing the feasibility o f increasing spatial resolution o f remotely sensed image using HNN super-resolution mapping
combined with a forward model
Nguyen Quang Minh*
Faculty o f Surveying and Mapping, Hanoi University o f M ining and Geology Received 03 September 2012
Revised 24 September 2012; accepted 15 October 2012
A bstract. Spatial resolution o f land covers from remotely sensed images can be increased using super-resolution m apping techniques for soft-classified land cover proportions. A further development o f super-resolution mapping techniques is downscaling the original remotely sensed image usmg super-resolution mapping techniques with a forward model. In this paper, the model for increasing spatial resolution o f remote sensing multispectral image is tested with real SPO T 5 imagery for an area in Bac Giang Province, Vietnam in order to evaluate the feasibility o f application o f this model to the real imagery. The soft-classified land cover proportions obtained using a fuzzy c-means classification are then used as input data for a Hopfield neural netw ork (HNN) to predict the m ultispecừal images at sub-pixel spatial resolution. Visually, the resulted image is compared with a SPOT 5 panchromatic image acquired at the same time with the multispecừal data. The predicted image is apparently sharper than the original coarse spatial resolution image.
Keywords: Hopfield neural network optimisation, soft classification, image downscaling, forward model.
1. Introduction m apping such as E lad a n d F e u e r [1], T ipping and B ishop [2]. A lthough w idely applied in Spatial resolution o f im age and photos can im age processing, these approaches are hardly be increased by the super-resolution algorithm s. applicable for super-resolution o f rem otely In the im age processing context, im age super- sensed m ultispectral (M S) im agery because o f resolution com m only refers to the process o f the lack o f a sequence o f im ages o f the scene at using a set o f cross-correlated coarse spatial the same or sim ilar tim es. The only feasible resolution im ages o f the sam e scene to obtain a application o f the super-resolution approaches single higher spatial resolution im age. T here are using im age sequences is for hyperspectral num erous studies on such super-resolution im agery [3]. F or o ther com m on m ultispectral rem otely sensed im agery, only few m ethods for
* Tel-84-982721243 in cre asin g th e sp a tia l re so lu tio n to sub-pixel E-mail: nguyenquangminh@humg.edu.vn
2 64
N.Q. Mirth / V N U Journal of Science, Earth Sciences 28 (2012) 264-275 265
level have been proposed such as a Point Spread Function-derived convolution filter [4], segm entation technique [5], and geostatistical m ethod [6],
Sub-pixel spatial resolution land cover m aps can be predicted using super-resolution m apping techniques. The input data for super
resolution m apping are com m only the land cover proportions estim ated by soft- classification [7], T here is a list o f super
resolution m apping techniques have been inừoduced including spatial dependence m axim isation [8], linear optim isation techniques [9], H opfield neural netw ork (H NN ) optim isation [10], tw o-point histogram optim isation [1 1], genetic algorithm s [12] and feed-forw ard neural netw orks [13], The supplem entary data are also supplied to H N N to produce m ore accurate sub-pixel land cover m aps such as m ultiple sub-pixel shifted im age [14], fused and panchrom atic (PA N ) im agery [15,16]. These latter approaches produce a synthetic M S or PA N im age as an interm ediate step for super-resolution m apping based on a forw ard m odel and then these im ages are com pared w ith the predicted and observed MS or PAN im ages to produce an accurate sub
pixel image classification.
The creation o f the predicted M S an d then PA N im age b y a forw ard m odel suggested a possibility to im plem ent a super-resolution for the M S image. A m ethod for increasing the
spatial resolution o f the original M S image IS
inừoduced by N guyen Q uang M inh et al [17].
T he new m odel is based on the H NN super
resolution m apping technique from unsupervised soft-classification com bined w ith a forw ard m odel using local end-m em ber spectra [15,16]. T he m ethod is exam ined w ith a degraded rem ote sensing im age and both visual and statistical evaluations show n a good result.
H ow ever, there still exist som e concerns about the feasibility o f the m odel because it is not tested in a m ore com plicated landscape w ith different kinds o f land cover features w hich are varying in sizes and shapes as well as specfral characteristics. This paper, therefore, is to im plem ent the test o f the algorithm in a com plicated landscape.
2. G eneral m odel
T he proposed m odel is an extension o f the super-resolution m apping approach based on H N N optim isation. T he prediction o f a MS im age at the sub-pixel spatial resolution is based on a forw ard m odel w ith local specfra as w as used in N guyen et al., 2006 [15], In addition to the goal functions and the proportion con sừ ain t o f the H N N for super- resolution m apping, a reflectance constraint is used to retain the brighừiess values o f the original M S im age.
266 N.Q. M in h / V N U Journal of Science, Earth Sciences 28 (2012) 264-275
Spatial SR MS image (2 0 m)
convolution
Figure 1. General model for super-resolution MS imagery prediction.
Synthetic MS image (4 0 m )
Figure 1 presents the H N N sub-pixel MS im age prediction algorithm . T he procedure is as follow s: From the M S im ages at the original M S spatial resolution, land cover area proportion images are predicted using a soft- classifier. A set o f local end-m em ber specừ a v a lu e s is calc u la te d b a se d o n th e e stim a te d land cover proportions and the original M S image.
L and cover proportions are then used to constrain the H N N for super-resolution m apping w ith a zoom factor z to produce the land cover map at the sub-pixel spatial resolution. From the super-resolution land cover m ap at the first iteration, an estim ated MS im age (at the sub-pixel spatial resolution) is then produced using a forw ard m odel and the estim ated local end-m em ber spectra. The estim ated M S image is then convolved spatially to create a synthetic M S im age at the coarse
spatial resolution o f the original image.
Follow ing a com parison o f the observed and synthetic M S im ages, an error value is produced to retain the brightness value o f the pixels o f the original M S image. The process is repeated until the energy function o f the HNN is m inim ised and the synthetic M S im age is generated.
A dem onsfration o f the algorithm for an image o f 2x2 pixels can be d escn bed in Figure 2. Firstly, the soft-classification predicts land cover proportion as in Figure lb from the MS specừal im age as in Figure la. There are two land covers in this im age called Class A and Class B. From the land cover proportions in Figure lb , the land cover classes at sub-pixel level are predicted as in Figure Ic w here a pixel is divided into 4 x 4 sub-pixels and the 2x2 pixels im age is super-resolved to 16x16 pixels
N.Q. M in h Ị V N U Journal of Science, Earth Sciences 28 (20Ĩ2) 264-275 267
land cover image o f Class A and Class B. The brightness o f the new 16x16 pixels image is predicted using end-m em ber spectra (standard brightness for the C lass A and B in this area o f the image). For exam ple, the brightness o f the pure pixel o f Class A is 35 and Class B is 50
and it is possible to produce a new spectra!
image by assigning all the sub-pixels belonging to Class A the brightness value o f 35 and the sub-pixels o f C lass B the brighừiess value o f 50 as in Figure Id.
100% land coverA
50% land c o v erA 50% land c o v er B
62.5% land cover A 37.5% land cover B 100% land coverB
(b)
Figure 1. Creation o f !6><16 pixels image from 2x2 pixels image.
2.Ỉ. Soft-classification f o r super-resolution m apping o f M S im agery
Soft-classification is an intennediate step in the sub-pixel M S im age predictio n process. The prediction o f the M S im age based on super
resolution m apping requires land cover proportions w hich are obtained from soft- classi fication as input data. C onventionally, there m ust be a set o f training data for m ost o f the soft-classifiers. A ccordingly, it is necessary to have som e prior infom iation about the specừal distribution o f land cover classes in the M S bands, although training data are not alw ays available for the im age. A nd som etim es, there is a requirem ent o f increasing the spatial resolution o f the im age w itho ut concerning the lan d cover classes in the im age scene. In these cases, the algorithm can be im plem ented w ith unsupervised soft-classified land cover proportions such as fuzzy c-means classifier [18].
Supervised soft-classifiers could also be used, such as B ayesian, neural netw ork or k 'N N classifiers. H ow ever, the training data for these soft-classification techniques should be
obtained from the unsupervised classifications.
In the research im plem ented by N guyen Q uang M inh et al [17], a test for algorithm w as im plem ented w ith a set o f degraded MS im age and soft-classified land cover proportions w as obtained using k-N N classification using a fraining data set ex ừ acted from unsupervised Interactive Self-O rganising D ata (ISO D A TA ) classifier. In this case, training data w ere clustered in the reference image. In this experim ent, a fuzzy c-m eans classification is used to predict land cover proportions o f a real SPO T im age to produce super-resolved specừ al image o f different spatial resolutions to evaluate the algorithm .
2.2. F orw ard m odel a nd end-m em ber spectra A fter the first iteration o f the H N N algorithm , once the sub-pixel classification is obtained, a forw ard m odel is used to p ro duce a sub-pixel M S im age from the sub-pixel land cover classes. The brighừiess value (e.g., reflectance, radiance, digital num ber) o f a sub
pixel {m,n) o f a spectral band 5 can be predicted as
268 N.Q . M in h / V N U journal o f Science, Earth Sciences 28 (2012) 264-275
+ (
1)
w here Ve is the output neuron o f the class e and Ss e is th e end-m em ber spectra o f the land cover cla ss e for a spectral band s. A s presented in N guyen et al., 2006 [15], the end-m em ber specfra vector Ss (Ss = [5^ o f the original pixel (x,y) o f the specừ al band A’ can be e s tim a te d lo c a lly u s in g th e p re d ic te d la n d c o v e r class proportions and the M S im age at the original coarse spatial resolution as
S ,= iP ^ W P )~ 'w P ^ R ,, (2) where p is a m aừix o f land cover proportions wiửi
r ^ x -\) { y - \)
p = PT / T
and w is the m atrix o f w eights w ith
w =
3. E xp erim en t condition 5.7. Data
T he experim ent in N guyen Q uang M inh et al [17] is conducted in an area having many large objects w ith linear boundaries. It m ay lead to a concern that th e algorithm proposed in this p aper is able to w o rk w ell only w ith some specific landscapes. Therefore, a second data set is used for testin g the algorithm in a m ore com plicated landscape. This im age was obtained in B ac G iang Province, Vietnam.
T he SPO T 5 im age used in this test was acquired in A ug ust 2011 w ith the spatial resolution o f 10m and four spectral bands (Figure 2). T he test im age is registered to W G S-1984 U TM m ap projection in Zone 48N and the location is at 21°17’53.65"N , 1 0 6 °H 7 .6 4 " E . T he test im age covers 1 square kilom etre area o f 102x102 pixels. To evaluate the results o f increasing the spatial resolution algorithm , a 2.5m spatial resolution panchrom atic im age acquired at the sam e time was used (Figure 3b).
N.Q. M in h / V N U journal of Science, Earth Sciences 28 (2012) 264-275 269
_(c) „
Figure 2. SPOT 5 image in Bac Giang Province, Viettiam: (a) Band 1, (b) Band 2, (c) Band 3 and (d) Band 4.
3.2. Soft-classification
The land cover proportions are estim ated from 10m spatial resolution SPO T 5 im age using fuzzy c-m eans classifiers so it is not necessary to have prior understanding about
land cover classes in the area. The soft- classified land cover proportions o f five land cover classes and six land cover classes are obtained as in Figure 3c and Figure 3d, respectively.
Figure 3. (a) original image, (b) Panchromatic image at 2.5m spatial resolution and land cover proportions from fiizzy c-means classification: (c) 5 land cover classes and (d) 6 land cover classes.
270 N.Q. M in h / V N U Journal o f Science, Earth Sciences 28 (2012) 264-275
4. R esults and discussions
4.1. R esuits
The m eth o d o f increasing the M S image using H N N an d forw ard m odel w as applied to super-resolve the 10m SPO T 5 M S im ages to p red ict M S im age at spatial resolutions o f 5m (zoom factor o f 2), 3.3m (zoom factor o f 3) and
2.5m (zoom factor o f 4). The predicted soft- classified proportions w ere used to consừain the H N N w ith w eighting factors o f kị = 100, k:
=1 0 0, k i = 100 and 100 to predict the sub
pixel land cover and then the M S image. The false colours com positions using B and 1, Band 2 and B and 4 as R ed, G reen and Blue are show n in Figure 4.
(e) ^ _ (f)
Figure 4. Super-resolution o f 10m SPOT 5 multispectral image: (a) 5m super-resolved image using 5 land cover classes, (b) 3.3m super-resolution image using 5 land cover classes, (c) 2.5m super-resolution image using 5 land
cover classes, (d) 5m super-resolution image using 6 land cover classes, (e) 3.3m super-resolution image using 6 land cover classes, and (Í) 2.5m super-resolution image o f 6 land cover classes.
N.Q. M in h / V N l ỉ Journal o f Science, Earth Sciences 28 (2012) 264-275 271
4.2. Evaluation
In the case o f degraded image as in N guyen Q uang Minh et al. [17], the dow nscaled imagery was com pared with the reference m ultispectral image at finer spatial resolution to obtain visual and qualitative evaluations. In this experim ent, the finer m ultispectral im agery is not available for the full assessm ent. Therefore, the predicted m ultispectral im age at finer resolution can be only com pared w ith 2.5m panchrom atic im age for a visual evaluation.
The visual com parison o f the super
resolved image from the real SPOT 5 data with the panchrom atic im age (Figure 3b) also shows an im provem ent in sharpness o f the results. The objects in Figure 4(a-f) are sharpened and look
clearer that o f original im age ( r ig u r e 3a).
A lthough the landscape o f im age area is com plicated w ith sm all and linear features such as houses and roads, the im provem ent o f the algorithm can be seen in the b ou ndaries o f between the objects. Figure 5 show s the im provem ent o f the algorithm for increasing the spatial resolution o f M S im age using H N N with a forw ard m odel to the original im age. The boundaries o f ponds in the centre o f the original M S image (Figure 5a) are blurred and fragm ented because o f the m ixing o f the land categories in these boundary pixels. In the predicted M S im age using HNN and a forw ard m odel (Figure 5b), these boundaries are clear and look m ore sim ilar to the real ponds in panchrom atic im age (Figure 5c).
(a) (b) (c)
Figure 5. Some land cover features in (a) original MS image (false colour composite), (b) increased resolution MS image to 2.5m spatial resolution from 5 land cover class proportions (false colour composite)
and (c) panchromatic image.
F or sm all objects such as houses and roads, there are few objects w hich is not clearly seen in the original im age can be recognised in the super-resolved image. In Figure 6a (com posite image using Band 1, Band 2 and Band 4 o f the original MS im age), the road is difficult to be recognised because it is fragm ented due to the m ixed pixels effect. U sing the H NN super resolution m apping and then the forw ard m odel
as in Figure 6b (com posite im age u sing B and 1, Band 2 and B and 4 o f the 2.5m spatial resolution increased im age), it is possible to recreate the road sim ilar to the shape o f the real road shown in the panchrom atic im age Figure 6c.
For the sm all features such as a group o f houses in Figure 6c, the perform ance o f the algorithm is not as good as that for the road.
272 N.Q. M inh / V N U Journal of Science, Earth Sciences 28 ( 2 0 W 264-275
However, the new ly proposed algorithm still show s some im provem ent in defining clear boundaries o f these features. T his m ay be because o f the soft-classifier cannot define the houses as a separate class. T his problem m ay be
resolved by increasing the num ber o f classes for fuzzy c-m eans classifier or using prior inform ation on these classes in supervised soft- classifiers.
Figure 6. Some land cover features such as roads and houses in (a) original image, (b) spatial resolution increased image (false colour composite) and (c) panchrom atic image.
4.3. D iscussions
The effect o f zoom factor to spatial resolution increasing algorithm can be seen in Figure 8. . C om paring the im age created by H N N using zoom factor o f 2 (Figure 4a), with the image created w ith zoom factor o f 3 (Figure 4b) and 4 (Figure 4c), it is possible to see that
w hen the zoom factor increases, the boundaries betw een the features are sm oother. The boundaries betw een the ponds and the surrounding features are fragm ented in Figure 8. a and F igure 8. b and look sm oother and clearer in F igure 8. c.
(a) (b) (c)
Figure 8. Effect o f zoom factor to spatial resolution increasing algorithm: (a) zoom factor of 2, (b) zoom factor o f 3 and (c) zoom factor o f 4.
In spite o f increasing the spatial resolution o f the rem otely sensed M S im ages, the proposed m ethod has a problem w ith pixels that
belong to the sam e class (referred to as pure pixels in this paper). A lthough the problem can be partly solved by increasing the num ber o f
N.Q. M in h / V N U Journal of Science, Earth Sciences 28 (2012) 264-275 273
classes so there m ore m ixed pixels or dividing a class into sub-classes so several pure pixels are defined as m ixed pixels, there w ill still exist pure pixels. The effect o f num ber o f land cover classes can be seen in Figure 9.. The ponds in Figure 9 .a apparently sm aller than those in
Figure 9.b due to som e “pure pixels” in the boundaries w ere re-classified as m ixed pixels w hen the num ber o f classes increased from 5 to 6. These m ixed pixels are then super-resolved to produce different boundaries betw een the same features in the tw o im ages.
Figure 9. (a) Result from HNN using 5 land cover classes, and (b) result from HNN using 6 land cover classes.
B ecause the H N N super-resolution m ethod w orks only on m ixed pixels, w hich are usually located across the border betw een different classes, it is suggested that the m ethod is suitable for the super-resolution o f im ages o f large objects, for exam ple the agricultural scenes. In these im ages, spatial variation is hom ogeneous w ithin the land p arcels and super-resolution based on the spatial clustering goal functions o f the H N N can the agricultural field boundaries or increasing the sharpness o f linear features such as roads or canals.
As m entioned above, the use o f unsupervised classification can reduce the eư o rs in land cover class p roportion prediction.
Furtherm ore, the use o f unsupervised classification facilitates the autom ation o f the spatial resolution increasing process because the class p roportions can b e obtained w ithout fraining data and w ithout a fraining step.
Through the choice o f the num ber o f classes, the user can control the effect o f the super
resolution algorithm on the resulting sub-pixel M S im ages. W hen the num ber o f classes is changed, the num ber o f m ixed pixels m ay be changed as a result.
A draw back o f the H N N super-resolution procedure is the subjective choice o f the param eters for the goal functions, the proportion co n sừ ain t and the m ulti-class constraint [15]. B y em pirical investigation, the values o f the param eters should retain an equal effect betw een the con sừ ain ts and the goal functions in the optim isation process. For exam ple, the em pirical investigation in this paper show s th at the values o f these param eters in this paper w ere sim ilar and around the value o f 100. T he finding is also obtained from the N guyen Q uang M inh et al [17].
27 4 N.Q. M in h / V N U journal of Science, Earth Sciences 28 (2012) 264-275
5. C onclusions
The approach for increasing the spatial resolution M S im agery utilised the H N N super
resolution m apping technique com bined w ith a forw ard m odel is tested w ith 10 SPO T 5 rem otely sensed data. In this research, the soft- classified land cover proportions were estim ated using a fuzzy c-m eans classifier. The feasibility o f the m ethod w as evaluated based on visual com parison o f the resulted im age w ith panchrom atic im age acquired at the sam e tim e w ith original image. T he com parison show ed that the proposed m ethod can generate MS im ages w ith m ore detail features. The super
resolved im age w as apparently sharper than the original coarse spatial resolution im age. In addition, the evaluation also dem onsừ ated that w hen the zoom factor increased, the resulting sub-pixel im ages w ere closer to the reference image.
R eferen ces
[1] M. Elad and A. Feuer, Super-resolution reconstruction of image sequences, ĨEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 21, pp. 817-834, 1999.
[2] M. E. Tipping and
c.
M. Bishop, Bayesian image super-resolution in Advances in Neural Information Processing Systems, Boston, 2003.[3] T. Akgun, Altunbasak Y., and R.M. Mersereau, Super-resolution reconstruction of hyperspectral images, ỈEEE Transactions on Image Processing, vol. 14, pp. 1860-1875,2005.
[4]
c.
Pinilla Ruiz and J. Ariza Popez F., Restoring SPOT images using PSF-derived deconvolution filters, International Journal o f Remote Sensing, vol. 23, pp. 2379-2391,2002/[5]
w.
Schneider and J. Steinwendner, Land cover mapping by interrelated segmentation and classification of satellite images. International Archives o f Photogrammetry and Remote Sensing, vol. 32, pp. part 7-4-3, 1999.[6] P. c . Kyriakidis and E. H. Yoo, Gcoaatiitica prediction and simulation of point values Tom the areal data, Geographical Analysis^ /o! 37, pp. 124-152, 2005.
[7] G. M. Foody, Hard and soft classificatuns by a neural network with a nonexhaustively defined set of classes, International Journal o f Remote Sensing, vol. 23, pp. 3853-3864, 2002.
[8] P. M. Atkinson, Mapping sub-pixel boindarics from remotely sensed images, in Ịnnoviỉion in GỈS 4, z. Kemp (Ed.), Ed. London: Tiyicr &
Francis, 1997, pp. 166-180.
[9] J. Verhoeye and R. De Wulf, Lane covcr mapping at sub-pixel scales using linear optimisation tcchniqucs, Remote Serving o f Environment, pp. 96-104, 2002.
[10] A. J. Tatcm, H. G. Lewis, p. M. Atkins)n, and M. S. Nixon, Supcr-resoiution land cover pattern prediction using a Hopfield neural network, in Uncertainty in Remote Sensng and GỈSy Foody G. M. and Atkinson p. V., Eds.
London; John Wiley & Sons, 2002, pp. 5Í-76.
[11] P.M. Atkinson, E. Pardo-Iguzquiza, ind M.
Chica-Oỉmo, Downscaling Cokriging forSupcr- Resolution Mapping of Continua in Remotely Sensed Images, ĨEEE Transacíioỉs on Geoscience and Remote Sensing, vol. 46 no. 2, pp. 573 - 580, 2008.
[12] K. c . Mcrtems, L., Duchcyiic, E. I. Virbekc, and R. De Wulf. Using gcnctic algoritims in sub-pixcỉ mapping, International Jouvial o f Remote Sensing, vol. 24, pp. 4241-4247,1003.
[13] K. c . Mertems, L. Verbeke, T. Westra, ind R.
De Wulf, Sub-pixel mapping and SLO-pixcl shaq^ening using neural network pridictcd wavelet coefficients, Remote Sensiĩ^ o f Environment, vol. 91, no. 2, pp. 225-236.2004.
[14] Feng Ling, Yun Du, Fci Xiao, Huapíriị Xue, and Sheng Jun Wu, Super-rcsolution lanc-covcr mapping using multiple sub-pixel ;hiftcd rem otely sensed im ages, ỉn tern a tio n a ỉ J)urnaỉ o f Remote Sensing, vol. 31, no. 19, pp. 5023 5040, 2010.
[15] Q. M. Nguyen, p. M. Atkinson, and H. c 2006 Lewis, Super-resolution mapping using Hopfleld neural netwok with fused inages, IEEE Transactions on GeoscAence and ỉemoĩe Sensing, vol. 44, no. 3, pp. 736-749, 2006 [16] Quang Minh Nguyen, Peter M. Alkinsoi, and
Hugh G. Lewis, Supcr-rcsolution mipping using Hopfield Neural Network with panchromatic imagery, International Journal o f
N.Q. M in h / V N U journal of Science, Earth Sciences 28 (2012) 264-275 275
Remote Sensing, vol. 32, no. 21, pp. 6149-6176, 2011.
7] Quang Minh NGUYEN, Van Duong DO, P.M.
Atkinson, and H.G.Lewis, Downscaling Multispectral Imagery Based on the HNN Using Forward Model, in 7'^‘ FIG Regional Conference on Spatial Data Serving People:
Land Governance and the Environment- Building the Capacity, Hanoi, 2009.
[18] J.
c.
Bczdck, J. M. Keller, R. Krishnapuram, and N. R. and Pal, Fuzzy Models and Algorithms fo r Paiiern Recognition and Image Processing. Boston: Kluwer, 1999.[19] G. M. Foody, The role of soft classification techniques in the refinement of estimates of Ground Control Point location, Photogrammetric Engineering and Remote Sensing, vol. 68, pp. 897-903, 2002.