Deep Learning-Based Landmark Detection for Mobile Robot Outdoor Localization

Transcription

1 maches Article Deep Learng-Based Landmark Detection for Mobile Robot Outdoor Localization Sivapong Nilwong 1, Delowar Hossa 2, Sh-ichiro Kaneko 3 and Genci Capi 4, * 1 Graduate School Science and Engeerg, Hosei University, Kajochō, Koganei, Tokyo , Japan; sivapong.nilwong.46@stu.hosei.ac.jp 2 Fairy Devices Inc., Tokyo , Japan; hossa@fairydevices.jp 3 Department Electrical and Control Systems Engeerg, National Institute Technology, Toyama College, 13, Hongo-machi, Toyama , Japan; skaneko@nc-toyama.ac.jp 4 Department Mechanical Engeerg, Hosei University, Kajochō, Koganei, Tokyo , Japan * Correspondence: capi@hosei.ac.jp; Tel.: Received: 28 February 2019; Accepted: 16 April 2019; Published: 18 April 2019 Abstract: Outdoor mobile robot applications generally implement Global Positiong Systems (GPS) for localization tasks. However, GPS accuracy outdoor localization has less accuracy different environmental conditions. This paper presents two outdoor localization methods based on deep learng and landmark detection. The first localization method is based on Faster Regional-Convolutional Neural Network (Faster R-CNN) landmark detection captured image. Then, a feedforward neural network (FFNN) is traed to determe robot location coordates and compass orientation from detected landmarks. The second localization employs a sgle convolutional neural network (CNN) to determe location and compass orientation from whole image. The dataset consists images, geolocation data and labeled boundg boxes to tra and test two proposed localization methods. Results are illustrated with absolute errors from comparisons between localization results and reference geolocation data dataset. The experimental results poted both presented localization methods to be promisg alternatives to GPS for outdoor localization. Keywords: outdoor localization; deep learng; landmark detection; Faster R-CNN; CNN 1. Introduction In present world, mobile robots operate various fields applications, such as logistics, medical, agriculture, health carg and housekeepg. Navigation is one key elements that mobile robots need order to accomplish ir given tasks. Success navigation requires success different factors, cludg localization, which robots must be able to determe ir positions environments [1]. Recent fdgs suggested a significant number localization methods for both outdoor and door environments. For outdoor environments, Global Positiong Systems (GPS) is method that has been widely applied among a variety outdoor applications, some which are: mobile robot for high-voltage transmission le spection [2], autonomous position control multiple aerial vehicles [3], mobile robot for gas level mappg [4], navigation system for mobile robots usg GPS and ertial navigation system (INS) [5], and map buildg with simultaneous localization and mappg (SLAM) for firefighter robots [6]. Despite large-scale implementation, localization through GPS suffers decle accuracy from several environmental conditions. Accordg to ficial U.S. government formation about GPS and related topics [7], common causes degradation GPS accuracy are: (1) satellite signal blockage due to large objects environments such as buildg, bridges and trees; (2) door or underground use; and (3) signal reflected f buildg or walls. Due to decle GPS accuracy and Maches 2019, 7, 25; doi: /maches

2 Maches 2019, 7, reliability, a large number GPS-based approaches also employ or sensors to improve localization accuracy. For example, fusion captured camera image features with GPS signals [8], use state chi-square test and simplified fuzzy predictive adaptive resonance ory (predictive ART or ARTMAP) neural network to diagnose sensors GPS/INS system [9], and combation GPS, wheel odometry and received signal strength (RSS) from wireless communication nodes to create a precise localization approach for mobile robots [10]. Apart from improvements available, re are also various alternative localization approaches that aim to replace GPS. Such approaches rely on odometry [11,12], visual odometry [13], visual patterns [14] and ultra-wideband network [15]. Research on deep learng has extended rapidly recent years. The implementation deep learng has been spreadg though many fields applications. Localization and positiong applications also adopt deep learng approaches for tasks. For stance, deep learng-based encoder determes locations from low-level features images [16]. Some significant fields that deep learng has been extensively implemented clude object detection and object recognition. Convolutional Neural Network (CNN) is one particular stance that has been implemented for object detection and recognition, due to its structure that can effectively handle visual data. CNN has been implemented as base for various object detectors, cludg Faster Regional Convolutional Neural Network (Faster R-CNN) [17]. Faster R-CNN is state---art object detector based on region proposals, which surrounds detected objects with boundg boxes. The approach region proposal-based method for Faster R-CNN object detector is same as its predecessor, Fast R-CNN [18]. One major difference between Faster R-CNN and Fast R-CNN is Region Proposal Network (RPN), which reduces Faster R-CNN detection time and creases accuracy. The creased speed Faster R-CNN for object detection makes it suitable for real time applications [17,18]. Implementations Faster R-CNN spread throughout various applications, such as detection cyclists depth images [19], pedestrian detection from security cameras [20], and ship detection remote sensg images that conta foggy scenes [21]. In [19 21] it is shown that Faster R-CNN has high accuracy (more than 80%), slightly higher than human volunteers that have approximately 75% accuracy [22]. It is well known that humans and or animals can use landmarks to determe where y are world and generate path to destations [23]. For localization mobile robots outdoor environments, signs and landmarks are commonly visible and usually distct. CNN and CNN-oriented Faster R-CNN are very useful for handlg 2D data such as images. Therefore, it is an advantage to use deep learng-based object detection approaches for mobile robot localization outdoor environments. Visual based mobile robot localization will mimic way humans and animals determe ir locations and directions. In addition, or conventional sensors such as GPS or compass will be replaced by vision. Therefore, this paper aims to propose and compare two localization methods based on CNN and Faster R-CNN. In CNN-based method, CNN analyzes robot captured image and determes current location and orientation robot. In Faster R-CNN-based method, Faster R-CNN is used to detect landmarks with image, before sendg detected landmarks to feedforward neural network (FFNN) that generates current location and compass orientation. Data amount becomes a challenge deep learng, sce performances deep learng approaches rely on a large amount data [16,21]. Thus, we also aim to develop and test performances proposed methods usg a smaller amount data than or deep learng implementations. The proposed method has been implemented as follows: first, image dataset with geolocation data that contas 1625 sets data is created. Second, we develop localization methods based on CNN and Faster R-CNN usg created dataset. Fally, we evaluate performance developed localization methods through test set dataset created first contribution. The paper is organized as follows: Section 2 describes two proposed localization methods, and ir essential components. Section 3 explas experimental results, and Section 4 concludes this paper.

3 Maches 2019, 7, 7, Localization Methods 2. Localization Methods Two localization methods based on CNN are proposed this paper. We vestigated object Two localization methods based on CNN are proposed this paper. We vestigated object detection capabilities CNN and one its successors, Faster R-CNN, to be used for localization detection capabilities CNN and one its successors, Faster R-CNN, to be used for localization based based on visual landmarks. The first localization method is two-step procedure based on Faster on visual landmarks. The first localization method is two-step procedure based on Faster R-CNN R-CNN object detector. Faster R-CNN is used to detect visible landmarks from a camera image. object detector. Faster R-CNN is used to detect visible landmarks from a camera image. Labeled Labeled boundg boxes detected landmarks are n used as puts for FFNN that generates boundg boxes detected landmarks are n used as puts for FFNN that generates location location coordates and compass orientation from landmarks. The second localization method is coordates and compass orientation from landmarks. The second localization method is based on based on conventional CNN. In second method, whole camera image is processed through conventional CNN. In second method, whole camera image is processed through CNN to CNN to directly generate location coordates and compass orientation. Furr details two directly generate location coordates and compass orientation. Furr details two localization localization methods and ir components are described followg subsections. methods and ir components are described followg subsections Faster R-CNN Localization 2.1. Faster R-CNN Localization The overview structure Faster R-CNN based localization method is illustrated The overview structure Faster R-CNN based localization method is illustrated Figure 1. Figure 1. The Faster R-CNN based method is a two-step procedure. Faster R-CNN object detector The Faster R-CNN based method is a two-step procedure. Faster R-CNN object detector and FFNN are and FFNN are two ma components first outdoor localization method. Durg robot two ma components first outdoor localization method. Durg robot navigation, Faster navigation, Faster R-CNN detects landmarks camera captured image. This landmark detection R-CNN detects landmarks camera captured image. This landmark detection process generates process generates three types answers for each stance detected landmarks, which are three types answers for each stance detected landmarks, which are boundg box, label and boundg box, label and score. Boundg box contas position and size detected landmark score. Boundg box contas position and size detected landmark put image. Label put image. Label dicates class name detected landmark. Score refers to an dicates class name detected landmark. Score refers to an objectness score, which measure objectness score, which measure membership boundg box to classes landmarks or membership boundg box to classes landmarks or background [17]. The components background [17]. The components each detected landmark are n sent to FFNN for localization. each detected landmark are n sent to FFNN for localization. The localization part uses detected The localization part uses detected landmarks from Faster R-CNN to localize robot landmarks from Faster R-CNN to localize robot real-world environment. FFNN for real-world environment. FFNN for localization utilizes boundg boxes and labels detected localization utilizes boundg boxes and labels detected landmarks to generate geolocation data as landmarks to generate geolocation data as result localization system. The generated result localization system. The generated geolocation data cludes location coordates and geolocation data cludes location coordates and compass orientation, which location compass orientation, which location coordates are form latitude and longitude angles, coordates are form latitude and longitude angles, and orientation is form and orientation is form magnetic-referenced compass orientation. magnetic-referenced compass orientation. Figure 1. System flows Faster Regional-Convolutional Neural Network (Faster R-CNN) based Figure 1. System flows Faster Regional-Convolutional Neural Network (Faster R-CNN) based localization method. localization method. Furr details on two ma components Faster R-CNN-based localization method are described Furr details followg on two subsections. ma components Faster R-CNN-based localization method are described followg subsections.

4 Maches 2019, 7, Maches 2019, 7, Faster R-CNN for Landmark Detection Landmark detection is is first process our our Faster Faster R-CNN localization method. method. We We employ employ standard standard version version Faster Faster R-CNN R-CNN object object detector detector for landmark for landmark detection detection tasks. tasks. Typically, Typically, Faster Faster R-CNN R-CNN comprises comprises two modules, two modules, i.e., Fast i.e., R-CNN Fast R-CNN object detector object detector and region and region proposal proposal network network (RPN). (RPN). The structure The structure Fast R-CNN Fast R-CNN object detector object detector contas contas several several convolutional convolutional and max and poolg max poolg layers, layers, a regiona region terest terest (RoI) poolg (RoI) poolg layer, and layer, a sequence and a sequence fully connected fully connected layers. Inlayers. Fast R-CNN, In Fast R-CNN, a set convolutional a set convolutional layers and layers maxand poolg max layers poolg constructs layers constructs a convolutional a convolutional feature map feature frommap an from entirean put entire image. put RoI image. poolg RoI poolg extracts extracts a fixed-length a fixed-length feature vector feature from vector from feature feature map map each at region, each which region, iswhich used as is put used as put Fast R-CNN. Fast At R-CNN. fal pots At fal pots Fast R-CNN, Fast fully R-CNN, connected fully connected layers estimate layers classes estimate feature classes vectors feature from RoI vectors poolg from and RoI refe poolg result and boundg refe result boxes from boundg se boxes featurefrom vectors se [18]. feature The RPN vectors is a[18]. deep The fully RPN convolutional is a deep fully neural convolutional network that neural shares network full-image that shares convolutional full-image features convolutional with detection features network, with Fast detection R-CNN. network, RPN proposes Fast R-CNN. high-quality RPN regions proposes to high-quality Fast R-CNN regions module. to InFast addition, R-CNN it helps module. guidg In addition, Fast it R-CNN helps guidg over locations Fast R-CNN objects over locations captured image objects [17]. captured image [17]. Architecture implemented standard Faster Faster R-CNN R-CNN is shown is shown Figure Figure 2. The 2. whole The whole put image put image is processed is processed through through set convolutional set convolutional and max and poolg max layers poolg layers CNN to generate CNN to a generate convolutional a convolutional feature map. feature The map. feature The map feature is n map put is to n RPN put to generate to RPN to a set generate rectangular a set region rectangular proposals. region Sce proposals. feature Sce map feature is shared map isacross sharedrpn acrossand RPN anddetection detection network, network, generated generated feature feature map map is also is also put put to to RoI RoI poolg layer. This is isused to extract fixed-length feature vectors, with help region proposals generated from RPN. Extracted feature vectors from RoI poolg layer are n processed through a series fully connected layers to estimate classes each feature vector through classifiers and refe result region proposals form feature vectors through regression process. Thus, refed and classified region proposals, or boundg boxes are generated from Faster R-CNN. Figure 2. Faster R-CNN for landmark detection and its components. Figure 2. Faster R-CNN for landmark detection and its components. The structure CNN used our Faster R-CNN is also shown Figure 2. As mentioned earlier this subsection, The structure set CNN convolutional used our Faster layers R-CNN CNN is also analyzes shown whole Figure put 2. As image mentioned to construct earlier a convolutional this subsection, feature set map. convolutional As size layers smallest landmarks CNN analyzes utilized whole dataset put image is nearly to construct pixels, a convolutional put size feature is set map. to 32 As 32 size 3, where smallest last 3 is landmarks for three color channels: utilized red, dataset green is nearly and blue. 32 The 32 pixels, set convolutional put size layers is set contas to two, 3, two-dimensional where last 3 is convolutional for three color layers, channels: with red, a rectified green lear and blue. unit The (ReLU) set attached convolutional after each layers convolutional contas two, layer. two-dimensional The set also cludes convolutional one max layers, with a rectified lear unit (ReLU) attached after each convolutional layer. The set also cludes one max poolg layer for down-samplg purposes. Each convolutional layer employs a 3 3 filter

5 Maches 2019, 7, Maches 2019, 7, 5 14 and poolg has layer stride forsettgs down-samplg 1 pixel purposes. for both horizontal Each convolutional and vertical layer strides. employs The number a 3 3 filter filters and has first stride convolutional settgs 1layer pixelis for 48, both while horizontal 96 filters and are vertical used for strides. second The number convolutional filters layer. The first max convolutional poolg layer layer is placed is 48, while at 96 end filters arelayers used set, for which second convolutional poolg size layer. is 2 2 The and max poolg stride settgs layer is is placed 1 pixel at for both endhorizontal layers and set, vertical which strides. poolg This small sizepoolg is 2 2size andis applied strideto settgs prevent is 1 premature pixel for both down-samplg horizontal and vertical put strides. image, This which small may poolg cause size isloss applied features to prevent premature result feature down-samplg map. put image, which may cause loss features result feature map. Trag Faster R-CNN consists followg four steps: Step Step1 RPN trag; Step Step2 Fast R-CNN trag usg region proposals from Step Step1; 1; Step Step3 RPN re-trag usg weight sharg with Fast Fast R-CNN for for fe-tung RPN; and andstep 4 Fast R-CNN re-trag usg updated RPN. These steps are are same as as origal Faster R-CNN trag [17]. The The trag uses uses whole images as asput, and and labeled boundg boxes as as target. Trag contues for for epochs, with itial 4 itial learng rate. rate Feedforward Neural Neural Network Network for for Localization The The localization part part generates generates robot robot location locationusg usgboundg boxes boxes and andlabels labels detected detected landmarks from from Faster Faster R-CNN. R-CNN. Sce Sce boundg boxes boxes and andlabels labels are are generated through through features features an animage imagedurg durg detection process process Faster FasterR-CNN, boundg boxes boxes detected detected landmarks are arearranged accordg to tolabels labels landmark classes without furr processg. The The localization part part Faster Faster R-CNN R-CNN method method has a has sgle a sgle FFNN FFNN as its as core itscomponent, core component, which which uses arranged uses arranged boundg boundg boxes as boxes put. asthe put. output The units output are location units arecoordates location coordates and magnetic andcompass magnetic orientation. compass orientation. The Theimplemented FFNN consists put put neurons, neurons, 48 hidden 48 hidden neurons neurons and three and output three output neurons, neurons, as shown as shown Figure 3. The Figure 72 puts 3. The are 72 slots puts boundg are slots box boundg elements box detected elements landmarks. detected Each landmarks. boundg Each box contas boundg four box elements contas forfour positions elements andfor sizes positions box. and We sizes have limited box. We number have limited landmark number classes to ne landmark due classes numberto ne landmark due classes number experiments. landmark Furr classes details experiments. ne landmark Furr classes details will bene givenlandmark Sectionclasses 3. Each will class be given has limitation Section 3. Each maximum class has two limitation detection stances. maximum For example, two detection if re are stances. three Crossg For example, landmarks if re detected, are two three stances Crossg with landmarks highest detected, objectness two scores stances will with be used for highest localization. objectness This scores results will be used total for localization. 72 (9 2 4) This put results neurons. Default total value 72 (9 2 put 4) put neurons neurons. is zerodefault if revalue is no detected put neurons stanceis for zero each if re slot is landmarks. no detected The stance threefor output each neurons slot landmarks. correspondthe to three latitude, output longitude neurons and correspond compass. to The latitude, number longitude hidden and neurons, compass. however, The number are acquired hidden from neurons, trial-and-error however, tests. are acquired Activation from function trial-and-error FFNN tests. neurons Activation is symmetric function saturatg FFNN neurons learis transfer symmetric function. saturatg lear transfer function. Figure 3. Feedforward neural network (FFNN) for localization with detected landmarks from Figure 3. Feedforward neural network (FFNN) for localization with detected landmarks from Faster Faster R-CNN. R-CNN. Trag FFNN for localization utilizes labeled boundg boxes and geolocation data for trag, with Trag Bayesian regularization FFNN for localization as trag utilizes algorithm. labeled Boundg boundg boxes boxes are and arranged geolocation accordg data to for trag, labels attached with Bayesian to boxes. regularization Elements arranged as trag boundg algorithm. boxes are Boundg used as boxes put are arranged trag accordg data. Geolocation to labels data attached which to cludes boxes. Elements latitude angles, arranged longitude boundg angles, boxes and are magnetic used as compass put orientations, trag data. are used Geolocation as target data trag which cludes data. latitude angles, longitude angles, and magnetic compass orientations, are used as target trag data.

6 Maches 2019, 7, CNN Localization 2.2. CNN Localization Convolutional neural network (CNN) is one type deep neural networks that is suitable for two-dimensional Convolutional array neural implementations, network (CNN) such is one as type images. deep Typically, neuralcnns networks are that applied is suitable for object for detection two-dimensional and recognition array implementations, purposes, which such asfal images. parts Typically, CNNs mostly areemploy appliedlayers for object for classification. detection and recognition For stance, purposes, Stmax classifier. which fal However, parts CNNs our mostly CNN employ localization layersmethod, for classification. CNN is used For stance, to directly Stmax determe classifier. proper However, geolocation our CNN data localization from image method, and CNN its features is usedwith. to directly The implemented determe CNN proper for geolocation localization data has from its fal image parts and replaced its features with regressors with. The stead implemented classifiers. CNN Consequently, for localizationour has CNN its fal determes parts replaced geolocation with regressors data stead through regression classifiers. output, Consequently, stead our classification. CNN determes Similar geolocation to Faster data R-CNN through localization, regression output, camera stead image is classification. used put Similar for to CNN Faster localization, R-CNN localization, where whole camera put image image is is used processed as putthrough for CNN all layers localization, CNN. wherethe result whole put our image CNN is localization processed method through is all layers geolocation CNN. data, The which result consists our CNN latitude localization angle, longitude method is angle, geolocation and data, magnetic which consists compass latitude orientation, angle, longitude same angle, as and magnetic Faster compass R-CNN localization orientation, results. same as Faster R-CNN localization results. Design implemented CNN is shown Figure 4. The best architecture is determed by trial-and-error method and and combation from from different different CNN CNN examples examples and and prciples prciples available available [24]. [24]. The CNN The CNN for localization for localization comprises comprises 37 layers 37 layers total. total. The The put put layer layer CNN CNN has has size size (320 3 (320 pixels pixels width, width, pixels pixels height height and three and three color color channels: channels: red, green red, green and blue). and blue). There There are 10are sets10 sets convolutional convolutional layers, layers, batch batch normalization normalization and ReLU and ReLU cluded cluded with with 37 layers 37 layers CNN, CNN, where where convolutional convolutional layer, layer, batch batch normalization normalization and ReLU and ReLU are displayed are displayed toger toger as oneas green one green layer layer diagram. diagram. There are There different are sizes different and different sizes and amounts different amounts filters among filters implemented among implemented convolutional convolutional layers. The earliest layers. convolutional The earliest layer convolutional employs layer largest employs filter size largest 5 filter 5, while size 5 filter 5, size while is decreasg filter size as is decreasg network contues as network deeper, to contues last convolutional deeper, to layers last convolutional which apply layers filter which size apply 2 2. On filter size contrary, 2 2. On number contrary, filters begs number with afilters small number begs with 24a filters small number first convolutional 24 filters layer. first convolutional The number layer. filters The number each convolutional filters each layerconvolutional set creases as layer set network creases progresses as network deeper, progresses to amount deeper, 64to filters amount last layers. 64 filters We employ last layers. four max We poolg employ four layers max at poolg size layers 3 3, at 3 3, size 2 2, 3 and 3, 23 3, 2, 2 with 2, and stride 2 2, with settgs as stride 2, 2, settgs 1, and 1as respectively. 2, 2, 1, and 1 Fal respectively. parts Fal CNNparts for localization CNN consist for localization a fully connected consist layer a fully andconnected regressor. layer Sce and regressor. localization Sce is determed localization basedis on determed latitude, longitude, based latitude, and compass longitude, orientation, and compass only three orientation, neurons only are employed three neurons are fully employed connected layer fully connected CNN. layer CNN. Figure 4. Structure Convolutional Neural Network (CNN) based localization. Figure 4. Structure Convolutional Neural Network (CNN) based localization. The implemented CNN trag employs a whole captured image as put, and geolocation datathe as implemented trag target. CNN Setup trag for CNN employs trag a whole cludes captured 48 trag image as epochs, put, batch and size geolocation 64 and data 1 10as 6 itial trag learng target. rate. Setup The learng for CNN rate trag decreases cludes at 48 rate trag 0.1epochs, every 20 batch epochs. size 64 and itial learng rate. The learng rate decreases at rate 0.1 every 20 epochs.

7 Maches 2019, 7, Experimental Results Experiments our localization methods start from dataset construction. The geotagged image dataset is constructed to provide data for both trag and testg our localization methods. In total we created a dataset 1625 data, which is relatively small compared to most deep learng implementations. From 1625 sets data dataset, 1198 sets were randomly selected for trag process, while remag 427 sets were used to test performance. Reducg amount trag data is a very important research issue deep learng community. Our developed localization methods were tested terms localization accuracy. The Faster R-CNN localization method was also tested terms landmark detection accuracy Geotagged Image Dataset Similar to or deep learng systems, Faster R-CNN, FFNN and CNN employed our localization methods require data for both trag and test. The dataset was constructed from 1625 images form JPEG color images at size pixels. Each image was tagged with correspondg geolocation data, cludg location coordates, form latitude and longitude angles, and compass orientation. Geotagged images were labeled with boundg boxes landmarks each image. In summary, one set data consists an image, latitude angle, longitude angle, magnetic compass orientation, boundg boxes landmarks image and labels landmarks for boundg boxes. The trag set, which cludes randomly-selected 1198 sets data, was employed for trag all components both Faster R-CNN and CNN localization methods. Faster R-CNN for landmark detection used whole images and labeled boundg boxes landmarks for trag. FFNN for localization Faster R-CNN localization method used labeled boundg boxes and correspondg geolocation data for trag. CNN for localization used whole images and correspondg geolocation data for trag. Experimental results proposed localization methods were generated with data test set as put both localization methods. The landmark detection tested Faster R-CNN performance with all images test set as puts, and compared results with correspondg boundg boxes test images. The proposed localization methods were tested usg all images test set as puts. The results from both localization methods were evaluated by comparg with correspondg geolocation data each test image Data Garg Geotagged images dataset were taken by a wheelchair robot equipped with camera, GPS receiver and compass sensor (Figure 5). The wheelchair robot is 55 cm width, 120 cm length and 140 cm height. Sensors for data garg were attached above seat. We used a Logitech C920 HD (Logitech, Lausanne, Switzerland) as robot camera, BU-353S4 (GlobalSat, Taipei, Taiwan ) as GPS receiver and an Octopus 3-axis digital compass sensor. All images dataset were taken from area near Koganei campus Hosei University, Japan. Two areas were selected for robot localization outdoor environments, as shown Figure 6. The length and width area 1 is 70 and 30 m, respectively. Area 2 is 75 m length and 30 m wide. The two areas for experiments were a distance 250 m from each or. There are different types landmarks available each area, which distguished one experimental area from anor. Durg data garg, robot was pushed by a human, and images were taken manually. Each time an image was taken, correspondg geolocation data was tagged to image automatically. The tagged geolocation data cludes location coordates and compass orientation. Location coordates were received from GPS receiver form a GGA message. Latitude and longitude formation side GGA message was extracted and tagged to image. Compass orientation was received from compass sensor, converted to magnetic compass orientation, before beg tagged to

8 Maches 2019, 7, Maches 2019, image. We7, collected 8 14 data different wear conditions order to crease robustness Maches 2019, 7, 8 14 proposed algorithms. (a) (a) (b) (b) Figure 5. Wheelchair robot equipped with sensors forfordata (a) view Figure Wheelchair robot equipped sensors datagarg: garg: (a) overall overall Figure Wheelchair robot equipped withwith sensors for data garg: (a) overall view view wheelchair wheelchair robot; (b) sensors for data garg: 1. Camera, 2. Global Positiong Systems (GPS) wheelchair robot; (b) sensors for data garg: 1. Camera, 2. Global Positiong Systems (GPS) robot; (b) sensors for data garg: 1. Camera, 2. Global Positiong Systems (GPS) receiver, and 3. receiver, and and 3. Compass sensor. receiver, 3. Compass sensor. Compass sensor. Figure6.6.Map Map experimental map). TheThe areas experiments are marked with Figure experimentalareas areas(google (Google map). areas experiments are marked with Figure 6. Map experimental areas (Google map). The areas experiments are marked with rectangles. rectangles. rectangles. DurgLabelg data garg, robot was pushed by a human, and images were taken manually Image Durg data robot was pushed by a geolocation human, anddata images taken manually. Each time an garg, image was taken, correspondg was were tagged to image All gared images dataset were hand-labeled with boundg boxestagged landmarks images. Eachautomatically. time an image was taken, correspondg geolocation data was to image The tagged geolocation data cludes location coordates and compass orientation. Ne types The landmarks were utilized for robot localization: FamilyMart, CocaCola, BicycleLane, automatically. tagged geolocation data cludes locationcoordates compass orientation. Location coordates were received from GPS receiver form and a GGA message. Latitude NoTruck, Crossg, Lawson, TimesParkg, LawsonParkg, and RoadSign 1. Figure 7 shows Location coordates were received from message GPS receiver form a GGA message. Latitude and longitude formation side GGA was extracted and tagged to image. Compass pictures se ne landmarks area experiments. Each boundg box is form and longitude compass GGA message extracted and tagged to orientation, image. Compass orientationformation was receivedside from sensor,was converted to magnetic compass before a vector which horizontal and conditions vertical position begwith tagged to member image. We compass collected contas dataconverted different ordercoordates to crease orientation wasfour received fromelements, sensor, towear magnetic compass orientation, before top-left corner, width, and height boundg box image. Unit position coordates, robustness proposed algorithms. beg tagged to image. We collected data different wear conditions order to crease and height boundg box is determed by number pixels. Horizontal and vertical width, robustness proposed algorithms Image Labelgare referenced from top-left corner image. For example, a boundg box position coordates that has a vector images {10, 20, 56, 72}dataset has itswere top-left corner at pixel number 10 horizontally Image All Labelg gared hand-labeled with boundg boxes landmarksand 20 vertically, and width and height were utilized box are for 56 and 72 localization: pixels, respectively. images. Ne types landmarks robot FamilyMart, CocaCola, All gared images dataset were hand-labeled with boundg boxes landmarks BicycleLane, NoTruck, Crossg, Lawson, TimesParkg, LawsonParkg, and RoadSign 1. images. Ne types landmarks were utilized for robot localization: FamilyMart, CocaCola, Figure 7 shows pictures se ne landmarks area experiments. Each boundg box is BicycleLane, NoTruck, Crossg, Lawson, TimesParkg, LawsonParkg, and RoadSign 1. form a vector with four member elements, which contas horizontal and vertical position Figure 7 shows pictures se ne landmarks area experiments. Each boundg box is coordates top-left corner, width, and height boundg box image. Unit position form a vector with four member elements, which contas horizontal and vertical position coordates, width, and height boundg box is determed by number pixels. Horizontal coordates top-left corner, width, and height boundg box image. Unit position coordates, width, and height boundg box is determed by number pixels. Horizontal

9 Maches 2019, 7, 9 14 and vertical position coordates are referenced from top-left corner image. For example, a Maches 2019, 7, boundg box that has a vector {10, 20, 56, 72} has its top-left corner at pixel number 10 horizontally and 20 vertically, and width and height box are 56 and 72 pixels, respectively. (a) (b) (c) (d) (e) (f) (g) (h) (i) Figure 7. Landmarks used experiments: (a) FamilyMart ; (b) CocaCola ; (c) BicycleLane ; (d) NoTruck ; (e) Crossg ; (f) Lawson ; (g) TimesParkg ; (h) LawsonParkg ; (i) (i) RoadSign Detection Experiments The goal landmark detection experiments was to evaluate performance Faster R-CNN, sce localization part is strongly related with landmark detection. All 427 images test set were processed through through Faster Faster R-CNN, R-CNN, and and embedded embedded with with boundg boundg boxes boxes and labels and labels landmarks landmarks detected detected by Fasterby R-CNN. Faster Evaluation R-CNN. Evaluation detection results detection cludes results qualitative cludes and qualitative quantitativeand tests. quantitative tests. The qualitative evaluation was done by analyzg detection results through human eyes. Some detection resultsfrom from Faster Faster R-CNN R-CNN on on images images test set testare setshown are shown Figure Figure 8. Most 8. Most generated generated boundg boundg boxes boxes are placed are placed well on well detected on detected landmarks landmarks with proper with proper positions positions and sizes. and Labels sizes. Labels attached attached to toboxes boxes correspond correspond to to classes classes landmarks shown shown Figure7. 7. However, some landmarks such as CocaCola Figure 8c has its boundg box placed area actual Maches landmark, 2019, but 7, box size did not match with landmark size Mean Average Precision (map) was used for quantitative evaluation landmark detection experiments. map is considered to be actual metric to measure accuracy object detectors. The map is mean value average precisions (AP) from all object classes. In this paper, we refer to this as landmark classes. AP is average maximum precisions at different recall values, which both precision and recall can be calculated by followg equations: TP P =, (1) TP + FP TP R =, (2) TP + FN where P is precision, R is recall, TP is amount correct boundg boxes comparg from detection and reference boxes dataset, FP is amount missed or misplaced boundg boxes that appeared detection results, and FN is amount missed boundg boxes that did not appear detection results, but existed reference dataset. The correct boundg boxes were measured from ratio tersection over union (IoU), which is ratio between tersection area and union area boundg boxes, comparg detection results with reference data. The higher IoU ratio means less detection error allowance, which can also reduce outcome (c) AP values. In (a) (b) this paper, IoU 0.5 and 0.7 were employed for measurg detection accuracy, similar to [17] which Figure used 8. an Samples IoU 0.7. images AP values test all landmark set classes Fasterand R-CNN mean landmark values detection (map) results: 0.5 and 0.7 IoU (a) ratio Sample values 1; (b) are Sample displayed 2; (c) Sample Table Table Mean Average 1. Precision precision (map) values was used detection for results quantitative from Faster evaluation R-CNN, with landmark 0.5 and detection 0.7 experiments. tersection map over is union considered (IoU). to be actual metric to measure accuracy object detectors. The map is mean value average precisions (AP) from all object classes. In this paper, we refer to Class AP0.5 AP0.7 1 ( FamilyMart ) ( CocaCola ) ( BicycleLane ) ( NoTruck ) ( Crossg ) ( Lawson ) ( TimesParkg )

10 Maches 2019, 7, this as landmark classes. AP is average maximum precisions at different recall values, which both precision and recall can be calculated by followg equations: P = R = TP TP + FP, (1) TP TP + FN, (2) where P is precision, R is recall, TP is amount correct boundg boxes comparg from detection and reference boxes dataset, FP is amount missed or misplaced boundg boxes that appeared detection results, and FN is amount missed boundg boxes that did not appear detection results, but existed reference dataset. The correct boundg boxes were measured from ratio tersection over union (IoU), which is ratio between tersection area and union area boundg boxes, comparg detection results with reference data. The higher IoU ratio means less detection error allowance, which can also reduce outcome AP values. In this paper, IoU 0.5 and 0.7 were employed for measurg detection accuracy, similar to [17] which used an IoU 0.7. AP values all landmark classes and mean values (map) 0.5 and 0.7 IoU ratio values are displayed Table 1. Table 1. Average precision values detection results from Faster R-CNN, with 0.5 and 0.7 tersection over union (IoU). Class AP 0.5 AP ( FamilyMart ) ( CocaCola ) ( BicycleLane ) ( NoTruck ) ( Crossg ) ( Lawson ) ( TimesParkg ) ( LawsonParkg ) ( RoadSign 1 ) Mean From Table 1, map values are and for 0.5 and 0.7 IoU, respectively. This implies that landmark detection accuracies Faster R-CNN are 82.80% for 0.5 IoU and 73.75% for 0.7 IoU. Though map values were higher than 80% when IoU is 0.5, map decreased to around 70% as IoU creased to 0.7. This means landmarks could be detected but may not be precise or have high accuracy. Comparg to well-configured examples presented [19 21], accuracy our Faster R-CNN was moderately lower. This reduction detection accuracy is cause a lower localization accuracy, as landmark detection results are required to generate localization results Faster R-CNN localization method Localization Experiments The localization methods presented this paper were implemented and evaluated several localization experiments. All images test set were processed Faster R-CNN localization and CNN localization methods to generate localization results. In addition to two proposed localization methods, we added a CNN localization method based on well-known CNN for classification called AlexNet [25]. We replaced last layers AlexNet with regression layers, similar to our second localization method. Trag AlexNet was same as our CNN second localization method. We employed AlexNet for localization as reference for our second localization method, sce re was no evaluation metric for testg our CNN design. Results from localization methods,

11 Maches 2019, 7, cludg location coordates latitude and longitude, and compass orientations were n passed on to evaluation. Evaluation localization results was done by calculatg absolute errors and distance between two pots, generated results and reference geolocation data test dataset. Three absolute errors were considered experiments: mean, mimum and maximum absolute errors. The mean absolute error is calculated from followg equation: MAE = 1 n n a i b i, (3) where MAE is mean absolute error, a is result from localization, b is reference value test dataset, and n is amount data test set, which was 427 experiments. The mimum and maximum absolute errors are smallest and largest values absolute errors. The distances between two pots were calculated from location coordates generated and reference data. We used haverse formula to calculate distances from latitude and longitude two pots. The haverse formula is widely used computer programmg to determe distance between two pots on a great sphere, which commonly referred to as Earth. The implemented haverse formula is as follows: D = 2 r arcs s 2 ( y2 y 1 2 ) i=1 ( ) + cos(y 1 ) cos(y 2 ) s 2 x2 x 1 2, (4) where D is distance between two pots kilometers, r is earth radius, which were applied as km [26], y 1 is latitude localization results radius, y 2 is reference latitude radius, x 1 is longitude localization results radius, and x 2 is reference longitude radius. In addition to absolute errors and distance errors, we also calculated standard errors from localization results and distance errors. The standard errors were calculated to measure deviations all results, which equation for standard errors can be described mamatically as; SE = σ n, (5) where SE is standard error, σ is standard deviation result, and n is amount data, which was 427 for test set. Table 2 shows each localization error and distances between real and generated robot location. The mean, mimum, maximum and standard errors are calculated from absolute errors. Localization errors are distances meters calculated from location coordates. It can be seen from Table 2 that Faster R-CNN localization method outperforms both CNNs terms location and distance errors. Mean absolute errors latitude and longitude from Faster R-CNN method are slightly lower than CNN methods, while mimum errors are also slightly lower case Faster R-CNN. There are some differences maximum errors latitude and longitude for Faster R-CNN, CNN and AlexNet. The AlexNet has lower errors than CNN localization, while Faster R-CNN yields least errors. These small errors latitude and longitude cause significant differences distance errors. The average distance error Faster R-CNN is 28 m which is less than half distance error CNN method (70 m). The reference AlexNet has an average distance error around 50 m. In case mimum errors, distance error from Faster R-CNN is less than 1 m, while both CNNs for localization have distance errors around 3 m. On maximum errors, Faster R-CNN method has a distance error around 177 m, which is lower than CNN localization methods that have a distance error 238 m. The reference AlexNet, however, gave distance error 322 m which is highest among maximum distance errors.

12 Maches 2019, 7, Table 2. Localization errors proposed methods. Errors Faster R-CNN CNN CNN (AlexNet) Mean Latitude Errors Longitude Compass Distance (m) M Latitude Errors Longitude Compass Distance (m) Max Latitude Errors Longitude Compass Distance (m) Standard Latitude Errors Longitude Compass Distance (m) However, performances Faster R-CNN localization suffer a decle compass accuracy. On average, compass orientations from Faster R-CNN method can have errors around 55. Comparg to average errors compass orientations from both CNNs, error from Faster R-CNN is higher, which mean orientations errors from CNN is only around 32 for our CNN and only 17 for AlexNet. In best case with mimum errors, CNN methods also gave good results with mimum compass error around 0.2 from our CNN and 0.03 error from AlexNet, which are smaller than mimum compass error from Faster R-CNN method 0.3. For worst case, all localization methods have maximum compass errors near 180, while our CNN gives error 152 which is smallest among maximum compass errors. The standard errors dicated that latitude and longitude coordates resulted from all localization methods share similar deviation, while Faster R-CNN method has higher compass differences, and CNN methods have higher distance error differences results. 4. Conclusions This paper proposed and tested two outdoor localization methods based on CNN and Faster R-CNN for mobile robots. The performance was evaluated outdoor environment localization tasks. In addition, AlexNet was implemented order to compare performance. Faster R-CNN localization method was also tested for landmark detection. Results from landmark detection yielded good performance, with more than 70% detection accuracy. Good detection performance Faster R-CNN led to good localization performance Faster R-CNN based method, with approximately less than 1 m distance error and less than 1 compass error best case. The CNN localization method also had good performance best case, with approximately 3 m location error and less than 1 compass error. The results from average and worst cases poted that Faster R-CNN performs best among tested approaches for localization tasks, while performance decles for compass orientations. However, re is still space to improve localization results. The average location errors proposed methods were relatively high, compared to GPS that has approximately 4.9 m error [7]. The orientation errors our proposed methods were relatively high resultg a poor performance compared with reference AlexNet. Some possible causes high orientation errors are as follows: The development and experiments proposed localization methods were done with a small amount data compared with or works that use more than half a million data.

13 Maches 2019, 7, There were less environmental variations durg data garg, despite attempts on garg data multiple environmental conditions. Small environmental variations can be cause poor performance. The performance Faster R-CNN localization method relied on landmark detection. If we improve landmark detection, it will improve performance Faster R-CNN localization. Despite small amount data, proposed Faster R-CNN and CNN localization methods performed well for robot localization tasks. In future we will focus on improvg performance localization methods and implement m on real robot. We will focus on contuous learng or transfer learng, order to improve performance without creasg amount data. Author Contributions: Conceptualization, S.N. and G.C.; methodology, S.N.; stware, S.N.; validation, S.N., D.H. and G.C.; formal analysis, S.N.; vestigation, S.N.; resources, G.C.; data curation, S.N.; writg origal draft preparation, S.N.; writg review and editg, S.N., D.H. and G.C.; visualization, S.N.; supervision, S.K. and G.C.; project admistration, G.C. Fundg: This research received no external fundg. Conflicts Interest: The authors declare no conflict terest. References 1. Roland, S.; Illah, R.N. Mobile Robot Localization. In Introduction to Autonomous Mobile Robots, 1st ed.; Bradford Company Scituate: Cambridge, MA, USA, 2004; pp Muhammad, A.G.; Kundan, K.; Muhammad, A.J.; Muhammad, S. High-voltage transmission le spection robot. In Proceedgs International Conference on Engeerg and Emergg Technologies (ICEET), Lahore, Pakistan, February Seiko, P.Y.; Filip, K.; Takanori, E.; Yukori, K.; Tadeusz, U. Autonomous position control multi-unmanned aerial vehicle network designed for long range wireless data transmission. In Proceedgs 2017 IEEE/SICE International Symposium on System Integration (SII), Taipei, Taiwan, December Richa, W.; Muhammad, R.; Roby, A.W.; Ontoseno, P. Path planng mobile robot usg waypot for gas level mappg. In Proceedgs 2017 International Semar on Intelligent Technology and Its Applications (ISITIA), Surabaya, Indonesia, August Bruno, V.M.; Sebastián, A.V.; Alejandro, R.; Gerardo, G.A. GPS aided strapdown ertial navigation system for autonomous robotics applications. In Proceedgs 2017 XVII Workshop on Information Processg and Control (RPIC), Mar del Plata, Argenta, September Shamsud, A.U.; Ohno, K.; Hamada, R.; Kojima, S.; Westfechtel, T.; Suzuki, T.; Okada, Y.; Tadokoro, S.; Fujita, J.; Amano, H. Consistent map buildg petrochemical complexes for firefighter robots usg SLAM based on GPS and LIDAR. Robomech 2018, 5, [CrossRef] 7. GPS.gov: GPS Accuracy. Available onle: (accessed on 26 December 2018). 8. Kumar, V.; Jawahar, C.V.; Visesh, C. Accurate localization by fusg images and GPS signals. In Proceedgs 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Boston, MA, USA, 7 12 June Chang, L.; Honglun, W.; Na, L.; Yue, Y. Sensor fault diagnosis GPS/INS tightly coupled navigation system based on state chi-square test and improved simplified fuzzy ARTMAP neural network. In Proceedgs 2017 IEEE International Conference on Robotics and Biomimetics (ROBIO), Macau, Cha, 5 8 December Santos, E.R.S.; Azpurua, H.; Rezeck, P.A.F.; Corrěa, M.F.S.; Freitas, G.M.; Macharet, D.G. Global localization mobile robots usg local position estimation a geo tagged wireless node sensor network. In Proceedgs Lat American Robotic Symposium, 2018 Brazilian Symposium on Robotics (SBR) and 2018 Workshop on Robotics Education (WRE), Joao Pessoa, Brazil, 6 10 November Nilesh, S.; Peshala, G.J.; Takashi, K. 3D pose trackg for GPS-denied terra rovers by fast state variable extension and enhanced motion model. In Proceedgs th International Conference on Control, Automation and Systems (ICCAS), Jeju, South Korea, October 2017.

14 Maches 2019, 7, Zhou, B.; Tang, Z.; Qian, K.; Fang, F.; Ma, X. A LiDAR Odometry for Outdoor Mobile Robots Usg NDT Based Scan Matchg GPS-denied environments. In Proceedgs IEEE 7th Annual International Conference on CYBER Technology Automation, Control, and Intelligent Systems (CYBER), Honolulu, HI, USA, 31 July 4 August Kottath, R.; Yalamandala, D.P.; Poddar, S.; Bhondekar, A.P.; Karar, V. Inertia constraed visual odometry for navigational applications. In Proceedgs th International Conference on Image Information Processg (ICIIP), Shimla, India, December Saska, M.; Baca, T.; Thomas, J.; Chudoba, J.; Preucil, L.; Krajnik, T.; Faigl, J.; Loianno, G.; Kumar, V. System for deployment groups unmanned micro aerial vehicles GPS-denied environments usg onboard visual relative localization. Auton. Robots 2017, 41, [CrossRef] 15. Hannes, S.; Peter, Z.; Frank, H.; Eric, S. GPS-dependent localization for f-road vehicles usg ultra-wideband (UWB). In Proceedgs 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), Yokohama, Japan, October Shuhui, J.; Yu, K.; Yun, F. Deep Geo-constraed Auto-encoder for Non-landmark GPS Estimation. IEEE Trans. Big Data 2017, press. [CrossRef] 17. Ren, S.; He, K.; Girshick, R.; Sun, J. Faster R-CNN: Towards real-time object detection with region proposal networks. In Proceedgs 29th Annual Conference on Neural Information Processg Systems (NIPS), Montreal, QC, Canada, 7 12 December Girshick, R. Fast R-CNN. In Proceedgs 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, 7 13 December Saleh, K.; Hossny, M.; Hossny, A.; Nahavandi, S. Cyclist detection LIDAR scans usg faster R-CNN and syntic depth images. In Proceedgs 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), Yokohama, Japan, October Zhang, H.; Du, Y.; Ng, S.; Zhang, Y.; Yang, S.; Du, C. Pedestrian Detection Method Based on Faster R-CNN. In Proceedgs th International Conference on Computational Intelligence and Security (CIS), Hong Kong, Cha, December Wang, R.; You, Y.; Zhang, Y.; Zhou, W.; Liu, J. Ship detection foggy remote sensg image via scene classification R-CNN. In Proceedgs 6th IEEE International Conference on Network Infrastructure and Digital Content (IC-NIDC), Guiyang, Cha, August Robert, G.; David, H.J.J.; Heiko, H.S.; Jonas, R.; Matthias, B.; Felix, A.W. Comparg deep neural networks agast humans: Object recognition when signal gets weaker. arxiv, 2017; arxiv: v Epste, R.A.; Vass, L.K. Neural systems for landmark-based wayfdg humans. Philos. Trans. R. Soc. B Biol. Sci. 2013, 369, 1 7. [CrossRef] [PubMed] 24. Bayar, B.; Stamm, M.C. Design Prciples Convolutional Neural Networks for Multimedia Forensics. Electron. Imagg Med. Watermark. Secur. Forensics 2017, 10, [CrossRef] 25. Alex, K.; Ilya, S.; Gefrey, E.H. ImageNet classification with deep convolutional neural networks. In Proceedgs 25th International Conference on Neural Information Processg Systems (NIPS 12), Lake Tahoe, NV, USA, 3 6 December [CrossRef] 26. Mamajek, E.E.; Prsa, A.; Torres, G.; Harmanec, P.; Asplund, M.; Bennett, P.; Capitae, N.; Christensen-Dalsgaard, J.; Depagne, É.; Folkner, M.W.; et al. Resolution B3 on Recommended Nomal Conversion Constants for Selected Solar and Planetary Properties. In Proceedgs 29th IAU General Assembly (IAU 2015), Honolulu, HI, USA, 3 14 August by authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under terms and conditions Creative Commons Attribution (CC BY) license (