厦门大学博硕士论文摘要库

Size: px

Start display at page:

Download "厦门大学博硕士论文摘要库"

廉棼常
4 years ago
Views:

5 摘要随着互联网和多媒体技术的不断发展, 海量的图像资源和数据涌入互联网, 它们深刻地影响着我们的日常生活, 此时对于图像内容准确地理解显得尤为重要图像分类技术融合了机器学习人工智能等学科的先进思想和理论, 旨在将原本无序分布的大批量的图像数据进行有序地归类, 是解决图像理解问题的基础随机森林算法建立在决策树模型基础之上, 它是多棵决策树的组合, 在分类场景中得到广泛应用作为一种优秀的分类器模型, 也为图像分类提供了新的思路但当图像资源的数据量特别大时, 分类将是一个十分耗时的过程为了解决以上问题, 本文结合 Hadoop 开源分布式计算平台, 研究图像处理过程的并行化以及利用 MapReduce 并行计算模型对随机森林算法进行了分布式并行设计以提高对大规模图像分类的效率首先, 本文介绍了 Hadoop 平台, 主要包括 HDFS 文件系统以及 MapReduce 框架 ; 接着, 分析了图像分类的关键技术, 总结了图像分类中常用到的一些特征, 重点介绍 SIFT 特征和 BagofVisual Words 特征代表的实现过程, 并阐述了几类重要的模式分类方法 ; 随后, 重点介绍随机森林算法, 并在此基础之上, 结合 MapReduce 计算框架, 对随机森林模型构建过程进行双重并行化的改进, 以提高随机森林算法的运行效率 ; 然后, 基于以上研究成果, 本文构建了基于 Hadoop 平台的图像分类原型系统, 包括 Hadoop 平台下处理图像接口的实现基于 DenseSift 算法的特征采样过程基于 BoVW 模型表示图像构建图像的空间金字塔模型和利用改进后的随机森林算法进行分类器的训练共五个部分最后, 通过实验验证了基于 Hadoop 平台的图像并行化处理地有效性, 同时证明并行化的随机森林算法相比较于单机版的随机森林, 处理过程所消耗的时间大为缩短, 并取得了不错的分类效果基于 Hadoop 平台的图像并行化处理过程, 大大提高了海量图像处理过程的速度, 同时针对于图像分类, 改进后的随机森林算法在效率上有了明显的提高关键词 : 图像分类 ; 随机森林 ;Hadoop;SIFT; 词袋模型 I

6 II

7 Abstract With the development of Internet and multimedia technology,large-scale digital image and data has come into Internet and change our lives. It s very important to let computer understand image s content in detail. Intergrated with other subjects like machine learning,artificial intelligence and etc,the goal of image classification technology is to categorize a large number of digital images into a certan class automatically. Image classification is very important to solve image understanding problem.random forest algorithm is based on decision tree model.it is made up of a series of decision trees and classification result is better.it is widely used in object classification field.it also provides a new way to manage the image classification problem.but, it is a time-consuming process on massive image data.to solve the problem mentioned above,this thesis studys the parallelization of image processing based on Hadoop platform and parallelized random forest algorithm with MapReduce scheme.the main work of the thesis is summarized as below: Firstly,this thesis introduces the Hadoop platform.it includes HDFS distributed file system and MapReduce framework.secondly,this thesis discusses the key technology of image classification.this part summarizes the commonly used features and focus on scale invariant feature transform and Bag of Visual Words model.and several important pattern classification methods has been introduced lately.thirdly,the parallelization of random forest algorithm has been studied. With the double parallelization of random forest algorithm,the operational efficiency of the algorithm has been improved.based on previous work,this thesis build an image classification system with hadoop image processing interface,dense sift extraction,bag of words model,build spatial pyramid and random forest classifier training part.finally,the experiments show that the algorithm can improve the time and performance. Parallelized image processing method based on Hadoop platform greatly increased the speed of processing massive image data.and Random Forest parallelization method also improve the efficiency of image classification. Key words: image classification; random forest; hadoop; sift; bag of words III

8 IV

9 目录第一章绪论研究背景及意义国内外研究现状图像分类问题研究现状随机森林算法研究现状 Hadoop 平台应用现状本文的主要研究内容本文的结构安排... 7 第二章 Hadoop 技术架构概述 Hadoop 简介项目背景 Hadoop 的基本架构模型分布式文件系统 HDFS 设计原则 NameNode 与 DataNode 数据存储编程模式 MapReduce 本章小结第三章图像分类问题研究综述图像特征提取底层特征 : 颜色纹理和形状 SIFT 特征 Bag of Visual Words 特征分类算法朴素贝叶斯 K 近邻 BP 神经网络支持向量机 V

10 3.3 本章小结第四章随机森林算法基本理论决策树及相关概念决策树算法决策树属性的选择决策树的剪枝随机森林算法定义随机性的引入随机森林的生长随机森林的分类本章小结第五章基于 MapReduce 的随机森林算法并行化研究 Mahout 中随机森林的并行化实现随机森林并行化算法进一步改进本章小结第六章基于 Hadoop 的图像分类系统设计与实现本文图像分类系统简介基于 Hadoop 的图像并行化处理 HipiImageBundle 格式键值对类型 ImageHeader/FloatImage 作业的输入格式 ImageBundleInputFormat 基于 DenseSift 算法和 BoVW 模型构建图像空间金字塔表示 BoVW 模型在图像分类场景中的应用简介 BoVW 模型构建及表征图像构造空间金字塔基于分布式 Random Forest 算法构建分类器输入数据准备运行过程本章小结 VI

11 第七章实验设计及结果分析实验数据集评价标准实验环境搭建实验环境系统部署实验结果分析分类结果分析并行化随机森林分析本章小结第八章总结与展望参考文献致谢 VII

12 VIII

13 Contents Chapter1 Intoduction Research Background and Motivation Research and Status Image Classification Research Status Random Forest Algorithm Research Status Hadoop Platform Contents Introduction Structure of Thesis... 7 Chapter2 Hadoop Platform Brief Introduction of Hadoop Background Basic Structure of Hadoop Distributed Filesystem HDFS Design Princible NameNode and DataNode Data Storage Program Pattern MapReduce Conclusion Chapter3 Summary Of Image Classification Study Image Feature Extraction Low-level Feature:Color,Texture and Shape SIFT Feature Bag of Visual Words Feature Classification Algorithm Naïve Bayes K Nearest Neighbor BP Neural Network Support Vector Machine IX

14 3.3 Conclusion Chapter4 Random Forest Algorithm Introduction Decision Tree and Related Concepts Decision Tree Algotithm Choice of Attributes Pruning of Trees Random Forest Definition Import of Randomness Build of Random Forest Classification of Random Forest Conclusion Chapter5 Parallelization of Random Forest Based on MapReduce Parallelization of Random Forest in Mahout Improvement on Parallelization of Random Forest Conclusion Chapter6 Implementation of Image Classification System Brief Introduction of My Image Classification System Image Processing Base on Hadoop HipiImageBundle Format Key/Value Type ImageHeader/FloatImage Input Format ImageBundleInputFormat Build Spatial Pyramid Based on DenseSift and BoVW Model Image Classification Based on BoVW Model Image Representation By BoVW Model Build Spatial Pyramid Build Classifier Based on Parallelized Random Forest Preparation of Input Data Running Process Conclusion Chapter7 Experiment and Result Anaylization X

15 7.1 Dataset Evaluation Criterion Experiment Environment Set Up Environment System Deployment Result Analyzation Classification Result Analyzation Distributed Random Forest Algorithm Analyzation Conclusion Chapter8 Conclusion and Prospect Reference Acknowledgement XI

16 XII

17 Degree papers are in the Xiamen University Electronic Theses and Dissertations Database. Full texts are available in the following ways: 1. If your library is a CALIS member libraries, please log on and submit requests online, or consult the interlibrary loan department in your library. 2. For users of non-calis member libraries, please mail to etd@xmu.edu.cn for delivery details.

5 5 7 5 4 4 6 9 8 40

5 5 7 5 4 4 6 9 8 40 4 5 0 959 68 959 55 959 4 957 4 5 998 4 9 959 98 8 980 8 84 4 987 4 5 6 7 8 9 986 998 989 4 977 8 5 988 6 985 8 7 98 990 5 8 998 4 9 987 4 5 4 96 9 959 8 959 4 986 4 5 5 8 96 4 5