幻灯片 1

Size: px

Start display at page:

Download "幻灯片 1"

鸯峨郝
5 years ago
Views:

1 高维数据可视化巫英才

2 大数据和高维数据可视化

3 目录数据维度高维数据可视化数据变换数据呈现数据交互

4 数据维度

5 一维数据

6 二维数据

7 三维数据 Elmqvist et al. IEEE TVCG 2008

8 多维数据身高体重年龄性别教育程度籍贯张三 180cm 65kg 23 男大学上海李四 168cm 55kg 18 女高中浙江赵五 175cm 75kg 53 男初中广东

9 简单的思路增加视觉通道, 以表达更多的属性信息散点的形状填充形式颜色大小等

10 简单的思路多视图协调关联

11 更高维度

12 高维数据可视化

13 数据变换

14 降低维度使用线性或非线性变换把高维数据投影到低维空间投影保留重要的关系 ( 无信息损失 ; 保持数据区分等 ) 降低维度

15 方法线性方法 * 主成分分析 (PCA) * 多维尺度分析 (Multidimensional Scaling,MDS) * 非负矩阵分解 (NMF) 非线性方法 * ISOMAP * 局部线性嵌套 (LLE)

16 主成分分析 (PCA) 最大化投影后的方差最小化投影后的损失第二主成分数据点 * * * * * * * * * * * * * * * * * * * * * * * * 第一主成分原始坐标轴

17 主成分分析 (PCA) 最大化投影后的方差最小化投影后的损失第二主成分数据点 * * * * * * * * * * * * * * * * * * * * * * * * 第一主成分原始坐标轴

18 方差 (1/2) 方差描述变量的信息量 y x 数据点 Noise * * * * * * * * * * * * * * * * * * * * * * * * Signal

19 方差 (2/2) 空间的中心点 xx = 1 NN NN ii=1 xx ii 假设 uu 为投影向量, 投影后的方差为 1 NN NN ii=1 NN (uu TT xx ii uu TT xx) 2 = 1 NN ii=1 uu TT (xx ii xx)(xx ii xx) TT uu 协方差矩阵方差为 NN SS = 1 NN ii=1 (xx ii xx)(xx ii xx) TT uu TT SSSS

20 最大化投影后的方差方差为 uu TT SSuu 最大化此方差 Maximize (uu TT SSuu) 同时满足 uu TT uu = 1 运用拉格朗日乘子法 uu TT SSuu + λλ 1 uu TT uu 将上式求导, 使之为 0, 得到 : SSSS = λλuu

21 特征向量也就是 SSuu = λλuu uu TT SSuu = λλ 降维后方差所对应的是协方差矩阵的特征值为了使方差最大, 选择最大的特征值最大特征值所对应的特征向量为最佳投影方向

22 特征值谱

23 PCA 应用于脸部数据 64 x 64 = 4096

24 PCA 应用于脸部数据把每个脸部重建为一系列人脸基或特征人脸的线性组合平均的人脸

25 重建 90% 的方差是可以被前 50 个特征向量捕获只需 50 张基图像就可以重建存在的人脸

26 问题 : 基于 PCA 的可视化过程主成分难以理解空间转换难以联系方法 : 基于 PCA 的可视化过程 Müller et al. APVIS 2006

27 用第一主成分排列数据

28 关联映射空间和原始空间

29 多维尺度分析 (MDS) 基于数据集相似程度的降维方法在某些情况下, 只能够衡量数据点之间的距离

30 输入多维尺度分析 (MDS) 数据点 X 间的相似矩阵 M, 以及投影的维度 K 输出所有数据点在 K 维平面上的坐标 Y 投影空间 ( 平面 ) 中点对间的相似度尽量逼近原始空间的相似度

31 目标函数多维尺度分析运用优化的方法来降维 JJ YY = ii<jj(dd ii,jj δδ ii,jj ) 2 ii<jj δδ ii,jj 2 δδ ii,jj 是数据点 i 和 j 之间在原始空间的相似度 dd ii,jj 是数据点 i 和 j 之间在 K 空间的相似度 YY 是数据集在 K 空间的投影

32 MDS 与 PCA 如果将数据点的相似度定义为数据点之间的欧式距离, 那么 MDS 等价于 PCA MDS 允许定义不同的相似度, 因而更加灵活

33 已知一些城市间的距离实例 : 美国地图

34 MDS 的结果

35 MDS 的运用案例分析 Cui et al. IEEECG&A 2010

36 文档数据

37 商业咨询和情报分析文本分析

38 字云

39 不适合做对比分析字云

40 直接的方法

41 挑战 (1/2) 富有语义意义布局 V.S. 紧凑的布局

42 挑战 (2/2) 稳定的动态布局

43 基本概念一个话题由一组语义相关的关键词所定义两个关键词如果经常在文本集里面出现在一起, 他们就被认为是语义相关的

44 我们的方法基于上下文的动态字云可视化方法语义关系和空间利用多个字云之间的稳定布局

45 趋势线图 + 动态字云系统总揽图

46 基于信息熵字云的相似度 H(X): X 所含的信息 H(X;Y): X 和 Y 所共有的信息 H(X Y): X 所独有的信息衡量当前时间点所独有的信息

47 关键词提取 1. 对文档做分词, 生成一个单词集合 2. 用 Porter Stemmer 清理单词 3. 生成一个 co-occurrence graph 4. 用 LexRank 提取关键词

48 生成初始字云 1. 生成关键词的相似矩阵 2. 利用 MDS 生成关键词的散点图 3. 将关键词放入相应的位置, 利用力学模型生成字云

49 基于力的字云生成

50 Example: AIG News Data 13,828 articles (Jan. 14, 2008 to Apr. 5, 2009)

51 Example: AIG News Data

52 Example: Apple News 1933 news articles from NYT ( )

53 Example: Apple News 1933 news articles from NYT ( )

54 Example: Apple News 1933 news articles from NYT ( )

55 数据呈现

56 高维数据呈现方法基于点的方法散点矩阵径向布局基于线的方法线图平行坐标径向轴基于区域的方法柱状图表格显示像素图维度堆叠马赛克图基于样本的方法切尔诺夫脸谱图邮票图

57 高维数据呈现方法基于点的方法散点矩阵径向布局基于线的方法线图平行坐标径向轴基于区域的方法柱状图表格显示像素图维度堆叠马赛克图基于样本的方法切尔诺夫脸谱图邮票图

58 散点矩阵

59 散点矩阵使用一个二维散点图表达每对维度之间的关系直观显示两个维度间的相关性散点图数目与数据维度平方成正比

60 改进使用自动方法寻找散点图矩阵中可能感兴趣的散点图聚类特征维度相关性特征 A. Tatu, G. Albuquerque, M. Eisemann, P. Bak, H. Theisel, M. Magnor, and D. Keim, Automated Analytical Methods to Support Visual Exploration of High-Dimensional Data, IEEE Transactions on Visualization and Computer Graphics, vol. 17, no. 5, pp , 2011.

61 有特征的维度对

62 径向布局法

63 径向布局法基于弹簧模型的圆形布局方法将代表 N 维的 N 个锚点至于圆周上根据 N 个锚点作用的 N 种力量将数据点散布于圆内

65 线图

66 线图单变量可视化方法通过多子图多线条等方法可以延伸表示高维数据通过不同的视觉通道编码不同的数据属性

67 Twitter 上的爆炸性新闻的传播 Hu et al. ACM CHI 2012

68 Twitter broke Bin Laden s death

69 Twitter broke Bin Laden s death 9:45 pm: White House announces Pres. Obama speech

70 Twitter broke Bin Laden s death 10:24 pm: Keith Urbahn tweets about Osama rumor

71 Twitter broke Bin Laden s death 10:45 pm: ABC, NBC & CBS report Bin Laden s death

72 Twitter broke Osama Bin Laden s death 11:30 pm: Pres. Obama begins remarks

73 Data May 1 st 10:20 pm EST May 2 nd 12:20 am EST laden 614,976 tweets Sampled (roughly 10%)

74 Did Twitter Break the news? Tweets posted between 10:20 and 10:45 pm

75 News Jill Jackson CBS News Capitol Hill Keith Urbahn - Chief of staff of former defense secretary Donald Brian Stelter New York Times reporter Mentions per min @brianstelter 22:21 22:23 22:25 22:28 22:30 22:33 22:35 22:37 22:39 22:41 22:43 22:45 Time

76 News breakers Mentions per min @brianstelter 10:24 pm 10:32 pm 22:21 22:23 22:25 22:28 22:30 22:33 22:35 22:37 22:39 22:41 22:43 22:45 Time

77 Did Twitter convince its audience? All English Tweets in data set: 420,134 Certainty classification Certain: They caught Osama Bin Laden! Uncertain: Rumor, Bin Laden dead. Don t know for sure. Irrelevant: Obama: "the US is not and never will be at war with Islam Bin Laden was not a Muslim leader; he was a mass murderer... Classifier Training set: 300 tweets, 2 raters agreed on 235 (78.3%) SVM with bag-of-words as features Cross-validation: 75.8% overall confidence

78 Did Twitter convince its audience? Keith Urbahn % of tweets classified as certain ABC, NBC and CBS 22:21 22:28 22:35 22:41 22:46 22:51 22:55 22:59 23:02 23:06 23:09 23:13 23:16 23:20 23:23 23:27 23:30 23:34 23:37 23:41 23:45 23:48 23:52 23:56 0:00 0:05 0:10 0:15 Time Obama speech

79 Who generated the buzz? 100 accounts mentioned in 18.10% of sampled tweets Mainstream @cnnbrk Media Twitter Real-life Political people & Osama, News

80 Who generated the buzz? Mentions per min media people mass media celebrities :21 22:43 22:57 23:09 23:21 23:32 23:45 23:57 0:14 Time 80

81 What did people share? 9.69% tweets contain links Top 26 sites made up 58.8% of total valid links URL number / tweet number 22:21 22:28 22:35 22:41 22:46 22:51 22:55 22:59 23:02 23:06 23:09 23:13 23:16 23:20 23:23 23:27 23:30 23:34 23:37 23:41 23:45 23:48 23:52 23:56 0:00 0:05 0:10 0:15 81

82 cnn.com msn.com nytimes.com twitpic.com yfrog.com globo.com twitlonger.com youtube.com tumblr.com reuters.com tmi.me lockerz.com mashable.com go.com bbc.co.uk wsj.com facebook.com whitehouse.gov huffingtonpost.com aljazeera.net yahoo.com google.com foxnews.com uol.com.br blogspot.com globovision.com Mass media and government 64.07% Video & image sharing, social networks, blogs and tweet-shortening services 35.93%

83 Discussion People are willing to trust information from social media if the sources are credible Concentration of attention on elite users Different groups of elite users play different parts Journalists utilize personal social media accounts to break news

84 高维数据分析案例酒店客户点评信息的可视分析

85 Customer Opinion Analysis Online customer feedback Customer behavior analysis Businesses Examples Identify influencing factors Avoid negative word-of-mouth Customer feedback from TripAdvisor.com Word-of-mouth

86 Analysis Task NOT for superficial questions, e.g., What is the overall customer opinion? What are the difference of customer opinions?

87 Traditional Analysis Approach A content analysis procedure Complaints are classified into categories Further relationship analysis is conducted It becomes challenging Large-scale Heterogeneous High-dimensional

88 Visual Text Analysis Visual analytics is the science of analytical reasoning facilitated by interactive visual interfaces [Thomas and Cook, 2005] Visual text analysis Visual analytics of text data Appropriate for customer analysis Keim et al., 2006

89 Opinion Visualization Morinaga et al. (KDD 2002) Gamon et al. (IDA 2005) Chen et al. (VAST 2006) Wanner et al. (VISSW2009) Gamon et al. (ICWSM 2008) Gregory et al. (SST 2006)

90 Challenges Challenging customer opinion analysis High-dimensional and heterogeneous data Ambiguous meanings of words Uncertainty and Trustworthiness No clear boundary between opinions

91 Our Solution: OpinionSeer Communicate opinion-mining results and facilitate the analytical reasoning process

92 Feature-based Opinion Mining A feature-based opinion-mining technique Major features (room, location, etc) Feature-entity mapping scheme Sentiment keyword dictionary Mining results A positive and a negative scores for each feature

93 Uncertainty Modeling (1/2) Uncertainty in the opinion mining results Simultaneous positive and negative opinions Inaccurate detection of opinions Uncertainty modeled by Gaussian

94 Uncertainty Modeling (2/2) The overall uncertainty score for a feature u = α*n +/- (μ 1, σ 1 ) + β*(1-n (μ 2, σ 2 )) N +/- indicates uncertainty from the difference between the positive and negative scores N denotes the certainty from the sentence length α = β = 0.5 The extracted opinion is (b, d, u) b and d: positive score and negative score u: uncertain score b + d + u = 1

95 Opinion Combination Combine multiple selected opinions Overall opinion of a group of products Overall opinion of a group of features General approaches do not work Negative Opinion Positive Opinion

96 Subjective Logic Subjective logic: a type of probabilistic logic Every opinion is represented as (b, d, u) b and d : positive and negative scores u: uncertainty A number of opinion operators defined in subjective logic

97 AND Operator Combine the opinions of a customer on multiple features The combined opinion ω x y of two opinions (b x, d x, u x ) and (b y, d y, u y ) for feature x and y

98 Fusion Operator Combine the opinions of multiple customers on the same feature The fused opinion of two opinions and for feature Customer A and Customer B on feature x

99 Opinion Visualization Opinion Wheel Opinion triangle Opinion rings

100 Opinion Triangle Opinion P (b,d,u) b: positive d: negative u: uncertain b + d + u = 1 Negative Opinions Uncertain P u Uncertain Opinions Positive Opinions Negative Positive Equilateral triangle

101 Opinion Ring (1/2) Opinion correlations Five radial layouts Size -> volume color -> category

102 Opinion Ring (2/2) Temporal and geographic rings

103 Data for Experiments Experimental data Customer feedback from Hong Kong Hotel at TripAdvisor.com High diversity of the cultural backgrounds of the customers

104 Experiments Comparison of two opinion groups of high-class hotels and low-class hotels in terms of trip type Group A Group B

105 Experiments Correlation between customer opinions and their trip types Trip type

106 Experiments Correlation between customer opinions and their genders

107 Experiments Correlation between customer opinions and their ages

108 Experiments Geography patterns Uncertainty Uncertainty Negative Positive Negative Positive

109 Experiments

110 谢谢巫英才 Bing Yingcai Wu to find my webpage

「本港青少年網上行為」調查

「本港青少年網上行為」調查 1) 2) Analysis of Local Net Group : the Implications of the Study on Net Behavior of Youth in Hong Kong CHAN King Chuen Information & Research Manager, Breakthrough CHAN Chi Fu Investigator of Research