PowerPoint 演示文稿

Size: px

Start display at page:

Download "PowerPoint 演示文稿"

琇宓
7 years ago
Views:

1 CSC 2523 Deep Learning in Computer Vision Winter 2016 Neural-based Image Question Answering Yunpeng Li Faculty of Information

2 Question Answering Traditional Approaches Textual question answering tasks Semantic parsing Symbolic representation Deduction system Image Credit: Question Answering over Linked Data: Challenges, Approaches & Trends ESWC 2014)

Deduction system Image Credit: Question Answering over

3 Image Question Answering Multi-modal problem Image Representation QA Bed What is on the right side of the cabinet? Natural Language Processing Using both visual & natural language inputs

4 Image Question Answering Neural-based Approach CNN QA Bed What is on the right side of the cabinet? LSTM Both can be processed with deep neural networks

5 Neural-based Question Answering Architecture Malinowski et al. (2015). Ask your neurons: A neural-based approach to answering questions about images. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1-9).

6 Neural-based Question Answering Architecture Malinowski et al. (2015). Ask your neurons: A neural-based approach to answering questions about images. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1-9).

7 Neural-based Question Answering Architecture Malinowski et al. (2015). Ask your neurons: A neural-based approach to answering questions about images. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1-9).

8 Neural-based Question Answering Training GoogleNet or AlexNet pretrained on ImageNet dataset Loss function: Cross entropy Malinowski et al. (2015). Ask your neurons: A neural-based approach to answering questions about images. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1-9).

Ask your neurons: A neural-based approach to answering questions about

9 Result

10 Evaluation WUPS Similarity based on the depth of two words in WordNet WUP(curtain, blinds) = 0.94 WUP(carton, box) = 0.94 WUP(stove, fire extinguisher) = 0.82 The best weighted match between answer & truth Smaller threshold More forgiving metric Malinowski et al. (2015). Ask your neurons: A neural-based approach to answering questions about images. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1-9). Malinowski, M., & Fritz, M. (2014). A multi-world approach to question answering about real-world scenes based on uncertain input. In Advances in Neural Information Processing Systems (pp ).

Ask your neurons: A neural-based approach to answering questions about images. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1-9).

11 Evaluation Result DAQUAR dataset 12, 468 human question answer pairs on images of indoor scenes Malinowski et al. (2015). Ask your neurons: A neural-based approach to answering questions about images. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1-9).

12 Evaluation Consensus How plausible are the ground truths? Malinowski et al. (2015). Ask your neurons: A neural-based approach to answering questions about images. In Proceedings of the IEEE International Conference on Computer Vision (pp. 1-9).

13 Exploring Models and Data for Image Question Answering Architecture 19-layer Oxford VGG ConvNet trained on ImageNet Frozen during training Ren et al. (2015). Exploring models and data for image question answering. In Advances in Neural Information Processing Systems(pp ).

14 Question Generation Generate question-answer pairs from image captions. Ren et al. (2015). Exploring models and data for image question answering. In Advances in Neural Information Processing Systems(pp ).

15 Question Generation Generate question-answer pairs from image captions. Ren et al. (2015). Exploring models and data for image question answering. In Advances in Neural Information Processing Systems(pp ).

16 Evaluation Image CNN LSTM Answer Image CNN LR Answer Question Question BOW 1. VIS+LSTM 3. IMG+BOW Image CNN Bi-LSTM Answer Average of the others Question 2. 2-VIS+BLSTM 4. FULL DAQUAR COCOQA 12, 468 human question answer pairs 117,684 auto-generated question answer pairs Ren et al. (2015). Exploring models and data for image question answering. In Advances in Neural Information Processing Systems(pp ).

FULL DAQUAR COCOQA 12, 468 human question answer pairs 117,684 auto-generated question answer pairs Ren

17 Evaluation Overall Ren et al. (2015). Exploring models and data for image question answering. In Advances in Neural Information Processing Systems(pp ).

18 Evaluation Category Ren et al. (2015). Exploring models and data for image question answering. In Advances in Neural Information Processing Systems(pp ).

19 Evaluation

20 CSC 2523 Deep Learning in Computer Vision Winter 2016 这个模板, 是那年毕业, 你陪我一起做的快两年过去了, 往事依然历历在目, 我们却变了好多可我还是好想回到那年的时光, 想时间在我们身上停住, 我们可以一直好好地走下去或许是我太自负, 从来没相信过有一天你真的会离开所以只顾着埋头奋斗, 为了我们的未来, 却忽视了给你当下的幸福, 忘记浇灌我们的感情, 没有让你感受到我是多么深深地爱着你, 让你一颗炽热的心, 慢慢地变冷现在我终于明白了你想要的是什么, 明白了该如何去爱, 却已经失去了那个我爱的你可是, 假如没有这一次变故, 我也不会有机会审视自己, 只知道如以前一样待你这次发生的一切给了我一个机会, 可以提升爱的能力, 成为一个更完美的情人谢谢你教会我如何去爱, 只是很遗憾辜负了你, 委屈了你 Thank You! 你曾跟我说过对未来的设想, 热恋, 由于异地而隔阂, 你说自己没有毅力坚持, 会离开我, 又在明白之后回来捡我, 我以前一直以为你只是在表达自己的不安, 只会哄哄你现在已然走到了第四个阶段, 我只能希望一切都如你所料, 希望下个阶段尽快到来到那一天, 我会重新点燃你心中的爱意, 救赎我们的爱情我愿相信, 我们共有的不安分的灵魂, 会带你我走过千山万水, 最终回到彼此的身边如果分开不是为了别离, 那么一定是我们错过了一些重要的课, 需要你我一起补我接受这份考验, 只愿命运可以眷顾你我, 下一次, 在对的时间对的地点, 让我们遇上最好的彼此到了那一天, 我会紧紧抱住你, 再也不分开了一生短暂, 只能和最爱的人一起渡过 L.J. 我好想你 L.Y.P. Yunpeng Li Faculty of Information

白了你想要的是什么, 明白了该如何去爱, 却已经失去了那个我爱的你可是, 假如没有这一次变故, 我也不会有机会审视自己, 只知道如以前一样待你这次发生的一切给了我一个机会, 可以提升爱的能力, 成为一个更完美的情人谢谢你

Microsoft Word - 第7章图表反转形态.doc

Microsoft Word - 第7章图表反转形态.doc 第七章图表反转形态我们知道市场趋势共有三种 : 上升趋势下降趋势和横向整理市场的价格波动都是运行在这三种趋势中, 所有的走势都是这三种趋势的排列组合如图市场趋势结构示意图 7-1 所示市场趋势结构示意图 7-1 图市场趋