Proceedings of the 15th NTCIR Conference on Evaluation of Information Access Technologies December 8-11, 2020 Tokyo Japan Table of Contents Preface Preface from the NTCIR-15 General Chairs............................................. 1 Charles L. A. Clarke and Noriko Kando Overview Overview of NTCIR-15................................................................. 3 Yiqun Liu, Makoto P. Kato and Noriko Kando Keynote From Offline to Online Experimentation: Considerations from Experiences at Spotify... 7 Ben Carterette Reproducibility, Replicability and Reliability: Reflections of a Statistician and a Data Science Editor........................................................................... 8 Xiao-Li Meng Invited Talk TREC in 2020.......................................................................... 9 Ellen Voorhees What s happening in CLEF and what s the Covid-19 @ MLIA Initiative................ 10 Nicola Ferro MediaEval 2020 Multimedia Benchmarking Initiative................................... 11 Gareth Jones NTCIR-Lifelog - A Journey in Collaborative Benchmarking............................. 12 Cathal Gurrin DialEval-1 Overview of the NTCIR-15 Dialogue Evaluation (DialEval-1) Task...................... 13 Zhaohao Zeng, Sosuke Kato, Tetsuya Sakai and Inho Kang TMUDS at the NTCIR-15 DialEval-1 Task............................................. 35 Yen Chun Huang, Yi Hsuan Huang, Yu Ya Cheng, Jung Y Liao and Yung-Chun Chang SKYMN at the NTCIR-15 DialEval-1 Task............................................. 40 Junjie Wang, Yuxiang Zhang, Tetsuya Sakai and Hayato Yamana
WUST at the NTCIR-15 DialEval-1 Task............................................... 47 Xuan Zhang, Maofu Liu, Zhanzhao Zhou and Junyi Xiang TUA1 at the NTCIR-15 DialEval-1 Task............................................... 53 Xin Kang, Yunong Wu and Fuji Ren RSLNV at the NTCIR-15 Dialogue Evaluation (DialEval-1) Task....................... 57 Ting Cao, Fan Zhang, Haoxiang Shi, Zhaohao Zeng, Sosuke Kato, Tetsuya Sakai, Injae Lee, Kyungduk Kim and Inho Kang NKUST at the NTCIR-15 DialEval-1 Task.............................................. 62 Tao-Hsing Chang, Jian-He Chen and Chi-Chia Chen IMTKU Multi-Turn Dialogue System Evaluation at the NTCIR-15 DialEval-1 Dialogue Quality and Nugget Detection........................................................... 68 Mike Tian-Jian Jiang, Zhao-Xian Gu, Cheng-Jhe Chiang, Yueh-Chia Wu, Yu-Chen Huang, Cheng-Han Chiu, Sheng-Ru Shaw and Min-Yuh Day FinNum-2 Overview of the NTCIR-15 FinNum-2 Task: Numeral Attachment in Financial Tweets.. 75 Chung-Chi Chen, Hen-Hsen Huang, Hiroya Takamura and Hsin-Hsi Chen MIG at NTCIR-15: Use the transfer learning and feature engineering for the FinNum-2 task..................................................................................... 79 Yu-Yu Chen and Chao-Lin Liu TLR at the NTCIR-15 FinNum-2 Task: Improving Text Classifiers for Numeral Attachment in Financial Social Data.................................................... 83 Jose Moreno, Emanuela Boros and Antoine Doucet TMUNLP at the NTCIR-15 FinNum-2................................................. 88 Yu-Chi Liang, Yi-Hsuan Huang, Yu-Ya Cheng and Yung-Chun Chang CYUT at the NTCIR-15 FinNum-2 Task: Tokenization and Fine-tuning Techniques for Numeral Attachment in Financial Tweets................................................ 92 Mike Tian-Jian Jiang, Yi-Kun Chen and Shih-Hung Wu WUST at NTCIR-15 FinNum-2 Task................................................... 97 Xinxin Xia, Wei Wang and Maofu Liu QA Lab-PoliInfo-2 Overview of the NTCIR-15 QA Lab-PoliInfo-2 Task.................................... 101 Yasutomo Kimura, Hideyuki Shibuki, Hokuto Ototake, Yuzu Uchida, Keiichi Takamaru, Madoka Ishioroshi, Teruko Mitamura, Masaharu Yoshioka, Tomoyosi Akiba, Yasuhiro Ogawa, Minoru Sasaki, Kenichi Yokote, Tatsunori Mori, Kenji Araki, Satoshi Sekine and Noriko Kando knlab Team: NTCIR-15 QA Lab-PoliInfo Stance Classification Task.................... 113 Kazuhiro Atsumi and Yoshinobu Kano
Summarizing Utterances from Japanese Assembly Minutes using Political Sentence-BERT-based Method for QA Lab-PoliInfo-2 Task of NTCIR-15................ 116 Daiki Shirafuji, Hiromichi Kameya, Rafal Rzepka and Kenji Araki WER99 at the NTCIR-15 QA Lab-PoliInfo-2 Classification Task........................ 124 Yuichi Sasazawa and Naoaki Okazaki JRIRD at the NTCIR-15 QA Lab-PoliInfo-2 Task: An Abstractive Dialog Summarization System for Japanese Assembly Minutes.................................. 130 Kazuma Kadowaki HUHKA at the NTCIR-15 QA Lab-PoliInfo-2 Entity Linking Task...................... 137 Takuma Himori, Yasutomo Kimura and Kenji Araki wfrnt Team at the NTCIR-15 QA Lab-PoliInfo-2 Task.................................. 140 Ken-Ichi Yokote selt Team s Entity Linking System at the NTCIR-15 QA Lab-PoliInfo2................. 142 Yuji Naraki and Tetsuya Sakai NUKL at the NTCIR-15 QA Lab-PoliInfo-2 Task....................................... 145 Yasuhiro Ogawa, Yuta Ikari, Takahiro Komamizu and Katsuhiko Toyama akbl at the NTCIR-15 QA Lab-PoliInfo-2 Tasks........................................ 153 Takanori Nekomoto, Ryoto Ohsugi, Tomoyosi Akiba, Shigeru Masuyama and Daiki Shirato Forst: A Challenge to the NTCIR-15 QA Lab-PoliInfo-2 Task........................... 160 Hiromu Onogi, Kiichi Kondo, Younghun Lim, Xinnan Shen, Madoka Ishioroshi, Hideyuki Shibuki, Tatsunori Mori and Noriko Kando NTCIR-15 QA Lab-PoliInfo2 Topic Detection based on LDA and Argument Structure Graph................................................................................... 168 Yuya Hirai, Yo Amano and Kazuhiro Takeuchi Ibrk at the NTCIR-15 QA Lab-PoliInfo-2............................................... 172 Ryo Kato and Minoru Sasaki LIAT Team s Extractive Summarizer at NTCIR-15 QALab PoliInfo-2................... 175 Kouta Nakayama and Satoshi Sekine SHINRA2020-ML Overview of SHINRA2020-ML Task.................................................... 177 Satoshi Sekine, Masako Nomoto, Kouta Nakayama, Asuka Sumida and Koji Matsuda Cross-lingual Extended Named Entity Classification of Wikipedia Articles.............. 182 The Viet Bui and Phuong Le-Hong Priberam Labs at the NTCIR-15 SHINRA2020-ML: Classification Task................. 187 Rúben Cardoso, Afonso Mendes and Andre Lamurias Rehoboam at the NTCIR-15 SHINRA2020-ML Task.................................... 195 Tushar Abhishek, Ayush Agarwal, Anubhav Sharma, Vasudeva Varma and Manish Gupta
UOM-FJ at the NTCIR-15 SHINRA2020-ML Task..................................... 201 Hiyori Yoshikawa, Chunpeng Ma, Aili Shen, Qian Sun, Chenbang Huang, Guillaume Pelat, Akiva Miura, Daniel Beck, Timothy Baldwin and Tomoya Iwakura LIAT Team s Wikipedia Classifier at NTCIR-15 SHINRA2020-ML: Classification Task.. 208 Kouta Nakayama and Satoshi Sekine HUKB at SHINRA2020-ML task....................................................... 211 Masaharu Yoshioka and Yoshiaki Koitabashi Studio Ousia at the NTCIR-15 SHINRA2020-ML Task................................. 216 Sosuke Nishikawa and Ikuya Yamada WWW-3 Overview of the NTCIR-15 We Want Web with CENTRE (WWW-3) Task............. 219 Tetsuya Sakai, Sijie Tao, Zhaohao Zeng, Yukun Zheng, Jiaxin Mao, Zhumin Chu, Yiqun Liu, Maria Maistro, Zhicheng Dou, Nicola Ferro and Ian Soboroff KASYS at the NTCIR-15 WWW-3 Task............................................... 235 Kohei Shinden, Atsuki Maruta and Makoto P. Kato RUCIR at the NTCIR-15 WWW-3 Task................................................ 239 Xiaochen Zuo, Jing Yao and Zhicheng Dou SLWWW at the NTCIR-15 WWW-3 Task.............................................. 243 Masaki Muraoka, Zhaohao Zeng and Tetsuya Sakai The Technion at the WWW-3 Task: Cluster-Based Document Retrieval................. 247 Fiana Raiber and Oren Kurland THUIR at the NTCIR-15 WWW-3 Task................................................ 249 Zhumin Chu, Jingtao Zhan, Xiangsheng Li, Jiaxin Mao, Yiqun Liu, Min Zhang and Shaoping Ma ESTUCeng at the NTCIR-15: Experimenting with new document quality features...... 253 Ahmet Aydın, Gökhan Göksel, Ahmet Arslan and Bekir Taner Dinçer MPII at the NTCIR-15 WWW-3 Task.................................................. 258 Canjia Li and Andrew Yates NAUIR at the NTCIR-15 WWW-3 Task................................................ 262 Zhu Liang, Jinling Shang, Xuwen Song and Si Shen Data Search Overview of the NTCIR-15 Data Search Task........................................... 267 Makoto P. Kato, Hiroaki Ohshima, Ying-Hsang Liu and Hsin-Liang Chen KSU Systems at the NTCIR-15 Data Search Task...................................... 274 Taku Okamoto and Hisashi Miyamori STIS at the NTCIR-15 Data Search Task: Document Retrieval Re-ranking.............. 282 Lya Hulliyyatus Suadaa, Lutfi Rahmatuti Maghfiroh, Isfan Nur Fauzi and Siti Mariyah
NII Table Linker at the NTCIR-15 Data Search Task: Re-ranking with Pre-trained Contextualized Embeddings, Data Content, Entity-centric, and Cluster-based approaches 286 Phuc Nguyen, Kazutoshi Shinoda, Taku Sakamoto, Diana Petrescu, Hung-Nghiep Tran, Atsuhiro Takasu, Akiko Aizawa and Hideaki Takeda Uhai at the NTCIR-15 Data Search task................................................ 293 Ryota Mibayashi, Huulong Pham, Naoaki Matsumoto, Takehiro Yamamoto and Hiroaki Ohshima MART Overview of NTCIR-15 MART......................................................... 299 Graham Healy, Tu-Khiem Le, Hideo Joho, Frank Hopfgartner and Cathal Gurrin Behavioral Classification Using Feature Selection in the Micro Activity Retrieval Task.. 304 Takuma Yoshimura, Pham Huulong, Ryota Mibayashi, Rui Kimura and Hiroaki Ohshima TMU19 at NTCIR-15 MART Retrieval Task............................................ 309 Duy-Duc Le Nguyen, Yu-Chi Lang and Yung-Chun Chang DCU team at the NTCIR-15 Micro-Activity Retrieval Task............................. 312 Tu-Khiem Le, Manh-Duy Nguyen, Ly-Duyen Tran, Van-Tu Ninh, Cathal Gurrin and Graham Healy NLP301 at the NTCIR-15 Micro-activity Retrieval Task: Incorporating Region of Interest Features into Supervised Encoder............................................... 318 Tai-Te Chu, Yi-Ting Liu, Chia-Chung Chang, An-Zi Yen, Hen-Hsen Huang and Hsin-Hsi Chen THUIR at the NTCIR-15 Micro-activity Retrieval Task................................. 323 Jiayu Li, Ziyi Ye, Weizhi Ma, Min Zhang, Yiqun Liu and Shaoping Ma