Leader Board

The leader board presents results on the test set of the MovieQA benchmark. The tables here are sorted based on the used answering source. Please follow the buttons for quick access to different sources.

Movie: Video+Subtitles

Date Team Name Affiliation Accuracy Notes Paper Code
08 Nov 2018, 22:15 Anonymous Anonymous 46.98 Universal QA model which works with any story - -
20 Jan 2019, 23:26 Anonymous Anonymous 45.31 new method to optimize all MEM network - -
12 Feb 2019, 08:37 Anonymous Anonymous 44.52 Video clip feature and Text feature fusion - -
23 Jan 2019, 09:02 Anonymous Anonymous 44.12 come up with a new method for QA - -
27 Jan 2019, 21:42 Anonymous Anonymous 43.56 A new method to optimize MemNetwork - -
25 Jan 2019, 01:25 Anonymous Anonymous 43.24 new method to optimize network(different paramter) - -
15 Aug 2018, 22:32 Anonymous Anonymous 43.08 Multi-modal End-to-end Memory Network_Version 2 - -
20 Jan 2019, 10:13 Anonymous Anonymous 42.77 memory network end to Peter memory network end to - -
11 Nov 2018, 08:37 Anonymous Anonymous 42.53 Progressive Attention Memory Network (PAMN) - -
10 Mar 2019, 04:17 Anonymous Anonymous 42.45 use a different word embdding - -
28 Jun 2018, 21:32 NTHU_VLLab Visual Learning Lab, National Tsing Hua University 41.97 -
25 May 2018, 03:02 Anonymous Anonymous 41.65 Multimodal End-to-end Memory Network - -
01 Jun 2018, 13:12 Anonymous Anonymous 41.65 A2A: Attention to Attention - -
17 Jun 2018, 03:27 Anonymous Anonymous 41.57 A2A: Attention to Attention (multimodal val best) - -
31 Jan 2018, 07:00 BI_kmkim Biointelligence Lab, Seoul National University 41.41 -
24 Aug 2018, 08:26 Anonymous Anonymous 40.06 Question guided Attention network_v2 - -
21 Aug 2018, 06:43 Anonymous Anonymous 39.67 Question guided Attention network - -
16 Aug 2018, 11:27 Anonymous Anonymous 39.27 Attention memory network based on MovieQA paper - -
08 Aug 2017, 07:32 TJU_MM School of Computer Science and Technology, Tianjin University 39.03
19 Jul 2018, 22:13 Anonymous Anonymous 38.71 Attention memory network based on MovieQA paper - -
15 Nov 2017, 22:50 umich-vl-lit University of Michigan - Ann Arbor 38.16 -
13 Oct 2017, 23:58 SNUVL_&_SKTVT Seoul National University Vision & Learning Lab + SK Telecom Video Tech. Lab 38.16 - -
13 Nov 2017, 23:49 CMUSCS CMUSCS 37.28
16 Apr 2017, 08:26 Anonymous Anonymous 37.20 Representing Movie Content Hierarchically - -
07 Oct 2017, 23:51 Anonymous Anonymous 37.12 Local Average Pooling Networks - -
27 Jul 2017, 10:16 Anonymous Anonymous 37.04 Sequential Video VLAD - -
19 Feb 2017, 23:37 SNUVL_&_SKTVT Seoul National University Vision & Learning Lab + SK Telecom Video Tech. Lab 36.25 -
24 Jul 2018, 01:26 Anonymous Anonymous 36.25 Attention end-to-end memory network - -
16 Jul 2018, 08:57 Anonymous Anonymous 36.01 Attention memory network based on MovieQA paper - -
10 Oct 2017, 23:58 Anonymous Anonymous 35.93 Local Average Pooling Networks - -
17 Mar 2017, 16:07 Anonymous Anonymous 35.69 Read-Write Memory Network for Story Understanding - -
27 Mar 2017, 15:32 Anonymous Anonymous 35.21 Multi Layer Read Write Memory Network - -
10 Nov 2017, 23:47 Anonymous Anonymous 35.14 Focal Visual-Text Attention - -
01 Feb 2019, 09:31 Anonymous Anonymous 35.06 Video clip feature and Text feature fusion - -
24 Mar 2017, 15:30 Anonymous Anonymous 34.82 sharpening Read-Write-Memory-Network - -
14 Sep 2017, 12:08 BI_kmkim Biointelligence Lab, Seoul National University 34.74 - -
02 Mar 2017, 13:06 Anonymous Anonymous 34.42 Read Write Memory Network for story understanding - -
01 Jun 2017, 02:57 Anonymous Anonymous 34.34 Normalized CNN for Encoding Movies - -
08 Apr 2017, 10:25 Anonymous Anonymous 34.34 Understanding visual regions for Movie QA - -
13 Mar 2017, 13:23 Anonymous Anonymous 33.86 Read Write Memory Network for story understanding - -
11 Sep 2017, 12:06 Anonymous Anonymous 33.78 Multimodal Sequence Memory for video story QA - -
06 Nov 2017, 19:05 Anonymous Anonymous 31.24 Baseline, emb+lstm, video+movie clips, - -
10 Feb 2017, 03:36 BI_kmkim Biointelligence Lab, Seoul National University 29.97 -
16 Feb 2017, 23:20 BI_kmkim Biointelligence Lab, Seoul National University 29.41 - -
11 May 2018, 12:01 Anonymous Anonymous 29.25 Multimodal Attention Network with convolution - -
06 Feb 2017, 20:31 BI_kmkim Biointelligence Lab, Seoul National University 28.54 - -
27 Mar 2017, 00:29 Anonymous Anonymous 26.47 video based baselines, ranking loss - -
17 Mar 2017, 01:44 Anonymous Anonymous 24.32 Visual Model + LSTM + discriminative CNN - -
12 Mar 2017, 22:50 Anonymous Anonymous 24.09 Video clip feature with simple MLP - -
21 Oct 2016, 10:58 Anonymous Anonymous 23.61 Only_Video_Question_Answers_Pairs - -
20 Jan 2019, 09:08 Anonymous Anonymous 23.53 memory network optimizer123456 - -
16 Mar 2017, 19:19 Anonymous Anonymous 23.45 Video clip feature with LSTM + CNN - -
20 Jan 2019, 08:10 Anonymous Anonymous 23.29 A new method to optimize all memory network - -
30 Apr 2018, 23:12 Anonymous Anonymous 21.70 Multimodal Attention Network with convolution - -
22 May 2018, 03:01 Anonymous Anonymous 19.95 Multimodal End-to-end Memory Network - -

Subtitles only

Date Team Name Affiliation Accuracy Notes Paper Code
08 Nov 2018, 22:15 Anonymous Anonymous 44.01 Universal QA model which works with any story - -
15 Nov 2017, 18:15 umich-vl-lit University of Michigan - Ann Arbor 39.36 -
24 Mar 2017, 15:30 SNUVL_&_SKTVT Seoul National University Vision & Learning Lab + SK Telecom Video Tech. Lab 38.53 -
21 Sep 2017, 19:18 Anonymous Anonymous 38.34 Soft Entity-based CNN-MemN2N - -
18 Sep 2017, 18:46 Anonymous Anonymous 37.99 Entity-based convolutional memory network - -
15 Sep 2017, 18:45 Anonymous Anonymous 37.92 Entity-based convolutional memory network - -
08 Apr 2017, 10:25 Anonymous Anonymous 37.38 Understanding visual regions for Movie QA - -
24 Oct 2016, 13:22 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 36.87
17 Mar 2017, 16:07 Anonymous Anonymous 36.55 Read-Write Memory Network for Story Understanding - -
13 Mar 2017, 13:23 Anonymous Anonymous 34.93 Read Write Memory Network for story understanding - -
28 Aug 2016, 22:52 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 26.45
25 Aug 2016, 00:59 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 23.65

DVS only

Date Team Name Affiliation Accuracy Notes Paper Code
08 Nov 2018, 22:15 Anonymous Anonymous 49.65 Universal QA model which works with any story - -
24 Oct 2016, 13:22 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 35.09
24 Mar 2017, 15:30 SNUVL_&_SKTVT Seoul National University Vision & Learning Lab + SK Telecom Video Tech. Lab 34.21 -
27 Mar 2017, 15:32 Anonymous Anonymous 34.21 Multi Layer Read Write Memory Network - -
10 Jun 2017, 04:23 Anonymous Anonymous 33.16 Normalized CNN for Encoding Movies - -
17 Mar 2017, 16:07 Anonymous Anonymous 32.81 Read-Write Memory Network for Story Understanding - -
01 Jun 2017, 02:57 Anonymous Anonymous 31.58 Normalized CNN for Encoding Movies - -
13 Mar 2017, 13:23 Anonymous Anonymous 30.18 Read Write Memory Network for story understanding - -
14 Mar 2017, 04:14 Anonymous Anonymous 28.95 Read Write Memory Network for story understanding - -
25 Aug 2016, 00:59 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 24.91
28 Aug 2016, 22:52 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 23.33

Scripts only

Date Team Name Affiliation Accuracy Notes Paper Code
08 Nov 2018, 22:15 Anonymous Anonymous 45.49 Universal QA model which works with any story - -
24 Mar 2017, 15:30 SNUVL_&_SKTVT Seoul National University Vision & Learning Lab + SK Telecom Video Tech. Lab 39.36 -
02 Dec 2017, 15:34 umich-vl-lit University of Michigan - Ann Arbor 37.86 -
24 Oct 2016, 13:22 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 37.05
13 Mar 2017, 13:23 Anonymous Anonymous 35.73 Read Write Memory Network for story understanding - -
17 Mar 2017, 16:07 Anonymous Anonymous 35.54 Read-Write Memory Network for Story Understanding - -
14 Mar 2017, 04:14 Anonymous Anonymous 31.91 Read Write Memory Network for story understanding - -
25 Aug 2016, 00:59 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 24.41
02 Oct 2017, 11:45 Anonymous Anonymous 24.16 Logistic regression over string similarities Anonymous -
28 Aug 2016, 22:52 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 23.90

Plot Synopses only

Date Team Name Affiliation Accuracy Notes Paper Code
04 Aug 2018, 13:30 IMS Institute for Natural Language Processing, University of Stuttgart 85.12
29 May 2018, 12:08 IMS Institute for Natural Language Processing, University of Stuttgart 84.70
14 Mar 2018, 09:28 IMS Institute for Natural Language Processing, University of Stuttgart 82.73
08 Mar 2018, 22:18 NTU_ Nanyang Technological University 80.72 - -
13 Oct 2017, 12:00 ADAPT ADAPT centre, Dublin City University 80.02 -
14 Sep 2017, 15:53 JustTest National Taiwan University 79.99
24 Aug 2017, 06:33 ADAPT ADAPT centre, Dublin City University 79.76 -
02 Oct 2017, 05:05 Anonymous Anonymous 79.48 Logistic regression (for script) - -
06 Aug 2017, 05:01 JustTest National Taiwan University 78.52 - -
26 Jan 2017, 09:07 UCLMR University College London, Machine Reading Group 77.63 - -
23 Jan 2017, 04:32 UCLMR University College London, Machine Reading Group 76.64 - -
30 Aug 2017, 09:50 ADAPT ADAPT centre, Dublin City University 76.04 -
11 Jun 2017, 04:15 JustTest National Taiwan University 75.84 - -
17 Nov 2016, 10:59 UCLMR University College London, Machine Reading Group 75.78 - -
02 Jun 2017, 19:42 JustTest National Taiwan University 75.65 - -
18 Sep 2016, 01:45 Sure Singapore Management University 72.91
02 Jan 2017, 10:58 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 57.97
12 Nov 2016, 00:41 UofT-Eng University of Toronto - ECE 47.45
28 Aug 2016, 22:52 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 47.36
10 Jun 2017, 04:23 Anonymous Anonymous 46.78 Normalized CNN for Encoding Movies - -
25 Aug 2016, 00:59 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 45.63
24 Oct 2016, 13:22 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 38.43
17 Mar 2017, 16:07 Anonymous Anonymous 34.80 Read-Write Memory Network for Story Understanding - -
30 Oct 2016, 01:28 Anonymous Anonymous 20.65 Using plain old simple tf - idf - -

Open ended

Date Team Name Affiliation Accuracy Notes Paper Code
17 Mar 2017, 16:07 Anonymous Anonymous 36.55 Read-Write Memory Network for Story Understanding (eval all QAs) - -
20 Aug 2016, 00:15 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 25.56