Leader Board

The leader board presents results on the test set of the MovieQA benchmark. The tables here are sorted based on the used answering source. Please follow the buttons for quick access to different sources.

Movie: Video+Subtitles

Date Team Name Affiliation Accuracy Notes Paper Code
08 Aug 2017, 07:32 TJU_MM School of Computer Science and Technology, Tianjin University 39.03 -
15 Nov 2017, 22:50 Anonymous Anonymous 38.16 Entity-based Convolutional Memory Network - -
13 Oct 2017, 23:58 SNUVL_&_SKTVT Seoul National University Vision & Learning Lab + SK Telecom Video Tech. Lab 38.16 - -
13 Nov 2017, 23:49 Anonymous Anonymous 37.28 Focal Visual-Text Attention - -
16 Apr 2017, 08:26 Anonymous Anonymous 37.20 Representing Movie Content Hierarchically - -
07 Oct 2017, 23:51 Anonymous Anonymous 37.12 Local Average Pooling Networks - -
27 Jul 2017, 10:16 Anonymous Anonymous 37.04 Sequential Video VLAD - -
19 Feb 2017, 23:37 SNUVL_&_SKTVT Seoul National University Vision & Learning Lab + SK Telecom Video Tech. Lab 36.25 -
10 Oct 2017, 23:58 Anonymous Anonymous 35.93 Local Average Pooling Networks - -
17 Mar 2017, 16:07 Anonymous Anonymous 35.69 Read-Write Memory Network for Story Understanding - -
27 Mar 2017, 15:32 Anonymous Anonymous 35.21 Multi Layer Read Write Memory Network - -
10 Nov 2017, 23:47 Anonymous Anonymous 35.14 Focal Visual-Text Attention - -
24 Mar 2017, 15:30 Anonymous Anonymous 34.82 sharpening Read-Write-Memory-Network - -
14 Sep 2017, 12:08 BI_kmkim Biointelligence Lab, Seoul National University 34.74 - -
02 Mar 2017, 13:06 Anonymous Anonymous 34.42 Read Write Memory Network for story understanding - -
01 Jun 2017, 02:57 Anonymous Anonymous 34.34 Normalized CNN for Encoding Movies - -
08 Apr 2017, 10:25 Anonymous Anonymous 34.34 Understanding visual regions for Movie QA - -
13 Mar 2017, 13:23 Anonymous Anonymous 33.86 Read Write Memory Network for story understanding - -
11 Sep 2017, 12:06 Anonymous Anonymous 33.78 Multimodal Sequence Memory for video story QA - -
06 Nov 2017, 19:05 Anonymous Anonymous 31.24 Baseline, emb+lstm, video+movie clips, - -
10 Feb 2017, 03:36 BI_kmkim Biointelligence Lab, Seoul National University 29.97 -
16 Feb 2017, 23:20 BI_kmkim Biointelligence Lab, Seoul National University 29.41 - -
06 Feb 2017, 20:31 BI_kmkim Biointelligence Lab, Seoul National University 28.54 - -
27 Mar 2017, 00:29 Anonymous Anonymous 26.47 video based baselines, ranking loss - -
17 Mar 2017, 01:44 Anonymous Anonymous 24.32 Visual Model + LSTM + discriminative CNN - -
12 Mar 2017, 22:50 Anonymous Anonymous 24.09 Video clip feature with simple MLP - -
21 Oct 2016, 10:58 Anonymous Anonymous 23.61 Only_Video_Question_Answers_Pairs - -
16 Mar 2017, 19:19 Anonymous Anonymous 23.45 Video clip feature with LSTM + CNN - -

Subtitles only

Date Team Name Affiliation Accuracy Notes Paper Code
15 Nov 2017, 18:15 Anonymous Anonymous 39.36 Entity-based Convolutional Memory Network - -
24 Mar 2017, 15:30 SNUVL_&_SKTVT Seoul National University Vision & Learning Lab + SK Telecom Video Tech. Lab 38.53 -
21 Sep 2017, 19:18 Anonymous Anonymous 38.34 Soft Entity-based CNN-MemN2N - -
18 Sep 2017, 18:46 Anonymous Anonymous 37.99 Entity-based convolutional memory network - -
15 Sep 2017, 18:45 Anonymous Anonymous 37.92 Entity-based convolutional memory network - -
08 Apr 2017, 10:25 Anonymous Anonymous 37.38 Understanding visual regions for Movie QA - -
24 Oct 2016, 13:22 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 36.87
17 Mar 2017, 16:07 Anonymous Anonymous 36.55 Read-Write Memory Network for Story Understanding - -
13 Mar 2017, 13:23 Anonymous Anonymous 34.93 Read Write Memory Network for story understanding - -
28 Aug 2016, 22:52 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 26.45
25 Aug 2016, 00:59 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 23.65

DVS only

Date Team Name Affiliation Accuracy Notes Paper Code
24 Oct 2016, 13:22 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 35.09
24 Mar 2017, 15:30 SNUVL_&_SKTVT Seoul National University Vision & Learning Lab + SK Telecom Video Tech. Lab 34.21 -
27 Mar 2017, 15:32 Anonymous Anonymous 34.21 Multi Layer Read Write Memory Network - -
10 Jun 2017, 04:23 Anonymous Anonymous 33.16 Normalized CNN for Encoding Movies - -
17 Mar 2017, 16:07 Anonymous Anonymous 32.81 Read-Write Memory Network for Story Understanding - -
01 Jun 2017, 02:57 Anonymous Anonymous 31.58 Normalized CNN for Encoding Movies - -
13 Mar 2017, 13:23 Anonymous Anonymous 30.18 Read Write Memory Network for story understanding - -
14 Mar 2017, 04:14 Anonymous Anonymous 28.95 Read Write Memory Network for story understanding - -
25 Aug 2016, 00:59 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 24.91
28 Aug 2016, 22:52 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 23.33

Scripts only

Date Team Name Affiliation Accuracy Notes Paper Code
24 Mar 2017, 15:30 SNUVL_&_SKTVT Seoul National University Vision & Learning Lab + SK Telecom Video Tech. Lab 39.36 -
02 Dec 2017, 15:34 Anonymous Anonymous 37.86 Entity-based Convolutional Memory Network - -
24 Oct 2016, 13:22 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 37.05
13 Mar 2017, 13:23 Anonymous Anonymous 35.73 Read Write Memory Network for story understanding - -
17 Mar 2017, 16:07 Anonymous Anonymous 35.54 Read-Write Memory Network for Story Understanding - -
14 Mar 2017, 04:14 Anonymous Anonymous 31.91 Read Write Memory Network for story understanding - -
25 Aug 2016, 00:59 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 24.41
02 Oct 2017, 11:45 Anonymous Anonymous 24.16 Logistic regression over string similarities Anonymous -
28 Aug 2016, 22:52 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 23.90

Plot Synopses only

Date Team Name Affiliation Accuracy Notes Paper Code
13 Oct 2017, 12:00 ADAPT ADAPT centre, Dublin City University 80.02 -
14 Sep 2017, 15:53 JustTest National Taiwan University 79.99 - -
24 Aug 2017, 06:33 ADAPT ADAPT centre, Dublin City University 79.76 -
02 Oct 2017, 05:05 Anonymous Anonymous 79.48 Logistic regression (for script) - -
06 Aug 2017, 05:01 JustTest National Taiwan University 78.52 - -
26 Jan 2017, 09:07 UCLMR University College London, Machine Reading Group 77.63 - -
23 Jan 2017, 04:32 UCLMR University College London, Machine Reading Group 76.64 - -
30 Aug 2017, 09:50 ADAPT ADAPT centre, Dublin City University 76.04 -
11 Jun 2017, 04:15 JustTest National Taiwan University 75.84 - -
17 Nov 2016, 10:59 UCLMR University College London, Machine Reading Group 75.78 - -
02 Jun 2017, 19:42 JustTest National Taiwan University 75.65 - -
18 Sep 2016, 01:45 Sure Singapore Management University 72.91
02 Jan 2017, 10:58 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 57.97
12 Nov 2016, 00:41 UofT-Eng University of Toronto - ECE 47.45
28 Aug 2016, 22:52 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 47.36
10 Jun 2017, 04:23 Anonymous Anonymous 46.78 Normalized CNN for Encoding Movies - -
25 Aug 2016, 00:59 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 45.63
24 Oct 2016, 13:22 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 38.43
17 Mar 2017, 16:07 Anonymous Anonymous 34.80 Read-Write Memory Network for Story Understanding - -
30 Oct 2016, 01:28 Anonymous Anonymous 20.65 Using plain old simple tf - idf - -

Open ended

Date Team Name Affiliation Accuracy Notes Paper Code
17 Mar 2017, 16:07 Anonymous Anonymous 36.55 Read-Write Memory Network for Story Understanding - -
20 Aug 2016, 00:15 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 25.56