Leader Board

The leader board presents results on the test set of the MovieQA benchmark. The tables here are sorted based on the used answering source. Please follow the buttons for quick access to different sources.

Movie: Video+Subtitles

Date Team Name Affiliation Accuracy Notes Paper Code
08 Aug 2017, 07:32 Anonymous Anonymous 39.03 Hierarchical Memory Networks - -
16 Apr 2017, 08:26 Anonymous Anonymous 37.20 Representing Movie Content Hierarchically - -
27 Jul 2017, 10:16 Anonymous Anonymous 37.04 Sequential Video VLAD - -
19 Feb 2017, 23:37 SNUVL & SKTVT Seoul National University Vision & Learning Lab + SK Telecom Video Tech. Lab 36.25 - -
17 Mar 2017, 16:07 Anonymous Anonymous 35.69 Read-Write Memory Network for Story Understanding - -
27 Mar 2017, 15:32 Anonymous Anonymous 35.21 Multi Layer Read Write Memory Network - -
24 Mar 2017, 15:30 Anonymous Anonymous 34.82 sharpening Read-Write-Memory-Network - -
02 Mar 2017, 13:06 Anonymous Anonymous 34.42 Read Write Memory Network for story understanding - -
01 Jun 2017, 02:57 Anonymous Anonymous 34.34 Normalized CNN for Encoding Movies - -
08 Apr 2017, 10:25 Anonymous Anonymous 34.34 Understanding visual regions for Movie QA - -
13 Mar 2017, 13:23 Anonymous Anonymous 33.86 Read Write Memory Network for story understanding - -
10 Feb 2017, 03:36 BI_kmkim Biointelligence Lab, Seoul National University 29.97 - -
16 Feb 2017, 23:20 BI_kmkim Biointelligence Lab, Seoul National University 29.41 - -
06 Feb 2017, 20:31 BI_kmkim Biointelligence Lab, Seoul National University 28.54 - -
27 Mar 2017, 00:29 Anonymous Anonymous 26.47 video based baselines, ranking loss - -
17 Mar 2017, 01:44 Anonymous Anonymous 24.32 Visual Model + LSTM + discriminative CNN - -
12 Mar 2017, 22:50 Anonymous Anonymous 24.09 Video clip feature with simple MLP - -
21 Oct 2016, 10:58 Anonymous Anonymous 23.61 Only_Video_Question_Answers_Pairs - -
16 Mar 2017, 19:19 Anonymous Anonymous 23.45 Video clip feature with LSTM + CNN - -

Subtitles only

Date Team Name Affiliation Accuracy Notes Paper Code
24 Mar 2017, 15:30 Anonymous Anonymous 38.53 sharpening Read-Write-Memory-Network - -
08 Apr 2017, 10:25 Anonymous Anonymous 37.38 Understanding visual regions for Movie QA - -
24 Oct 2016, 13:22 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 36.87
17 Mar 2017, 16:07 Anonymous Anonymous 36.55 Read-Write Memory Network for Story Understanding - -
13 Mar 2017, 13:23 Anonymous Anonymous 34.93 Read Write Memory Network for story understanding - -
28 Aug 2016, 22:52 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 26.45
25 Aug 2016, 00:59 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 23.65

DVS only

Date Team Name Affiliation Accuracy Notes Paper Code
24 Oct 2016, 13:22 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 35.09
24 Mar 2017, 15:30 Anonymous Anonymous 34.21 sharpening Read-Write-Memory-Network - -
27 Mar 2017, 15:32 Anonymous Anonymous 34.21 Multi Layer Read Write Memory Network - -
10 Jun 2017, 04:23 Anonymous Anonymous 33.16 Normalized CNN for Encoding Movies - -
17 Mar 2017, 16:07 Anonymous Anonymous 32.81 Read-Write Memory Network for Story Understanding - -
01 Jun 2017, 02:57 Anonymous Anonymous 31.58 Normalized CNN for Encoding Movies - -
13 Mar 2017, 13:23 Anonymous Anonymous 30.18 Read Write Memory Network for story understanding - -
14 Mar 2017, 04:14 Anonymous Anonymous 28.95 Read Write Memory Network for story understanding - -
25 Aug 2016, 00:59 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 24.91
28 Aug 2016, 22:52 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 23.33

Scripts only

Date Team Name Affiliation Accuracy Notes Paper Code
24 Mar 2017, 15:30 Anonymous Anonymous 39.36 sharpening Read-Write-Memory-Network - -
24 Oct 2016, 13:22 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 37.05
13 Mar 2017, 13:23 Anonymous Anonymous 35.73 Read Write Memory Network for story understanding - -
17 Mar 2017, 16:07 Anonymous Anonymous 35.54 Read-Write Memory Network for Story Understanding - -
14 Mar 2017, 04:14 Anonymous Anonymous 31.91 Read Write Memory Network for story understanding - -
25 Aug 2016, 00:59 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 24.41
28 Aug 2016, 22:52 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 23.90

Plot Synopses only

Date Team Name Affiliation Accuracy Notes Paper Code
06 Aug 2017, 05:01 JustTest National Taiwan University 78.52 - -
26 Jan 2017, 09:07 UCLMR University College London, Machine Reading Group 77.63 - -
23 Jan 2017, 04:32 UCLMR University College London, Machine Reading Group 76.64 - -
11 Jun 2017, 04:15 JustTest National Taiwan University 75.84 - -
17 Nov 2016, 10:59 UCLMR University College London, Machine Reading Group 75.78 - -
02 Jun 2017, 19:42 JustTest National Taiwan University 75.65 - -
18 Sep 2016, 01:45 Sure Singapore Management University 72.91
02 Jan 2017, 10:58 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 57.97
12 Nov 2016, 00:41 Anonymous Anonymous 47.45 TFIDF replicate of Makarand's work Anonymous Anonymous
28 Aug 2016, 22:52 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 47.36
10 Jun 2017, 04:23 Anonymous Anonymous 46.78 Normalized CNN for Encoding Movies - -
25 Aug 2016, 00:59 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 45.63
24 Oct 2016, 13:22 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 38.43
17 Mar 2017, 16:07 Anonymous Anonymous 34.80 Read-Write Memory Network for Story Understanding - -
30 Oct 2016, 01:28 Anonymous Anonymous 20.65 Using plain old simple tf - idf - -

Open ended

Date Team Name Affiliation Accuracy Notes Paper Code
17 Mar 2017, 16:07 Anonymous Anonymous 36.55 Read-Write Memory Network for Story Understanding - -
20 Aug 2016, 00:15 KIT-UofT-MIT Karlsruhe Inst. of Tech. + Univ. of Toronto + Massachusetts Inst. of Tech. 25.56