The MovieQA benchmark on GitHub provides access to the list of movies, splits, and QAs as simple JSON files.
Movies are referenced using their IMDb key and each question comes with a unique identifier "qid".
- train: The 9,848 QAs (269 movies) whose qid starts with train may be used to train a model.
We encourage people to make a further split into a train/dev to prevent overfitting, monitor training loss, try different hyperparameters, etc.
- val: The 1,958 QAs (56 movies) whose qid starts with val can be used to report and compare results for several model configurations.
The val set should not be used for training.
- test: The 3,138 QAs (83 movies) whose qid starts with test are the held out test set.
Test set evaluation is performed on the server, and results from the leaderboard should be reported in papers.