Is What They Learn Worth What They Forget