For many years, sleep-wake scoring algorithms were created and validated with small and private datasets with no more than one hundred participants. In this paper, we devised the largest dataset up to date for the sleep-wake classification problem and analyzed the performance of popular traditional algorithms as well as state-of-the-art machine learning techniques to tackle this problem. By making this dataset public, researchers can use our data and results as a benchmark to develop newer algorithms.