The Reading the Mind in the Eyes Test (RMET) is widely used in clinical and non-clinical research. However, the structural properties of RMET scores have yet to be rigorously examined. We analyzed the structural properties of RMET scores in nine existing datasets comprising non-clinical samples ranging from 558 to 9,267 (median,=,1,112) participants each. We used confirmatory factor analysis to assess two theoretically derived factor models, exploratory factor analysis to identify possible alternative factor models, and reliability estimates to assess internal consistency. Neither of the theoretically derived models was a good fit for any of the nine datasets, and we were unable to identify any better fitting multidimensional models. Internal consistency metrics were acceptable in six of the nine datasets, but these metrics are difficult to interpret given the uncertain factor structures. Our findings contribute to a growing body of evidence questioning the reliability and validity of RMET scores.