B-P.38: autoeval: A robust toolkit for standardized ad-hoc protein language model evaluation
Lisa M. Spindler
Technical University of Munich
Aleena Siji
Helmholtz Munich
Egor Rakcheev
Technical University of Munich
Michael Heinzinger
Helmholtz Munich
Burkhard Rost
Technical University of Munich
Keywords
protein language model, biotrainer, autoeval, protein function, protein structure
[sponser-meet-now-chat][/sponser-meet-now-chat]
Protein Language Models (pLMs) became an essential cornerstone when working with proteins over the last years. For the development of new pLMs, however, it is essential to put the performance of the new model into perspective of existing ones. To complement existing evaluation frameworks which tend to focus on variant effect prediction (VEP), we are presenting autoeval, a robust toolkit for standardized ad-hoc pLM evaluation. Autoeval checks the representation quality of pLM by performing linear probing on several curated, established supervised prediction tasks. We provide validation splits to allow direct integration of autoeval into model development workflows without risking information leakage. Once hyperparameters are set and pLM pre-training is finished, autoeval can be switched into evaluation mode to assess the final model quality on established test sets, putting the new model's performance directly into perspective of existing ones. For simplifying comparison to existing work, we also support automated zero-shot evaluation on de-facto standard VEP benchmarking with ProteinGym as well as contact map prediction. To simplify usability, autoeval can be used via a simple Python API that directly integrates with a comprehensive web platform to analyze the results. The toolkit is available publicly at https://autoeval.biocentral.cloud.
Co-authors: Lisa M. Spindler, Michael Heinzinger, Burkhard Rost, Aleena Siji, Egor Rakcheev
Contact Attendee
Warning: Attempt to read property "user_email" on string in /home/1276969.cloudwaysapps.com/ydbgzhdjeq/public_html/wp-content/plugins/my-conference-now/functions.php on line 2826