B-ELIXIR.02: BioTerm-Bench: A Harbor-Compatible Benchmark for Tracking Open-Model Progress on Terminal Bioinformatics Tasks
Large language model agents are increasingly proposed as assistants for routine bioinformatics, yet existing benchmarks either score notebook reasoning (BixBench), need cloud infrastructure (BioAgent Bench), or depend on drifting LLM judges. None lets a small group track open-model progress reproducibly under limited resources.
We present BioTerm-Bench V1, a Harbor-compatible benchmark of 25 deterministically-graded terminal tasks spanning file-format manipulation, region operations, variant calling and filtering, RNA-seq normalisation, sequence search, and quality control. The benchmark is specifically designed to evaluate AI agents acting as orchestrators within federated data infrastructure — agents that assist with analysis directly on the node where sensitive genomic data resides, without data ever leaving the institution. This federated deployment requirement drives three core design choices: tasks target smaller, locally-runnable open models; evaluation is fully deterministic, enabling reliable automated quality checks within each node; and all reference data and graders are containerised for reproducible local execution.
We evaluated ten models, three times each, ranked by strict all-three-pass rule: GLM-5, Kimi-K2.5 24/25; DeepSeek-V3.2, Gemma-4-26b-A4B 23/25; GPT-5.4-mini, Qwen3.5-35b-a3b 18/25; Mistral-Small-2603 17/25; GPT-OSS-120B 13/25; GPT-5.4-nano 12/25; GPT-OSS-20B 6/25. Main discriminators were database search, multi-step ORF selection, and trio-aware variant logic. A full sweep runs ~1h per model. Harder successor packs are planned as V1 saturates, mirroring ARC-AGI. BioTerm-Bench V1 is released under Apache 2.0 at https://github.com/ML-Bioinfo-CEITEC/BioTerm-Bench.
Co-authors: Petr Simecek
Contact Attendee
Warning: Attempt to read property "user_email" on string in /home/1276969.cloudwaysapps.com/ydbgzhdjeq/public_html/wp-content/plugins/my-conference-now/functions.php on line 2826