Do Large Language Models Know Colombian Law? A Reliability Benchmark for the Colombian Legal System

Failed to add items

Sorry, we are unable to add the item because your shopping basket is already at capacity.

Add to cart failed.

Please try again later

Add to wishlist failed.

Please try again later

Remove from wishlist failed.

Please try again later

Follow podcast failed

Unfollow podcast failed

Do Large Language Models Know Colombian Law? A Reliability Benchmark for the Colombian Legal System

Listen for free

View show details

LLMs are increasingly used in legal work, yet their reliability outside the United States is poorly documented. This expert-validated benchmark has 1,042 items across ten areas of law and three formats, and was used to test 15 models. Closed-question accuracy reaches 0.905, but free-text correctness never exceeds 0.45, and only about half of cited norms are correct. Models often sound responsive while being wrong, which is risky for non-experts. Authors: Rubén Manrique, Michelle Castellanos, Jorge Morales, Juan David Gutiérrez, Antonio Barreto Rozo, Joaquín Vélez Navarro Paper: https://arxiv.org/abs/2610.03639v1

No reviews yet