Back office ops · Production

Beyond the Hype: Real-World Lessons and Insights from Working with Large Language Models at Mercado Libre

The problem

Mercado Libre lacked a centralized system to answer developer questions about internal tooling, half of their 4,000 productive data tables had no adequate documentation, and their internal expert-booking platform required structured manual input that natural language queries could simplify.

First attempt

The initial RAG prototype built with Llama Index hallucinated responses when documentation gaps existed, and the documentation generation produced output that 10% of stakeholders did not accept, citing missing structure and internal acronyms.

Workflow diagram · grounded in source
1
Developer queries RAG system
trigger
“a Question-Answering system that generates personalized answers by retrieving and combining information from relevant documents”
2
RAG retrieves and generates answer
ai_action
“It effectively navigates through an index of knowledge based on a user's query. By retrieving relevant context or information and then generating a response, RAG enables the LLM to provide accurate and pertinent answers to user inquiries…”
3
Answer delivered with source links
output
“instantly receive answers, complete with links to source materials for further exploration”
4
Documentation gaps identified and fixed
feedback_loop
“This process revealed shortcomings in our documentation — certain actions or tools that users inquired about were not covered. So, what was the solution? Enhancing our documentation.”
5
LLM generates table documentation
ai_action
“Using a generic prompt such as "You're an expert documenter, please create documentation for table {TABLE_NAME} based on the following elements," we were able to generate documentation that was well-received by 90% of our stakeholders”
6
Stakeholders review generated docs
human_review
“Table owners agreed with our suggested documentation and made only minor adjustments”
7
User submits natural language booking query
trigger
“when a user queried "I want to consult an expert in Tableau who is available next thursday", we needed to understand first what topics the user was interested in and then, what agenda was being requested”
8
Function calling extracts structured data
ai_action
“By leveraging this functionality, you can interpret raw text and understand the underlying meaning behind words or numbers, which are retrieved in a more structured manner”
Reported outcome

LLM-generated table documentation was well-received by 90% of stakeholders with only minor adjustments needed, documentation gaps were iteratively identified and addressed to improve RAG accuracy, and function calling enabled structured extraction from natural language booking queries.

Reported metrics
Stakeholder acceptance of generated documentation90%
Stakeholders who did not accept generated documentation10%
Productive tables lacking adequate documentationhalf
Total productive tables requiring documentation4,000
Show all 5 reported metrics
stakeholder acceptance of generated documentation90%
stakeholders who did not accept generated documentation10%
productive tables lacking adequate documentationhalf
total productive tables requiring documentation4,000
adjustments needed to generated documentationminor adjustments
Reported stack
Function Calling
Source
https://medium.com/mercadolibre-tech/beyond-the-hype-real-world-lessons-and-insights-from-working-with-large-language-models-6d637e39f8f8
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

LLM-generated table documentation was well-received by 90% of stakeholders with only minor adjustments needed, documentation gaps were iteratively identified and addressed to improve RAG accuracy, and function calling…

What tools did this team use?

Function Calling.

What results were reported?

Stakeholder acceptance of generated documentation: 90%; Stakeholders who did not accept generated documentation: 10%; Productive tables lacking adequate documentation: half; Total productive tables requiring documentation: 4,000 (source-reported, not independently verified).

What failed first in this deployment?

The initial RAG prototype built with Llama Index hallucinated responses when documentation gaps existed, and the documentation generation produced output that 10% of stakeholders did not accept, citing missing structu…

How is this back office ops AI workflow structured?

Developer queries RAG system → RAG retrieves and generates answer → Answer delivered with source links → Documentation gaps identified and fixed → LLM generates table documentation → Stakeholders review generated docs → User submits natural language booking query → Function calling extracts structured data.

Related back office ops cases