Skip to content
Natalia Nario

Natalia Nario

AI agent evaluation · Process transformation · Training · Banking and finance · Uruguay

natalianario.com · contact@natalianario.com · linkedin.com/in/natalianario

I solve complex problems by connecting business, risk, and technology.

Portrait of Natalia Nario

What drives me is seeing processes improve, people grow, and our work get better every day. Today I work in three areas:

  • AI agent evaluation. Strategy, criteria, and evidence to decide whether an agent is ready. Strategy, criteria, and evidence to decide whether an agent is ready for use, evaluating it end to end.
  • Process transformation. Faster, more effective processes in banking, finance, and audit. I turn manual, slow, and costly processes into fast, effective ones that work without depending on me, in banking, finance, fintech, and audit.
  • Training. Courses and workshops on AI evaluation and responsible AI. Courses and workshops on AI evaluation and responsible AI, for teams and organizations.

Public Accountant (Uruguay). Over 25 years of experience, 14 in the financial system, on both the regulator's and the bank's side. Independent since November 2026.

My career

Experience

In every organization, I went beyond the role I was hired for and took on problems that had no solution yet.

  • Public Accountant (Uruguay)
  • Over 25 years of experience, 14 in the financial system, on both the regulator's and the bank's side
  • Quality since 2014 and AI since 2023: QA Leader and creator of ArtificialQA at QAlified
  • Independent since November 2026 · Uruguay
  1. Since November 2026

    Independent consultant

    Today I'm applying my guiding question to my own career: how can we do this better? I work in three areas: AI agent evaluation; process transformation in banking, finance, fintech, and audit; and training in AI evaluation and responsible AI.

    What I do as an independent consultant
    • What I offer. AI agent evaluations, from strategy and criteria to the evidence needed to decide whether an agent is ready; process transformation projects; and courses and workshops. In every case, I think, research, innovate, propose, and follow through as a team, with change management.
    • How I evaluate agents. When I evaluate an agent, I look beyond its answers. I also look at how it is implemented, its steps, its data, the people involved, and how it is monitored once in production.
    • How I use AI. First I specify what we want to achieve, then I build, and finally I validate that it works as intended.
    • Continuous learning. I keep learning every day and stay up to date in my field.
  2. 2022 to 2026

    QAlified (GeneXus Consulting)

    QA Leader and go-to person for applied AI · Creator and Product Manager of ArtificialQA

    I designed and carried out the AI agent evaluation strategy in real projects. That experience gave rise to ArtificialQA, which I envisioned, designed, built, and took to production at QAlified. I drove the adoption of generative AI across the company, with a focus on security. I also led testing and quality teams where young people, often in their first job, grew and came to value their work.

    What I did at QAlified
    • Agent evaluation. I conceived, designed, and carried out the AI agent evaluation strategy in real projects, from RAG-based chatbots to agents that make decisions and run multi-step tasks: what to evaluate, with which criteria, and what evidence each decision required.
    • Team leadership. I coordinated more than 20 testing projects for public sector agencies and private companies, several at a time, with teams of up to 7 people reporting to me, from onboarding to follow-up and performance reviews.
    • Systems quality. I led quality for accounting, financial, billing, and payroll systems, with functional, automated, performance, security, data migration, and user acceptance testing.
    • Consulting. I took part in quality consulting engagements on development and release processes, with diagnosis and improvement proposals. I also led consulting engagements to help other organizations move from traditional QA to agentic QA.
    • Internal improvement. I redesigned the onboarding program for new team members, improved reporting, and created communication templates so the same messages didn't have to be written every time.
    • First steps with AI. In January 2023, I used ChatGPT for the first time to design test cases: compared with the 16 we had, it proposed more than 40. That same year, I took the lead on the company's AI-for-functional-testing workstream.
    • Assistants and agents. I designed AI assistants for testers, and later agents to support the daily work of leads and teams. I started the company's AI meetings and organized them for a long time.
    • Training. I became a certified instructor in generative AI applied to testing and taught courses for testers inside and outside QAlified.
    • Innovation. I proposed more than 15 initiatives between 2022 and 2026, from building quality in from the requirements to using AI to test AI.
    • ArtificialQA. AI agent evaluation was manual and very slow. First we looked for ways to evaluate agents, then to automate that evaluation, and then to make it faster and better. That is how ArtificialQA was born: a platform to evaluate chatbots and AI agents with human review. It was an idea I had, and we built it together as a team over time. For almost a year, we developed it with a traditional approach. That first version launched in 2025 and gave us solid product knowledge. In early 2026, I proposed rebuilding it with AI-assisted development, with a different language and way of working: AI wrote the code, I provided the judgment and product decisions, and we validated each version before it went into production. The team later adopted this faster, more organized, and more secure way of working. With a smaller team, the new version went into production in April 2026, validated by security and performance specialists.
    • What I learned. It was one of the experiences I enjoyed most: building a product based on my knowledge of the system and its goals, without writing a single line of code. It confirmed something I always apply: what matters most is being clear about what you want to build and how to verify that it works.
  3. 2012 to 2022

    Banco Hipotecario del Uruguay

    Lead, Analysis and Testing group, Process Department (2014 to 2022) · Joined through a competitive selection process as an accounting professional (2012)

    At this state-owned mortgage bank, I saw many opportunities for improvement in the Accounting Department. I started with the manual reconciliation of suspense accounts, in a group of about 15 people that I later coordinated. Instead of just reconciling, we tackled the cause of each problem together with every area of the bank, until the task required only 2 people. Applying that approach to other processes led to the Analysis and Testing group, within the Process Department, which I led until 2022. Over those years, we analyzed, specified, and tested every system change. Then we automated testing of key processes and, finally, some of the bank's operational processes.

    What I did at Banco Hipotecario del Uruguay
    • Analysis. For each request or problem, I identified where it originated, which process it belonged to, and which areas it involved. I assessed its timing, merits, and risks.
    • Prioritization committee. I took part in the committee where the head of accounting and the bank's department managers prioritized system improvements by the risk they mitigated, the problems they solved, and the work they saved.
    • Lowest-impact solution. I determined whether a parameter or process adjustment was enough or a system change was needed, aiming to mitigate risk with the least impact on the business. I agreed on scope, budget, and reasonableness with IT, and got sign-off from the areas involved.
    • Specification. I specified requirements with IT and each process owner, for the core banking system and the systems that work with it, including their integrations, databases, and batch processes.
    • Testing. I verified that each change worked as intended, in the core system and related systems, and that accounting, risk, and every other area did its part. Acceptance testing was done together with users.
    • Change management. I supported the areas so they were part of each change, understood it, and applied it.
    • Automation. We automated testing of key processes and later some of the bank's operational processes, so people could focus on higher-value work.
    • Process design. I designed new operational processes for the bank and redesigned existing ones at the areas' request, modeled in BPMN with Bizagi, together with the accounting, finance, legal, and risk areas.
    • Documentation and parameters. I contributed to process documentation and to defining system parameters and roles, based on the problems we found and the changes we implemented.
    • Risk and audit. I worked with risk matrices and tracked audit action plans and reports to management.
    • Third parties. I worked with vendors, consulting firms, and external auditors.
  4. 2008 to 2012

    Banco Central del Uruguay

    Analyst, Non-Bank Financial Institutions Supervision

    At Uruguay's central bank and financial regulator, I took part in supervising non-bank financial institutions, with a focus on risk analysis and regulatory compliance.

    What I did at Banco Central del Uruguay
    • Joining. I joined the Non-Bank Financial Institutions Supervision area through a competitive selection process.
    • Supervision. I took part in supervising the securities market, pension fund managers, and other non-bank financial institutions.
    • Analysis. I analyzed the risks, regulatory compliance, and operations of supervised institutions.
    • Reports and examinations. I prepared technical reports and took part in examinations.
    • Education. During this period, I completed the Postgraduate Diploma in Finance, Banking concentration, and a risk-based supervision workshop by OSFI, Canada's financial supervisor.
  5. Until 2008

    Early career

    Accountant and financial analyst

    While handling administration for a family-owned business, I organized its processes and accounting so the partners could see clearly where their funds went. The people who worked with me grew along with me. Later I opened an accounting practice with a partner. As a financial analyst at a brokerage firm, I proposed improvements to client reporting.

    What I did early in my career
    • First job. Administrative work while I finished my degree.
    • Family-owned business. I handled administration and accounting for a family-owned business of about 100 people, in an administrative team of three, covering payroll and supplier payments.
    • Construction company. I worked in administration at a company that built apartment buildings and housing complexes.
    • Accounting practice. I opened an accounting practice with a partner (2006 to 2007), focused on accounting and tax filing.
    • Brokerage firm. As a financial analyst (2007 to 2008), I handled client portfolios and followed up on their transactions. I proposed automating repetitive tasks and improving reports to give clients better information.
    • Degree. I earned my degree as a Public Accountant in 2004.
My approach

How I work

In any organization, with any tool, and also when evaluating AI agents, I follow the same path.

  1. Understand the problem

    Before proposing anything, I look into where it originates, which processes and areas it involves, and how the work is done today.

  2. Design the solution

    I weigh alternatives, from a process adjustment to a technology change, and choose the one that solves the problem with the least impact and the lowest risk.

  3. Align it with the areas and the team

    I bring together the perspectives of every area the change involves, which vary by business and process: technology, risk, legal, and operations, among others. I also involve the team that will work with the process from the start, so they help build the change instead of receiving it at the end. Decisions hold because a team stands behind them.

  4. Implement and verify

    I take part in the implementation and use testing to verify that the result is what we agreed on.

  5. Support the people who use it

    I train and support the people who work with the new process until they make it their own.

  6. Document and automate

    I document what we learned and automate what is repetitive, starting with what cannot fail, so the team can spend its time on what adds value. That way, when I finish a project or leave a role, the team can carry on without me.

At the end of each stage, what I value most is leaving behind processes that work better and more efficiently, and a team and a client who are satisfied with the result.

How I solved it

Selected examples

Situations I took part in, told without internal details from any company, because those stories are theirs to tell.

ArtificialQA: evaluating what can't be predicted

What started as a spreadsheet to evaluate answers became a platform in production.

My approach

In mid-2024, we were asked to test a generative AI assistant, and we had no clear way to decide whether an answer was acceptable. I started with a spreadsheet of criteria, thresholds, and weights. When that approach became unmanageable, we added an LLM-based evaluator for each criterion, which assigns a score and explains its decision. In 2025, we built a minimum viable product with the team. In 2026, I proposed rebuilding it with AI-assisted development and built that new foundation, which we then kept developing as a team. Meanwhile, I kept evaluating agents in real projects, and each new challenge became a requirement for the platform.

Outcome

The new version went into production in April 2026, with a public API and an MCP server. We presented it as a team at conferences and workshops across the region.

From daily reconciliation to fixing the cause

In a bank's accounting, we moved from reconciling to solving the root cause.

My approach

In a bank's accounting department, a large number of suspense accounts were reconciled by hand. First we built a shared catalog so everything was recorded the same way. Then we defined validated standard journal entries and tackled the cause of the most frequent cases. We prioritized improvements by what they solved and the risk they mitigated, together with every area.

Outcome

Suspense accounts were significantly reduced, and reconciling them went from requiring about 15 people to 2. The others were reassigned to higher-value work within the bank. This way of working was later applied to other processes.

Redesigning the settlement of a complex transaction with every area

In the system it was a single transaction, but in practice it was a process that involved many areas.

My approach

We analyzed in depth the operation, the process, and the people and departments involved. We defined the solution together, in line with regulations and without increasing risk.

Outcome

Better coordination between departments, a better process for the people who used it, and a better outcome for the client.

Automating, starting with what cannot fail

A risk matrix to decide what to automate first.

My approach

We built a risk matrix to identify the key processes. First we automated testing of those processes, and then we applied what we learned to operational processes.

Outcome

Fewer manual tasks and more team time for analysis.

Driving generative AI adoption across a company

Researching first and then sharing it, so everyone could work better.

My approach

First I researched what it was and how it could improve daily work. Then I shared what I learned, through training on safe use and one-on-one conversations with leads. I proposed assistants, and later agents, for daily work, always with human review.

Outcome

Assistants, and later agents, became part of the teams' daily work.

What I studied and why

Education and training

I have been learning continuously for 25 years. I enjoy studying and staying up to date.

Public Accountant (Contadora Pública) · Universidad de la República (Uruguay's national public university) · 2004

  • AiU-Certified GenAI-Assisted Test Engineer, and certified instructor for the program

    Brightest · 2024 and 2025

    I trained in AI applied to quality and then got certified to train others.

  • Risk Management certificate program

    Némesis · 2018

    Risks are everywhere. To identify them, you have to understand how to think about them.

  • Postgraduate Diploma in Finance, Banking concentration

    Universidad Católica del Uruguay · 2011

    Working in the financial system, I needed a deep understanding of its financial side and its processes.

  • Professional NLP (Neuro-Linguistic Programming) Coach

    Instituto PNL del Uruguay · 2021

    I'm interested in understanding people and helping them, starting with my team.

AI and evaluation training

  • Introduction to evals · Anthropic's Claude Academy · 2026
  • Introducción a las pruebas de sistemas de IA · Udemy · 2024
  • Generative AI Essentials · University of Michigan · 2024
  • GeneXus Enterprise AI · GeneXus · 2024

Today I keep learning with my own agentic system, which selects news, courses, and events for me.

Languages

  • Spanish, native
  • English, intermediate
  • Portuguese, fluent in conversation, basic writing

AI and evaluation training. Introduction to evals (Anthropic's Claude Academy, 2026), Introducción a las pruebas de sistemas de IA (Udemy, 2024), Generative AI Essentials (University of Michigan, 2024), and GeneXus Enterprise AI (2024).

All training, by topic

Artificial intelligence

  • Introduction to evals, Anthropic's Claude Academy (2026)
  • Generative AI Essentials, University of Michigan on Coursera (2024)
  • GeneXus Enterprise AI, GeneXus (2024)
  • Introducción a las pruebas de sistemas de IA, Udemy (2024)
  • Taller de IA para Testers, Testing Para Todos (2023)
  • ChatGPT MasterClass para profesionales en la tecnología, Udemy (2023)

Leadership and communication

  • Programa Internacional Liderazgo y Gestión I y II, CESA Management Solutions (2021 and 2022)
  • Taller de presentaciones efectivas, Daniel Ríos Public Speaking (2024)
  • Taller Negociación, Arte y Ciencia, CMI Interser (2023)
  • Negociación Efectiva, Universidad de la Empresa (2021)
  • Gestión de Equipos de Alto Rendimiento, PNL Nakle (2019)
  • Taller de Redacción, Oficina Nacional del Servicio Civil (2010)

Risk, control, and regulation

  • Gestión de Riesgo Operacional, ALIDE (2016)
  • Gestión de los Servicios de TI, fundamentos ISO/IEC 20000, UNIT (2016)
  • Ratio de Cobertura de Liquidez, CPA Ferrere (2016)
  • Auditoría de Gestión, Deloitte (2015)
  • Normas Internacionales de Información Financiera, PwC (2014 and 2017)
  • El Presupuesto y el Sistema Contable en la Administración Pública, UPAE (2013)
  • Advanced Financial Analysis Techniques Course (2011)
  • Taller de Supervisión Basada en Riesgo, OSFI (Canada) (2009)
  • IRAE e IRPF, Colegio de Contadores, Economistas y Administradores (2007)

Processes and management

  • Especialista en Gestión por Procesos para Instituciones Financieras, ALIDE (2019)
  • Gestión de Proyectos, Universidad Católica del Uruguay (2019)
  • Formulación y Gestión de Proyectos, Oficina Nacional del Servicio Civil (2014)

Testing

  • Automatización de pruebas con Katalon Studio, GeneXus Consulting (2022)
  • Técnicas de Testing para Equipos Ágiles, Peregrinus and Abstracta (2018)
  • Testing Funcional para Usuarios, CES (2014)

Data

  • Excel Avanzado, SQL y Power BI (2021)
  • Cubos y reportes de Data Warehouse (2020)
  • Consultas SQL para la toma de decisiones, UPAE (2014)

Languages: Spanish (native) · English (intermediate) · Portuguese (fluent in conversation, basic writing)

Technical skills

Technical skills

  • AI agent evaluation strategy: criteria, thresholds, deterministic and non-deterministic checks, LLM-as-a-Judge, and evaluator calibration.
  • Generative AI, agents, MCP, RAG, LLM-as-a-Judge, AI-assisted development, and GeneXus Enterprise AI.
  • BPMN in Bizagi, end-to-end process discovery and redesign.
  • Risk matrices.
  • Advanced Excel, Power BI, and SQL.
  • Prompting for testing, and mitigation of bias and risk in AI systems.
A personal challenge

Sharing what I've learned

In recent years, I started sharing what I've learned, something I now especially enjoy. I have given dozens of talks and workshops, to audiences in five countries. In each one, I want participants to take away something useful and understand it, even without a technical background.

Natalia Nario giving a talk at TestingUY 2025
TestingUY 2025 · Montevideo
Natalia Nario speaking to the audience at Testear.la 2025
Testear.la 2025 · Buenos Aires

Teaching

  • I taught more than 20 Open Labs, hands-on workshops on testing AI agents (2026), where each participant evaluates an agent with their own cases and decides, based on evidence, whether to trust it.
  • I co-taught two editions of the course on generative AI applied to testing in Brightest's AiU program (2025), on using prompts to review requirements, create test cases and test data, and report defects.
  • I taught testing courses with my own exercises, from requirements review to test case design, for teams and clients (2022 to 2024).
  • I gave a guest lecture in a software engineering postgraduate program at the Facultad de Ingeniería (UdelaR, 2025).
  • I taught classes in two CES testing diploma programs (2026) and was invited to their final project review panel.

Selected talks: Quality Sense Conf 2025 (Abstracta), TestingUY 2025 and 2026, Testear.la 2025 (Buenos Aires), CUTI Talk IA, IT Builders Live 26, Testing Day Bolivia, and Testing Perú 2026. Teaching: two editions of the course on generative AI applied to testing (Brightest, 2025), classes in two CES diploma programs, and a guest lecture in a postgraduate program at the Facultad de Ingeniería (UdelaR).

Details, examples, and videos at natalianario.com

Talks, workshops, and interviews

Talks delivered in Spanish.

2026

2025 and 2024

Contact

Let's talk

We can work together in any of these three areas:

  • AI agent evaluation
  • Process transformation in banking, finance, fintech, and audit
  • Training in AI evaluation and responsible AI

Send me a message

What would you like to do?

Messages reach only me. A note is published only if you check the box, and only after I read it. Visits to this page are measured anonymously, without cookies.