The Problem We Had to Solve
Picture this: You have templates with placeholders like a form letter with blanks to fill in. Simple enough, right? But what if those blanks need to be filled with data that comes from different sources, needs formatting, calculations, or conditional logic? And what if this needs to happen automatically, hundreds of times a day, without anyone manually copying and pasting?
That was our challenge.
We were working on a workflow automation system where templates would arrive with embedded variables, and our system needed to intelligently populate them with real data and pass them along all triggered by a simple approval action. Think of it like a highly sophisticated mail-merge, but for complex data structures instead of simple text fields.
Writing custom code to handle every possible string manipulation, date formatting, and data transformation would be like reinventing the wheel every time we needed to roll something. The technical debt would pile up fast, and maintenance would become a nightmare.
So we went looking for a better way.
A Quick Detour: Understanding DSLs
Before we dive into what we found, let’s talk about Domain Specific Languages (DSLs).
You know how Excel has formulas? That’s a DSL a mini language designed specifically for spreadsheet calculations. You don’t need to be a programmer to write =SUM(A1:A10). It’s concise, focused, and does exactly what it needs to do.
That’s what we needed: a language specifically designed for data transformation. Not a full programming language with all its complexity, but something focused, safe, and powerful enough to handle our use cases.
Exploring the Landscape
JMESPath: The Query Specialist
JMESPath is like a really good search tool for JSON data. If you’ve used AWS CLI, you’ve probably encountered it.
What it’s good at:
- Finding specific data in JSON structures
- Filtering arrays based on conditions
- Simple, readable syntax
- Fast and lightweight
Where it falls short:
- It’s built for querying, not transforming
- Limited string manipulation
- Can’t define custom functions
- Not great at constructing new data structures
Quick example:
patients[?age > `18`].{name: name, id: patientId}
Translation: “Give me the name and ID of all patients over 18.”
Great for finding data, but not for reshaping or manipulating it.
Handlebars: The Template Engine
Handlebars is the template engine you’ve probably seen in web applications. It’s all about taking data and plugging it into presentation templates.
What it’s good at:
- Simple variable substitution
- Clean separation of templates and data
- Easy to learn
- Great for generating HTML or text
Where it falls short:
- Focused on presentation, not transformation
- Limited data manipulation out of the box
- Need to write lots of custom helper functions
- Not designed for complex JSON restructuring
Quick example:
{{#each patients}}
Patient: {{name}}, DOB: {{formatDate dateOfBirth}}
{{/each}}
Perfect for display logic, but you’re on your own for complex transformations.
Why We Chose JSONata
After evaluating both options, we chose JSONata and it turned out to be exactly what we needed.
1. It’s Built for Transformation
JSONata doesn’t just query data or display it transforms it. You can take JSON in one shape and output it in a completely different shape. Need to combine fields? Format dates? Filter and aggregate? It’s all there.
2. Incredibly Rich Function Library
Out of the box, JSONata gives you:
- String magic: substring, replace, split, join, trim, uppercase, lowercase, and more
- Date/time operations: current timestamps, date parsing, formatting
- Math and aggregations: sum, average, max, min, count
- Array superpowers: map, filter, reduce, sort functional programming at its finest
- Higher-order functions: write elegant, expressive transformations
3. Custom Functions = Future-Proofing
This was the game changer. JSONata lets you register your own JavaScript functions that can be called from expressions.
For our HL7 template project, we needed special date formatting. Instead of building an entire custom engine, we just registered a function:
// Define once
const customFunctions = {
formatHL7Date: (date) => date.replace(/-/g, '')
};
// Use anywhere in expressions
$formatHL7Date(patient.dateOfBirth)
This meant we could give users powerful, domain specific capabilities without building everything from scratch.
4. Zero Technical Debt
By choosing JSONata, we avoided:
- Building our own expression parser
- Writing and testing hundreds of string manipulation functions
- Creating our own function registry system
- Debugging edge cases in date/time handling
Someone else had already solved these problems—really well.
5. Battle-Tested in Production
JSONata isn’t some experimental library. It’s used by:
- AWS Step Functions for state transformations
- n8n for workflow automation
- Node-RED (IBM’s automation platform)
When the big players trust it with mission-critical workflows, that’s a good sign.
A Solution to our Real-World Usecase
Let me walk you through how we actually used JSONata to solve our HL7 template automation problem.
The Scenario
In healthcare, HL7 messages are the standard format for exchanging patient data between systems. They look something like this:
MSH|^~\&|SYSTEM|FACILITY|APP|DESTINATION|20241020120000||ADT^A08|...
PID|||12345||DOE^JOHN||19850315|M|...
Each line is a “segment,” and each segment has fields separated by pipes (|).
The Challenge
Our clients needed to:
- Store message templates with placeholders
- Receive an approval trigger (like clicking “Approve Patient Update”)
- Automatically populate the template with real patient data
- Send the completed message to the destination system
All without anyone touching the data manually.
The Solution
Step 1: Template with Variables
We stored templates like this:
MSH|^~\&|SYSTEM|FACILITY|APP|${facility.code}|${$now()}||ADT^A08|...
PID|||${patient.id}||${patient.lastName}^${patient.firstName}||${$formatHL7Date(patient.dob)}|${patient.gender}|...
Notice the ${} placeholders—those are JSONata expressions.
Step 2: Automatic Evaluation
When an approval comes in, our workflow:
- Loads the template
- Loads the patient data
- Evaluates every
${}expression using JSONata - Produces the final message
Step 3: Complex Transformations Made Easy
Here’s where JSONata really shines:
// Combine multiple address fields with HL7 separator
${$join([address.street, address.city, address.state, address.zip], '^')}
// Conditional formatting
${patient.dob ? $formatHL7Date(patient.dob) : 'UNKNOWN'}
// Handle multiple diagnoses (repeating segments)
${$join($map(patient.diagnoses, function($d) {
'DG1|' & $d.sequence & '|' & $d.code & '|' & $d.description
}), '\n')}
All of this happens inline, in the template, with no custom code required.
Step 4: True Automation
Now the entire flow is hands-off:
- User clicks “Approve”
- Template loads
- Data populates
- Message sends
No copy paste. No manual formatting. No errors.
The Takeaway
1. Match the Tool to the Job
- JMESPath → When you need to query and extract from JSON
- Handlebars → When you need to display data in templates
- JSONata → When you need to transform and restructure data
2. DSLs Lower the Barrier
With JSONata, we could hand off template creation to analysts who understood the data but weren’t programmers. The syntax was approachable enough to learn, but powerful enough to handle complex cases.
3. Extensibility Is Everything
The ability to add custom functions meant we could start simple and grow as needs evolved without refactoring the entire system.
4. Don’t Reinvent What’s Already Solved
JSONata has years of community testing, edge case handling, and performance optimization for free. Building it ourselves would have taken months and never been as robust.
If you’re facing challenges with dynamic data transformation whether it’s message templates, API response mapping, report generation, or workflow automation JSONata is worth serious consideration.
It’s one of those tools that’s powerful enough for complex enterprise needs but approachable enough to hand to non-developers. And it’s proven itself in production at some of the biggest tech companies in the world.
For us, it turned what could have been months of custom development into a weekend project. That’s the kind of leverage every development team needs.
Have you tackled similar data transformation challenges? What tools or approaches did you use? I’m always curious to learn how others solve these problems. Drop a comment or reach out let’s share notes!
P.S. If you want to explore JSONata yourself, their interactive exerciser tool is a great place to start. You can experiment with expressions in realtime and see the results immediately.
References:
- jmespath - JMESPath is a query language for JSON.
- handlebars
- JSONata - Playground