Dataweave Cookbook

9 min readAug 2, 2020

Some common, useful and transformative recipe’s using DW 2.0

In the recipe’s which follow, you can explore various Dataweave transform recipe’s using AnypointStudio or using the Dataweave Playground that I talked about in an earlier post.

I’ll present these recipe’s using the Playground because it’s simple, easy to use and follow along. I’ll share the input payload, transform and output payload of each for completeness. If you have any transformations that you would like to have included, feel free to reach out to me. When I apply a pattern more than once it makes me consider if it has a more general purpose and might be used again. My goal for the cookbook is to build a catalog of commonly used transformations that I can come back to when I need one. To that end, i’ll be adding new patterns over time. Hopefully the cookbook can be useful for you too.

In this cookbook you won’t find any temperature settings and cooking duration. If you were expecting to see them, this may be the wrong cookbook for you. I expect these recipes to be a bit easier to follow than the standard culinary guide. Hopefully you’ll find them useful across the pattern space, wherever they can be applied.

Concatenation
Redaction
Contains
Lambda calculators
Lambda calculator summing prices in an array
A specific product from a list
Converting CSV data to JSON
Date format and conversion

Concatenation recipe’s

This recipe demonstrates the use of some simple lambda functions with some concatenations. It also uses the logmsg function, which can be useful when you’re debugging a complex script. By moving the logging line around in your script it can help to isolate a particular line among the many, to help you pinpoint what’s causing scripting problems.

Ingredients

Mix in (2) log statements
Array concatenation
String concatenation
JSON Object concatenation

Payload

{
   "message": "Earthling",
   "a1": [0, 1, 2, false],
   "a2": [{"id": 42}, "a", "b", "c"],
   "phone": {"cel": "212-555-1212"}
}

Transform

%dw 2.0output application/json// 1st class lambda functions
var sayHowdy = (whom) -> "Greetings $(whom)"
var dbgMsg = (loglvl, logmsg) -> log(loglvl, logmsg)---
{
   greetMe: sayHowdy(payload.message),
   logme: dbgMsg("WARNING", "Houston, we have a problem."),
   arrayCat : payload.a1 ++ payload.a2,
   strCat: "Go Home " ++ payload.message,
   // concat two objects, flatten
   objCat: {"person": payload.message} ++ payload.phone,
   finis: dbgMsg("INFO","Return to Earth.")
}

Redaction recipe

Redaction is not always reduction. With reduction your goal is to thicken a sauce, thereby intensifying the flavors. Redaction takes some ingredient away. It would be like taking away some salt if you seasoned a recipe with a bit too much. Or removing some heat or sugar from a craft beer recipe to temper the fermentation.

Redaction is a popular pattern used by law enforcement, government and the security services. By following these patterns redaction can lead to a reduction, but it can also result in a reduction of meaning. A proper solution should consider all the pros and cons.

In the culinary arts it’s often difficult to take something away when you’ve added too much of. Not so with Dataweave, where redaction is formulaic. For example, let’s say you just received a new use case to redact King Pins and Mob Bosses from a FOIA request, leaving just the fall guy. How might you go about solving the request?

Ingredients

Sensitive data in the form of a payload
Keyword list to redact from payload

Transform

%dw 2.0output application/json---// redact any vip's
payload -- ["kingPin", "mobBoss"]

Payload

{
   "kingPin": "Al Capone",
   "fallGuy": "James Belcastro",
   "mobBoss": "Bugs Moran"
}

Contains recipe

Searching for needles in haystacks can be simplified by using the contains function. You might find this to be a handy pattern when you need to apply a script to and error handling chain. For the purpose of this demonstration we’ll add the example in line.

Let’s say you have a requirement for handling exceptions that are thrown when performing database inserts of duplicate records. One way you might implement this, isto the inspect the error code to determine whether it contains the specific error code which is emitted for duplicates.

Ingredients

A payload containing error codes

When your script exhibits the behavior you’re looking for it can be inserted into the error handling chain.

Transform

%dw 2.0output application/json---{
   duplicateError: payload.error contains "-803"
}

Payload

{
   error": "Blah blah error SQLCODE=-803"
}

Lambda calculator recipe

A restaurant, a wine bar or any other merchant for that matter, won’t remain in business for very long if they can’t charge for their product. It’s likely, in your career, that some of these businesses will come to you for a proper solution.

In this recipe we’ll look at some basic calculators using variables as lambda functions. I don’t think lambda’s are very scary to many developers these days, even though they use pointy spear like symbols →. Just the same, i’ve also created a function fCalcTax which duplicates the behavior of the lambda function calcTax. Functions seem to be the result of sprinkling syntactic sugar on the lambda to get them gussied it up as a functions, just for fun. I don’t really know, but have included the fCalcTax function commented out below in case you’d like to compare the two.

Ingredients

Currency string format
A bottle of Mogen David to bill for
Tax and total calculators

Transform

%dw 2.0output application/jsontype Currency = String { format: "\$#,###.00" }var dbgMsg = (loglvl, logmsg) -> log(loglvl, logmsg)var calcTax = (price,taxRate) -> (price * taxRate)
var calcTotal = (price,taxRate) -> (price * taxRate + price)fun fCalcTax(price,taxRate) = dbgMsg("INFO", (price * taxRate))---{
   id: payload.wine.id,
   price: payload.wine.price,
   tax: calcTax(payload.wine.price, payload.wine.taxRate) 
     as Currency,
   // dbg: fCalcTax(payload.wine.price, payload.wine.taxRate) 
   //  as Currency,
   total: calcTotal(payload.wine.price, payload.wine.taxRate) 
     as Currency,
   product: payload.wine.details
}

Payload

{
   "wine":  {
     "id": 42,
     "product": "MD 20/20",
     "price": 13.99,
     "details": "https://www.totalwine.com/wine/fruit-wine/fruit-blends/mogen-david-blackberry/p/13618030-1",
     "taxRate": 0.065
   }
}

Lambda array calculator recipe

When you have no one to impress but yourself, does it really make sense to purchase good wine?

If you answered no, be sure to purchase some cheap-but-good (cheap-but-good — is that an oxymoron?) wines which won’t leave your head in a hurt in the morning. In other words, don’t attempt drinking these examples or do so at your own peril.

We’ll reuse our calculators from the previous example to derive a subtotal using an accumulator to sum the prices using the reduce function. We’ll also calculate the taxes due to our rich Uncle and the total cost. This pattern might actually be something you an apply elsewhere.

Ingredients

A couple bottles to go
A loop
Some simple math

Transform

%dw 2.0
output application/json
type Currency = String { format: "\$#,###.00" }var dbgMsg = (loglvl, logmsg) -> log(loglvl, logmsg)
var calcTax = (price,taxRate) -> (price * taxRate)
var calcTotal = (price,taxRate) -> (price * taxRate + price)var subTotal = (payload.wine reduce (wine, total = 0) -> 
            total + wine.price)---{
    order: "",
    items: payload.wine map {
        id: $.id,
        price: $.price       
    },subtotal: (payload.wine reduce (wine, total = 0) -> 
      total + wine.price)  as Currency,
    taxes: calcTax(subTotal, payload.taxRate) as Currency,
    total: calcTotal(subTotal,  payload.taxRate) as Currency 
}

Payload

{
    "taxRate": 0.065,
    "wine": [ 
        {
            "id": 42,
            "product": "MD 20/20",
            "price": 13.99      
        },
        {
            "id": 24,
            "product": "Boones Farm",
            "price": 14.92      
        }
    ]
}

Specific product in a list recipe

Ingredients

A number of different products
A filter

Transform

%dw 2.0
output application/json
type Currency = String { format: "\$#,###.00" }
var dbgMsg = (loglvl, logmsg) -> log(loglvl, logmsg)
var calcTax = (price,taxRate) -> (price * taxRate)
var calcTotal = (price,taxRate) -> (price * taxRate + price)
var subTotal = (payload.products.*wine reduce (product, total = 0) -> 
            total + product.price)
---
{
    order: "",
    items: payload.products.*wine map {
        id: $.id,
        price: $.price       
    },
    subtotal: (payload.products.*wine reduce (product, total = 0) -> 
      total + product.price)  as Currency,
    taxes: calcTax(subTotal, payload.taxRate) as Currency,
    total: calcTotal(subTotal,  payload.taxRate) as Currency 
}

Payload

{
    "taxRate": 0.065,
    "products": { 
        "wine": {
            "id": 42,
            "product": "MD 20/20",
            "price": 13.99      
        },
        "wine": {
            "id": 24,
            "product": "Boones Farm",
            "price": 14.92      
        },
        "beer": {
            "id": 99,
            "product": "Dead Guy Ale",
            "price": 8.99
        }
    }
}

CSV Recipe

The rumors of the demise of CSV’s have been greatly exaggerated. It’s quite likely that you’ll encounter files which need to be transformed at some point in your development career, so lets take a look at a basic CSV recipe.

The use case we’re given is to convert a CSV file containing home sale data into structured data. When it comes to file data there’s lots of ways that errors can creep in. In our first attempt at a structured transform the data appears to be ok, but we get an error when converting the List price to a numeric value.

While the data looks ok, the leading space after the comma which may make the data more readable to the eye, causes problems with our conversion. The leading space causes the data to be recognized as a string. To solve the problem, we decide to apply the trim function to remove whitespace surrounding the column values.

The refactored transform works much better!

Ingredients

Home sales
Ask, Sell
An Error

Transform

%dw 2.0output application/jsonvar invoiceNo = 10000---
csvFile flatMap (line, xy) -> [
  {
    sellPrice  : line.Sell  as Number,
    listPrice  : trim(line.List) as Number,
    itemAmount : trim(line.Taxes) as Number
  }
]

Payload

Sell,List,Living,Rooms,Beds,Baths,Age,Taxes
142, 160, 28, 10, 5, 3,  60, 3167
175, 180, 18,  8, 4, 1,  12, 4033
129, 132, 13,  6, 3, 1,  41, 1471
138, 140, 17,  7, 3, 1,  22, 3204

Date format and conversion

Added 16 Sep 2020

There will be times when you need to transform the Date’s you receive into a format which is more palatable to your business logic.

Let’s say you’re getting Date values that are causing exceptions to be throw when the format doesn’t match the RAML definition you’re expecting.

You can solve for his by defining a DW function which discards illegal values and formats valid dates into the allowable structure you require.

The format string follows Java Date Formatter conventions.

We can also trap null values which can cause DW to fail with type errors. In the NullDate mapping you would replace the variable with the value you’re mapping.

Ingredients

A valid Date
An empty Date, null Date and an invalid Date
A conversion format

Transform

%dw 2.0
output application/json skipNullOn="everywhere"var dt = '20200916'
var nilDate = nullfun dateFmt(d: String) = if (d == "" or d == "00000000") null
  else 
    d as Date {format: 'yyyyMMdd'}  as String {format: "yyyy-MM-dd"}---{
  CreateDate: dateFmt("20200916"),
  EmptyDate: dateFmt(""),
  ZerosInDate: dateFmt("00000000"),
  NullDate: if (nilDate == null) null
    else dateFmt(dt),
  EpochDate: dateFmt("19700101")
}

Payload

{
  "CreateDate": "2020-09-16",
  "EpochDate": "1970-01-01"
}

I hope you’ve enjoyed these recipes and can find some use for them in your projects. Be sure to check back from time to time, as I discover common transforms worth sharing, or receive some from readers, i’ll add them to the cookbook.

Dataweave Cookbook

Table of Contents

Written by Mitch Dresdner