Transformative moments

A new tool for optimizing your interactions with data

Now and again innovations change the way we live and digital tooling simplifies the way we work. In the early days of the internet, XML shaped e-commerce and system integration. Later REST optimized the time it took to bring products to market. However, only well endowed businesses that were able to spend millions in licensing fees could effectively use these technologies.

In our daily interactions with the digital world we often find ourselves transforming data between XML, JSON, CSV, Text, YAML and Properties to name but a few. To do so we needed to create complex programs from scratch. What we needed was a scripting language where we could use simple intuitive commands to invoke complex transformations. DataWeave is that language. Popularized by Mulesoft and an integral component of their technology stack, DataWeave is an Open Source product licensed under Apache. Mulesoft is a company that maintains a close collaboration with the open-source community. Their development team actively improves DataWeave, the current version at the time of this writing is 2.0.

In my daily routines, I always felt there was a need for a way of creating information pipelines between Linux processes that worked with internet API’s. An example of this might be: using curl or httpie to pull API data from a server, pipe the output to another command to transform, filter and produce another form. The result could then be copied from the terminal or redirected to a file.

Recently I came across a blog post by Josh Erney, describing the DataWeave CLI, it was exactly what I was looking for. When I first looked at the Github repository there was only a version which could be installed on a Mac using Brew. Today there are versions for Mac, Linux and Windows (see the paragraph titled install Manually). Kudo’s to Josh and the DataWeave team for creating an awesome CLI!

In the sections below i’ll share some examples to help you get going. The sample data and transforms can be found in my git repository. They’re based on some example and explanations found in the Dataweave Cookbook. Don’t worry about the code now, we’ll fetch it later using the DW CLI.

First steps

Ensure you have the DW CLI installed on your Windows, Mac or Linux laptop. By default, DW CLI will use a .dw folder in your Home directory to manage local files, but we’re going to override that, as shown below. The examples which follow were verified to run with DW CLI version 1.0.9.

rem Override DW CLI HOME in Windows
set DW_HOME=C:\dev\Mule\dw
set DW_LIB_PATH=%DW_HOME%\libs
---
# Override DW CLI HOME in Linux
export DW_HOME=$HOME/dev/Mule/dw
export DW_LIB_PATH=$DW_HOME/libs

To make these changes permanent you can add them to your path in the Windows environment dialog settings or to the .bashrc file in your Linux HOME directory.

You’ll also need to add the location where you installed the DW CLI to your path variable. In Window I install into a C:\Tools folder and $HOME/tools in Linux. Be sure to adjust the locations below if you’ve installed elsewhere.

rem Configure DW CLI install location in Windows
set PATH=\tools\dw\bin;%PATH%
---
# Configure DW CLI install location in Linux
export PATH=$HOME/tools/dw/bin:$PATH

To ensure your the DW CLI is properly configured type the dw command with no arguments and press enter. If it’s working you should get a help screen.

Examples

With the setup out of the way let’s play with some simple examples. Like with the Go programming language, the DW CLI will fetch necessary artifacts from Github repositories and download them to DW_HOME. You’ll have to familiarize yourself a bit with the concepts of: grimoires, spells, wizards and incantations. In short, wizards are similar to Linux Guru’s - experts who create complex scripts that can perform magical transformations. Grimoires are representative of the many Github pages that contain the magic and the incantation created by a wizard, which will create your magical data transforms.

The HelloWizard transformation is shown below. The first time you run it, it goes to the DW CLI wizard’s Github page, downloads the Grimoire, creates a new folder in your DW_HOME where it’s saved and runs the code (invokes the incantation). By the time you run this, the typo may have already been corrected :)

Hello incantation

dw --spell HelloWizard
Fetching `null's` Grimoire.
Cloning into 'C:\Home\dev\Mule\dw\grimoires\data-weave-grimoire'...
remote: Enumerating objects: 46, done.
remote: Counting objects: 100% (46/46), done.
remote: Compressing objects: 100% (37/37), done.
remote: Total 46 (delta 9), reused 29 (delta 1), pack-reused 0
Unpacking objects: 100% (46/46), 6.92 KiB | 77.00 KiB/s, done.
"Hello My Wise Friend. Wellcome to the MAGIC world of DataWeave"

Re-run the same spell again and DW CLI will run it from your DW_HOME.

dw --spell HelloWizard
"Hello My Wise Friend. Wellcome to the MAGIC world of DataWeave"

You can also download code created by a Wizard, from which you can later run an incantation.

dw --add-wizard MitchDresdner
Downloading Grimoire From The Wise: `MitchDresdner`.
Fetching `MitchDresdner's` Grimoire.
Cloning into 'C:\dev\Mule\dw\grimoires\MitchDresdner-data-weave-grimoire'...
remote: Enumerating objects: 36, done.
remote: Counting objects: 100% (36/36), done.
remote: Compressing objects: 100% (20/20), done.
remote: Total 36 (delta 2), reused 33 (delta 1), pack-reused 0
Unpacking objects: 100% (36/36), 4.43 KiB | 70.00 KiB/s, done.

You gotta love the accolade it gives to the wizard when downloading!

With the Grimoire’s downloaded you can begin making incantations. Be sure to prefix the incantation with the name of the wizard or you’ll be referencing the default grimoire.

dw --spell MitchDresdner/HelloWizard
"Per ardua ad astra"

If you haven’t done so already, be sure to create a libs folder under DW_HOME.

rem create libs folder under DW_HOME in Windows
md %DW_HOME%\libs
---
# create libs folder under DW_HOME in Linux
mkdir $DW_HOME/libs

In the libs folder create the following file named utils.dwl

%dw 2.0
fun addUnderscore(s: String): String = s ++ "_"

The next example demonstrates how you would import custom functions into your incantation.

We’ll need to pass the path parameter in the incantation to let the DW CLI know to look for import in the libs folder. The code for Underscore is show below, it’s already created. We show it for completeness so you can see how the utils function is invoked.

%dw 2.0
output application/json
import * from utils
---
{
 underscored: addUnderscore("howdy")
}

Underscore

dw --path %DW_HOME%\libs --spell MitchDresdner/Underscore
{
  "underscored": "howdy_"
}

Next let’s invoke an incantation which passes a JSON payload, filters the data and transforms the results. Before we start, we’ll bind a folder which contains our sample data to an environment variable.

rem create env var for JSON sample data in Windows
set JSON_DATA=%DW_HOME%\grimoires\MitchDresdner-data-weave-grimoire\JSON
---
# create env var for JSON sample data in Linux
export JSON_DATA=$DW_HOME/grimoires/MitchDresdner-data-weave-grimoire/JSON

This example comes from the DW CLI Github page, it filters for users old enough to consume beer (legally). You may need to adjust the business logic to conform with minimum age rules for your locality. :)

By passing payload with input data, the incantation will know to find input data using the payload reference.

dw -i payload %JSON_DATA%\users.json --spell MitchDresdner/UserAge
[
  {
    "name": "User1",
    "age": 19
  },
  {
    "name": "User2",
    "age": 18
  }
]

Now we’ll move on to try a few additional examples from the DW Cookbook. Feel free to add additional spells as you feel confident using the DW CLI.

In the GreetMe spell, we demonstrate how you might add some log messages to your spell. The logging might be useful when you’re trying to figure out where something might be going wrong.

%dw 2.0
output application/json
// 1st class lambda functions
var sayHowdy = (whom) -> "Greetings $(whom)"
var dbgMsg = (loglvl, logmsg) -> log(loglvl, logmsg)
---
{
   greetMe: sayHowdy(payload.message),
   logme: dbgMsg("WARNING", "Houston, we have a problem."),
   arrayCat : payload.a1 ++ payload.a2,
   strCat: "Go Home " ++ payload.message,
   // concat two objects, flatten
   objCat: {"person": payload.message} ++ payload.phone,
   finis: dbgMsg("INFO","Return to Earth.")
}

GreetMe

dw -i payload %JSON_DATA%\\Greet.json --spell MitchDresdner/GreetMe
WARNING - "Houston, we have a problem."
INFO - "Return to Earth."
{
  "greetMe": "Greetings Earthling",
  "logme": "Houston, we have a problem.",
  "arrayCat": [
    0,
    1,
    2,
    false,
    {
      "id": 42
    },
    "a",
    "b",
    "c"
  ],
  "strCat": "Go Home Earthling",
  "objCat": {
    "person": "Earthling",
    "cel": "212-555-1212"
  },
  "finis": "Return to Earth."
}

In our next examples we show how you might convert a CSV file to JSON. You can see how you might do the same with data pulled down from the internet.

Before we get started, lets create an environment variable which contains the location for our CSV sample data.

rem create env var for CSV sample data in Windows
set CSV_DATA=%DW_HOME%\grimoires\MitchDresdner-data-weave-grimoire\CSV
---
# create env var for CSV sample data in Linux
export CSV_DATA=$DW_HOME/grimoires/MitchDresdner-data-weave-grimoire/CSV

HomeSales

We’ll reference our CSV data using csvFile same as we did in our earlier payload example.

dw -i csvFile %CSV_DATA%\HomeSales.csv --spell MitchDresdner/HomeSales
[
  {
    "sellPrice": 142,
    "listPrice": 160,
    "itemAmount": 3167
  },
  {
    "sellPrice": 175,
    "listPrice": 180,
    "itemAmount": 4033
  },
  {
    "sellPrice": 129,
    "listPrice": 132,
    "itemAmount": 1471
  },
  {
    "sellPrice": 138,
    "listPrice": 140,
    "itemAmount": 3204
  }
]

In our last example we show how you might go about calculating total prices from a shopping cart. In our sample data, the user chooses a couple bottles of infamous though relatively inexpensive wine.

%dw 2.0
output application/json
type Currency = String { format: "\$#,###.00" }
var dbgMsg = (loglvl, logmsg) -> log(loglvl, logmsg)
var calcTax = (price,taxRate) -> (price * taxRate)
var calcTotal = (price,taxRate) -> (price * taxRate + price)
var subTotal = (payload.wine reduce (wine, total = 0) -> 
            total + wine.price)
---
{
    order: "",
    items: payload.wine map {
        id: $.id,
        price: $.price       
    },
subtotal: (payload.wine reduce (wine, total = 0) -> 
      total + wine.price)  as Currency,
    taxes: calcTax(subTotal, payload.taxRate) as Currency,
    total: calcTotal(subTotal,  payload.taxRate) as Currency 
}

CartCalc

dw -i payload %JSON_DATA%\CartCalc.json --spell
MitchDresdner/CartCalc
{
  "order": "",
  "items": [
    {
      "id": 42,
      "price": 13.99
    },
    {
      "id": 24,
      "price": 14.92
    }
  ],
  "subtotal": "$28.91",
  "taxes": "$1.88",
  "total": "$30.79"
}

By now your confidence should be pretty high and we encourage you to try out more examples. If you find some fun or profitable ones be sure to let us know.

Be sure to thank the DW Team for creating an awesome CLI!