Scripts & Tasks


CUE’s scripting layer is built on top of tools/flow . In this area, you can define tasks with CUE which interact with the outside world. In other words, where non-hermetic operations are allowed.

Enforce Dependencies

cue/flow can automatically infer task dependencies based on references between their values. Tasks, where all tasks it dependens have on have completed, are available to run.

autodep_tool.cue

package scripts

import "tool/cli"

import "tool/file"

command: autodep: {
	read: file.Read & {
		filename: "data.json"
		contents: string
	}

	print: cli.Print & {
		text: read.contents // an inferred dependency
	}
}

But what happens when there is no dependency, yet the order is important?

unenforced_tool.cue

package scripts

import "tool/file"

command: unenforced: {
	mkdir: file.MkdirAll & {
		path: "out/write/"
	}

	write: file.Create & {
		filename: "out/write/foo.txt"
		contents: "hello world"
	}
}

$ cue write

task failed: open out/write/foo.txt: no such file or directory

The above code may or may not have this output, as both tasks are available to run and the script is non-deterministic.

When this situation arises, we need to tell cue/flow that a dependency between the tasks exists by making an explicit reference. This is like “using” the output from a first task in the second task, but actually ignoring the output.

enforced_tool.cue

package scripts

import "tool/exec"

import "tool/file"

command: enforced: {
	mkdir: file.MkdirAll & {
		path: "out/write/"
	}

	write: file.Create & {
		$dep:     mkdir.$done //  explicit dependency 
		filename: "out/write/foo.txt"
		contents: "hello world"
	}

	clean: exec.Run & {
		$dep: write.$done //  explicit dependency 
		cmd:  "rm -rf ./out"
	}
}

This new version, with $dep: mkdir.$done is guaranteed to have consistent and correct order.

Comprehending Tasks

Another common pattern in scripting and tasks are

  • running tasks for each entry in a list or map
  • conditionally running a task

We can use CUE’s comprehenesion capabilities for both of these, even together.

comprehend_tool.cue

package scripts

import "tool/cli"

import "tool/file"

maxlen: 16

command: foreach: {

	// get a list of files
	list: file.Glob & {
		glob: "*.cue"
	}

	// comprehend tasks for each file, also an inferred dependency
	for _, filepath in list.files {
		// make a unique key for the tasks per item
		(filepath): {
			// and have locally referenced dependencies
			read: file.Read & {
				filename: filepath
				contents: string
			}
			print: cli.Print & {
				text: read.contents // an inferred dependency
			}
		}
	}
}

command: maybe: {
	// get a list of files
	list: file.Glob & {
		glob: "*.cue"
	}

	// comprehend tasks for each file, also an inferred dependency
	for _, filepath in list.files {
		// only print long names
		if len(filepath) > maxlen {
			(filepath): print: cli.Print & {
				text: filepath // an inferred dependency
			}
		}
	}

}

Sequential Comprehended Tasks

What if we combine the last two examples?

  • The comprehended tasks would run in parallel
  • We actually want them to run sequentially

We can conditionally refer to the previous task as follows:

sequential_tool.cue

package sequential

import "list"

import "tool/cli"

command: print: {
	for i, _ in list.Range(0, 5, 1) {
		"p\(i)": cli.Print & {
			// enforce dependencies between consecutive tasks
			if i > 0 {
				$dep: command.print["p\(i-1)"].$done
			}

			// input fields for the task
			text: "hello \(i)"
		}
	}
}

Scripts with args

We can use the @tag() attribute as a way to inject values or args when running a script.

cue cmd -t msg=foobar print args_tool.cue

args_tool.cue

package args_tool

import "tool/cli"

args: {
	msg: string | *"hello world" @tag(msg)
}

command: print: cli.Print & {
	text: args.msg
}

Reusable Scripts and Tasks

We can define reusable tasks, though some extra care is needed when using them. Additionally we can create values containing many tasks and even import them.

The follow does not work because we reference a single field within the task, but never the full task value. cue cmd only runs tasks which are eventually nested under the root value of the command you run.

brokenref_tool.cue

package scripts

import "tool/cli"

import "tool/file"

// get a list of files
locallist: file.Glob & {
	glob: "*.cue"
}

command: brokenref: {

	// comprehend tasks for each file, also an inferred dependency
	for _, filepath in locallist.files {
		(filepath): print: cli.Print & {
			text: filepath // an inferred dependency
		}
	}

}

To fix this, we use localized references to add the task within the task to run. Note, this can be used to import tasks which import tasks, ad infinitum, as long as every task has localized references to the tasks it depends on.

localref_tool.cue

package scripts

import "tool/cli"

import "tool/file"

// get a list of files
locallist: file.Glob & {
	glob: "*.cue"
}

command: localref: {

	list: locallist

	// comprehend tasks for each file, also an inferred dependency
	for _, filepath in list.files {
		(filepath): print: cli.Print & {
			text: filepath // an inferred dependency
		}
	}

}

Importing Tasks

Importing tasks works around two things:

  1. _tool.cue files cannot be imported
  2. tasks packages can only be imported in _tool.cue files.

Instead, we code with care and

  1. Put reusable tasks in regulare .cue files
  2. Skip importing the packages and schemas
  3. Use the $id field and ensure the other fields are correct

import_tool.cue

package scripts

import (
	"encoding/json"
	"hof.io/example/load"
)

vars: {
	// @tag() is used with the -t flag to inject dynamic values
	user: string | *"dr_verm" @tag(user)
}

meta: {
	secrets: {
		// we can import and add a task to our tasks and scripts
		tLoad: load.load
		token: tLoad.say
	}

	req: {
		url:    "https://postman-echo.com/get?cow=\(secrets.token)"
		method: "GET"
	}
}

command: authd: {
	// localized dependency
	cfg: meta

	get: {
		// we can reuse values and infer dependencies
		req: cfg.req & {
			$id: "tool/http.Do"
		}
		resp: req.response
		// we can process task outputs, and still infer depenencies
		Out: json.Indent(resp.body, "", " ")
	}
	print: {
		$id: "tool/cli.Print"
		// still makes a dependency (get.Out)
		text: "\(get.Out) @\(vars.user)"
	}
}

load/secret.cue

// This flow gets an api code with OAuth workflow
package load

import (
	"encoding/json"

	"hof.io/example/utils"
)

meta: {
	vars: {
		// import a reusable task
		RR:   utils.RepoRoot // localized dependency
		root: RR.Out         // inferred dependency
		fn:   "\(root)/code/patterns/scripts-and-tasks/data.json"
	}
}

load: {
	cfg: meta // localized dependency

	read: {
		$id:      "tool/file.Read"
		filename: cfg.vars.fn // inferred dependency
		contents: string
	}

	data: json.Unmarshal(read.contents)
	say:  data.cow

	print: {
		$id:  "tool/cli.Print"
		text: read.contents // inferred dependency
	}
}

utils/dir.cue

package utils

import "strings"

// A reusable task to return the root path of a git repository
RepoRoot: {
	// this is the special, task identifying field
	$id: "tool/exec.Run"
	cmd: ["bash", "-c", "git rev-parse --show-toplevel"]
	stdout: string
	Out:    strings.TrimSpace(stdout)
}

hof/flow

At Hofstadter, we are building a custom task engine on cue/flow. It adds task analytics, visualization, distributability, and many additional task types.

Injecting values

Get CUE support and training from Hofstadter

Published 2024-03-31

Edit this page Create page issue Create project issue
We'll never share your email with anyone else.
2024 Hofstadter, Inc