YAML Processing with YQ: A Practical Guide

Table of Contents

1. Introduction
2. What is YQ?
3. Getting Started with YQ
4. Common Workflows (TLDR Style)
5. Working with Complex YAML Structures
6. Common Pitfalls and Solutions
7. Advanced Techniques
8. Conclusion

1. Introduction

YAML has become the de facto standard for configuration files in modern software development, particularly in Kubernetes, CI/CD pipelines, and other DevOps tools. While its human-readable format is one of its greatest strengths, manipulating YAML files programmatically can be challenging. This is where YQ comes in – a lightweight, portable command-line utility that brings the power of jq-like operations to YAML processing.

This post documents my experience using YQ for various YAML processing tasks, from simple value extraction to complex multi-document transformations.

2. What is YQ?

YQ is a command-line tool designed to process YAML files, similar to how JQ works with JSON. It allows you to read, filter, update, and transform YAML documents with a concise and expressive syntax.

Key features include:

• Reading and querying YAML files
• Updating and modifying YAML content
• Converting between YAML and other formats (JSON, XML, etc.)
• Processing multiple YAML documents in a single file
• Preserving comments and file structure when desired

YQ's power comes from its ability to handle complex transformations with simple, composable expressions, making it an essential tool for DevOps engineers, Kubernetes administrators, and anyone working with YAML files regularly.

3. Getting Started with YQ

3.1 Installation

You can install YQ on various platforms:

# macOS with Homebrew
brew install yq

# Linux with apt (Debian/Ubuntu)
apt-get install yq

# Windows with Chocolatey
choco install yq

# Using Go
go install github.com/mikefarah/yq/v4@latest

3.2 Basic Usage

After installation, check that YQ is working correctly:

yq --version

Basic operations follow this pattern:

# Reading a value
yq '.key' file.yaml

# Updating a value
yq -i '.key = "new value"' file.yaml

# Converting YAML to JSON
yq -o=json '.' file.yaml

4. Most Frequently Used YQ Commands

This section focuses on the commands you'll use 80% of the time when working with YAML files.

4.1 Reading Values (The Essentials)

# Get a simple value
yq '.apiVersion' deployment.yaml

# Get a nested value
yq '.metadata.name' deployment.yaml

# Get an array element
yq '.spec.containers[0].image' deployment.yaml

# Get all matching values (extremely useful!)
yq '.spec.containers[].name' deployment.yaml

4.2 In-place Editing (-i flag)

The -i flag is your best friend for modifying files directly:

# Update a value in-place (the most common operation)
yq -i '.spec.replicas = 5' deployment.yaml

# Nested updates are straightforward too
yq -i '.metadata.labels.environment = "production"' deployment.yaml

4.3 Recursive Operations with (..)

The recursive descent operator .. is extremely powerful for working with complex YAML:

# Find and update all instances of a field, regardless of nesting
yq -i '(.. | select(has("image"))).image = "nginx:latest"' deployment.yaml

# Remove all fields with a specific name throughout the document
yq -i 'del(.. | .difficulty?)' question-file.yml

4.4 Working with Multiple Documents

For YAML files with --- separators:

# Process all documents (crucial for Kubernetes manifests)
yq 'all(.metadata.name)' multi-doc.yaml

# Update a field in all documents
yq -i 'all(.metadata.namespace = "production")' multi-doc.yaml

4.5 Format Control and Conversion

# Convert YAML to JSON (extremely common for API integrations)
yq -o=json '.' config.yaml > config.json

# Pretty-print YAML for better readability
yq -P '.' messy.yaml > pretty.yaml

# Change a field's style from quoted to block literal (|)
yq -i '.description style="literal"' config.yaml

5. Working with Complex YAML Structures

5.1 Manipulating Arrays

# Add to an array
yq -i '.spec.containers += {"name":"sidecar", "image":"nginx"}' deployment.yaml

# Filter array elements
yq '.spec.containers[] | select(.name == "app")' deployment.yaml

# Map over array elements
yq '.items[] | .metadata.name' resources.yaml

# Sort an array
yq -i '.spec.containers |= sort_by(.name)' deployment.yaml

5.2 Working with Multi-Document YAML Files

Many configuration files (especially in Kubernetes) contain multiple YAML documents separated by ---. YQ can handle these easily:

# Process all documents
yq 'all(.metadata.name)' multi-doc.yaml

# Process specific document
yq 'select(documentIndex == 0).metadata.name' multi-doc.yaml

# Update all documents
yq -i 'all(select(has("kind")) | .metadata.namespace = "production")' multi-doc.yaml

# Add a new document
yq -i '. as $item ireduce ([]; . + $item) | .[0].metadata.namespace = "new-value"' file.yaml

5.3 Controlling Output Format

YQ provides fine-grained control over output formatting:

# Pretty print with color
yq -C '.' file.yaml

# Output as single line
yq -j '.' file.yaml

# Control indentation
yq -P '.' file.yaml

# Preserve comments
yq --indent=2 '.' file.yaml

5.4 Changing YAML Block Style

YAML offers different styles for representing strings and blocks. You can control these with YQ:

# Convert string to literal block style (|)
yq -i '.solutionDetails style="literal"' question.yaml

# Convert to folded style (>)
yq -i '.description style="folded"' config.yaml

# Apply literal style to all matching fields
yq -i 'eval(.. | select(has("solutionDetails")).solutionDetails) |= style("literal")' questions.yaml

5.5 Controlling Field Order

While YAML is technically an unordered format, YQ can help maintain a specific field order:

# Order specific fields first
yq -i '... | select(has("type")) |= with_entries(select(.key == "type") + select(.key == "id") + select(.key == "problemStatement") + select(.key == "correctAnswer") + select(.key == "answerPrompt") + select(.key == "solutionDetails"))' questions.yaml

6. Common Pitfalls and Solutions

6.1 The Temp File Pattern

Problem: You need to make complex changes that are difficult to express in a single YQ command.

Solution: The temporary file pattern is your friend:

# Create a temporary file, process it, then replace the original
yq 'del(.. | .difficulty?)' input.yml > temp.yml && mv temp.yml input.yml

While using -i is cleaner, this pattern gives you a chance to verify changes before committing them.

6.2 Working with String Formats

Problem: YAML string formatting (quotes vs. block style) doesn't match your needs.

Solution: Use YQ's style modifiers:

# Convert multi-line strings to block literal format (|)
yq -i '.. | select(has("solutionDetails")).solutionDetails style="literal"' questions.yaml

# Convert to double-quoted style
yq -i '.description style="double"' config.yaml

This is particularly useful for making YAML more readable when it contains multi-line text.

6.3 Modifying Nested Arrays

Problem: Updating specific elements in arrays can be tricky.

Solution: Combine select() with array indexing:

# Update the container named "app" in a Kubernetes deployment
yq -i '.spec.containers[] |= select(.name == "app").image = "new-image:v2"' deployment.yaml

6.4 Multiple Document Files

Problem: Processing files with multiple YAML documents (separated by ---).

Solution: Use the all or select functions with document awareness:

# Process only Deployment resources in a multi-document file
yq -i 'select(.kind == "Deployment").spec.replicas = 3' k8s-manifests.yaml

7. Real-World Command Examples

This section provides practical examples that solve common YAML processing needs.

7.1 Extract All Container Images from Kubernetes Manifests

Finding all container images across multiple Kubernetes manifests is a common task:

# List all container images across all deployments
yq '.spec.template.spec.containers[].image' */deployment.yaml

# With better formatting
yq '.spec.template.spec.containers[] | [.name, .image] | join(": ")' */deployment.yaml

7.2 Standardize Field Order in YAML Documents

When you want consistent field ordering (especially for version control clarity):

# Order fields consistently for question files
yq -i '... | select(has("type")) |= with_entries(
  select(.key == "type") + 
  select(.key == "id") + 
  select(.key == "problemStatement") + 
  select(.key == "correctAnswer")
)' questions.yaml

7.3 Convert YAML String Formatting

Changing how multi-line strings are represented:

# Convert all solution details to use the literal block style (|)
yq -i 'eval(.. | select(has("solutionDetails")).solutionDetails) |= style("literal")' questions.yaml

This is incredibly useful for making educational content or documentation more readable.

7.4 Batch Update Values Recursively

Finding and updating values deeply nested in YAML is a common need:

# Removing all difficulty fields from nested YAML structures
yq -i 'del(.. | .difficulty?)' content/all-question-types.yml

7.5 Working with Multiple Document Files

For Kubernetes manifests or other multi-document YAML files:

# Apply namespace to all resources in a file
yq -i 'all(select(has("metadata")) | .metadata.namespace = "production")' k8s-manifests.yaml

# Update only resources of a specific type
yq -i 'select(.kind == "Deployment" or .kind == "StatefulSet").spec.replicas = 3' k8s-manifests.yaml

8. YQ Command Cheat Sheet

Here's a quick reference guide to the most essential YQ commands you'll use daily:

9. Conclusion

YQ has become an essential tool for anyone who works with YAML files regularly. Its JQ-like syntax provides a powerful way to query, modify, and transform YAML documents directly from the command line.

The most valuable aspects of YQ are:

1. In-place editing with the -i flag, which eliminates the need for temporary files in many cases
2. Recursive processing with the .. operator for handling deeply nested structures
3. Multi-document support for working with complex configuration files
4. Format control for ensuring your YAML looks exactly how you want it

While YQ has many advanced features, mastering just the core commands covered in this guide will handle 90% of your YAML processing needs. Start with simple reads and updates, then gradually incorporate more complex operations as you become comfortable with the syntax.

Remember that YQ is a living project with regular updates. Check the official documentation for the latest capabilities and syntax.

Whether you're managing Kubernetes manifests, CI/CD configurations, or any other YAML-based files, YQ's powerful yet concise operations will dramatically improve your efficiency when working with structured configuration data.