diff --git a/AGENTS.md b/AGENTS.md index 941ee6e5..130b6b1d 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,725 +1,63 @@ # AGENTS.md -Guidance for AI Agents working with this repository. +Guidance for AI coding agents working in this repository. -## Project Overview +## Project -A Python CLI for managing infrastructure boilerplates, plus archetypes and template-loading logic. Built with Typer (CLI) and Jinja2 (templating). +Project name: Boilerplates CLI -**Important library location note:** -- The canonical template library does **not** live in this repository for modern development. -- The checked-in `library/` directory here is **legacy/backward-compatibility content for versions older than `0.2.0`**. -- For `0.2.0+`, the active template library lives in the separate GitHub repository `boilerplates-library`. +A Python CLI for managing infrastructure boilerplates, template libraries, template rendering, and validation. Built with Typer and Jinja2. -## Development Setup +## Scope and precedence -### Running and Testing +- This file applies to the repository tree rooted at the directory containing this `AGENTS.md` file. +- More deeply nested `AGENTS.md` files may add or override instructions for their subtrees. +- Before GitHub-facing work such as issues, branches, PRs, merges, tags, or releases, read this file and the relevant docs listed below. +- For release-related work, always read `RELEASE.md` first. -```bash -# Create and activate a local virtual environment first -python3 -m venv .venv -source .venv/bin/activate -python3 -m pip install --upgrade pip -python3 -m pip install -e ".[test]" +## Repository map -# Run the CLI application -python3 -m cli -# Debugging and Testing commands -python3 -m cli --log-level DEBUG compose list -# Run tests -python3 -m pytest -``` +- Source: `cli/` - main CLI application and core runtime code. +- Tests: `tests/` - automated tests, fixtures, and test utilities. +- Contribution guide: `CONTRIBUTING.md` - setup, validation, GitHub workflow, and command overview. +- Docs: `docs/` - developer documentation and architecture notes. +- Release process: `RELEASE.md` - release PR, version, changelog, tag, and publish workflow. +- Scripts: `scripts/` - installer and helper scripts. +- GitHub automation: `.github/` - workflows, issue/PR templates, and repository scripts. +- Legacy library: `library/` - backward-compatibility template content for versions older than `0.2.0`; not the canonical modern template library. -### Production-Ready Testing +## Documentation -For detailed information about testing boilerplates in a production-like environment before release, see **WARP-LOCAL.md** (local file, not in git). This document covers: -- Test server infrastructure and Docker contexts -- Step-by-step testing procedures for Docker Compose, Swarm, and Kubernetes -- Comprehensive testing checklists -- Production release criteria +Project documentation lives in `docs/`. -### Linting and Formatting +Agents should read the relevant docs before making changes and update docs when behavior, configuration, or workflows change. -Should **always** happen before pushing anything to the repository and before making a release. +### Documentation map -- Use `ruff` for Python code: - - `ruff check --fix .` - Check and auto-fix linting errors (including unused imports) - - `ruff format .` - Format code according to style guidelines - - Both commands must be run before committing -- Minimum pre-release verification: - - `ruff check --fix .` - - `ruff format .` - - `python3 -m pytest` -- Agents should treat these as release gates, not optional cleanup. - -### Project Management and Git - -The project is stored in a public GitHub Repository, use issues, and branches for features/bugfixes and open PRs for merging. - -**Naming Conventions and Best-Practices:** -- Branches, PRs: `feature/2314-add-feature`, `problem/1249-fix-bug` -- Issues should have clear titles and descriptions, link related issues/PRs, and have appropriate labels like (problem, feature, discussion, question). -- Commit messages should be clear and concise, following the format: `type(scope): subject` (e.g., `fix(compose): correct variable parsing`). - -## Architecture - -### File Structure - -- `cli/` - Python CLI application source code - - `cli/core/` - Core Components of the CLI application - - `cli/modules/` - Modules implementing technology-specific functions - - `cli/__main__.py` - CLI entry point, auto-discovers modules and registers commands -- `library/` - Legacy in-repo template content kept only for backward compatibility with versions older than `0.2.0` - - Do not treat this directory as the canonical source for new template work - - The active `0.2.0+` library is maintained in the separate GitHub repository `boilerplates-library` -- `archetypes/` - Testing tool for template snippets (archetype development) - - `archetypes/__init__.py` - Package initialization - - `archetypes/__main__.py` - CLI tool entry point - - `archetypes//` - Module-specific archetype snippets (e.g., `archetypes/compose/`) - -### Core Components - -- `cli/core/collection.py` - VariableCollection class (manages sections and variables) - - **Key Attributes**: `_sections` (dict of VariableSection objects), `_variable_map` (flat lookup dict) - - **Key Methods**: `get_satisfied_values()` (returns enabled variables), `apply_defaults()`, `sort_sections()` -- `cli/core/config.py` - Configuration management (loading, saving, validation) -- `cli/core/display/` - Centralized CLI output rendering package (**Always use DisplayManager - never print directly**) - - `__init__.py` - Package exports (DisplayManager, DisplaySettings, IconManager) - - `display_manager.py` - Main DisplayManager facade - - `display_settings.py` - DisplaySettings configuration class - - `icon_manager.py` - IconManager for Nerd Font icons - - `variable_display.py` - VariableDisplayManager for variable rendering - - `template_display.py` - TemplateDisplayManager for template display - - `status_display.py` - StatusDisplayManager for status messages - - `table_display.py` - TableDisplayManager for table rendering -- `cli/core/exceptions.py` - Custom exceptions for error handling (**Always use this for raising errors**) -- `cli/core/library.py` - LibraryManager for template discovery from git-synced libraries and static file paths -- `cli/core/module.py` - Abstract base class for modules (defines standard commands) -- `cli/core/prompt.py` - Interactive CLI prompts using rich library -- `cli/core/registry.py` - Central registry for module classes (auto-discovers modules) -- `cli/core/repo.py` - Repository management for syncing git-based template libraries -- `cli/core/section.py` - VariableSection class (stores section metadata and variables) - - **Key Attributes**: `key`, `title`, `toggle`, `needs`, `variables` (dict of Variable objects) -- `cli/core/template.py` - Template Class for parsing, managing and rendering templates -- `cli/core/variable.py` - Variable class (stores variable metadata and values) - - **Key Attributes**: `name`, `type`, `value` (stores default or current value), `description`, `sensitive`, `needs` - - **Note**: Default values are stored in `value` attribute, NOT in a separate `default` attribute -- `cli/core/validators.py` - Semantic validators for template content (Docker Compose, YAML, etc.) -- `cli/core/version.py` - Version comparison utilities for semantic versioning - -### Modules - -**Currently supported kinds in code:** -- `ansible` -- `compose` -- `helm` -- `kubernetes` -- `packer` -- `static` -- `swarm` -- `terraform` - -These are the kinds supported by the registered CLI modules in `cli/modules/`. When answering "what kinds do we support?", prefer this code-level list over whatever legacy content happens to exist under `library/`. - -**Module Structure:** -Modules can be either single files or packages: -- **Single file**: `cli/modules/modulename.py` (for simple modules) - -**Creating Modules:** -- Subclass `Module` from `cli/core/module.py` -- Call `registry.register(YourModule)` at module bottom -- Auto-discovered and registered at CLI startup - -**Module Discovery and Registration:** - -The system automatically discovers and registers modules at startup: - -1. **Discovery**: CLI `__main__.py` imports all Python files in `cli/modules/` directory -2. **Registration**: Each module file calls `registry.register(ModuleClass)` at module level -3. **Storage**: Registry stores module classes in a central dictionary by module name -4. **Command Generation**: CLI framework auto-generates subcommands for each registered module -5. **Instantiation**: Modules are instantiated on-demand when commands are invoked - -**Benefits:** -- No manual registration needed - just add a file to `cli/modules/` -- Modules are self-contained - can be added/removed without modifying core code -- Type-safe - registry validates module interfaces at registration time - - - - -``` - compose/ - v1.0.json - v1.1.json - v1.2.json - terraform/ - v1.0.json - ansible/ - v1.0.json - ...other modules... -``` - - - -```json -[ - { - "key": "section_key", - "title": "Section Title", - "description": "Optional section description", - "toggle": "optional_toggle_variable_name", - "needs": "optional_dependency", - "required": true, - "vars": [ - { - "name": "variable_name", - "type": "str", - "description": "Variable description", - "default": "default_value", - "required": true, - "sensitive": false, - "autogenerated": false, - "options": ["option1", "option2"], - "needs": "other_var=value", - "extra": "Additional help text" - } - ] - } -] -``` - - - -```python - -class MyModule(Module): - name = "mymodule" - - # Convert JSON array to OrderedDict format - return self._convert_json_to_dict(json_spec) -``` - -- **Auto-created toggle variables**: Sections with `toggle` automatically create boolean variables -- **Conditional visibility**: Variables use `needs` constraints to show/hide based on other variable values -- **Mode-based organization**: Related settings grouped by operational mode (e.g., network_mode, volume_mode) - - - -Older Python-based `spec_v*.py` files have been migrated to JSON. The module `__init__.py` now: -2. Converts JSON array format to OrderedDict for backward compatibility - -**Existing Modules:** -- Other modules (ansible, terraform, kubernetes, helm, packer) - Work in Progress - -**(Work in Progress):** terraform, docker, ansible, kubernetes, packer modules - -### LibraryManager - -- Loads libraries from config file -- Stores Git Libraries under: `~/.config/boilerplates/libraries/{name}/` -- Uses sparse-checkout to clone only template directories for git-based libraries (avoiding unnecessary files) -- Supports two library types: **git** (synced from repos) and **static** (local directories) -- Priority determined by config order (first = highest) -- In this repository, the checked-in `library/` directory is legacy compatibility data only and should not be treated as the primary source for current template development -- For current `0.2.0+` template work, use the external GitHub repository `boilerplates-library` - -**Library Types:** -- `git`: Requires `url`, `branch`, `directory` fields -- `static`: Requires `path` field (absolute or relative to config) - -**Duplicate Handling:** -- Within same library: Raises `DuplicateTemplateError` -- Across libraries: Uses qualified IDs (e.g., `alloy.default`, `alloy.local`) -- Simple IDs use priority: `compose show alloy` loads from first library -- Qualified IDs target specific library: `compose show alloy.local` - -**Config Example:** -```yaml -libraries: - - name: default # Highest priority (checked first) - type: git - url: https://github.com/user/templates.git - branch: main - directory: library - - name: local # Lower priority - type: static - path: ~/my-templates - url: '' # Backward compatibility fields - branch: main - directory: . -``` - -**Note:** Static libraries include dummy `url`/`branch`/`directory` fields for backward compatibility with older CLI versions. - -### ConfigManager - -- User Config stored in `~/.config/boilerplates/config.yaml` - -### DisplayManager and IconManager - -**CRITICAL RULE - NEVER violate this:** -- NEVER use `console.print()` outside of display manager classes (`cli/core/display/` directory) -- NEVER import `Console` from `rich.console` except in display manager classes or `cli/__main__.py` -- ALWAYS use `module_instance.display.display_*()` or `display.display_*()` methods for ALL output -- Display managers (`cli/core/display/*.py`) are the ONLY exception - they implement console output - -**Rationale:** -- `DisplayManager` provides a **centralized interface** for ALL CLI output rendering -- Direct console usage bypasses formatting standards, icon management, and output consistency -- `IconManager` provides **Nerd Font icons** internally for DisplayManager - never use emojis or direct icons - -**DisplayManager Architecture** (Refactored for Single Responsibility Principle): - -`DisplayManager` acts as a facade that delegates to specialized manager classes: - -1. **VariableDisplayManager** - Handles all variable-related rendering - - `render_variable_value()` - Variable value formatting with context awareness - - `render_section()` - Section header display - - `render_variables_table()` - Complete variables table with dependencies - -2. **TemplateDisplayManager** - Handles all template-related rendering - - `render_template()` - Main template display coordinator - - `render_template_header()` - Template metadata display - - `render_file_tree()` - Template file structure visualization - - `render_file_generation_confirmation()` - Files preview before generation - -3. **StatusDisplayManager** - Handles status messages and error display - - `display_message()` - Core message formatting with level-based routing - - `display_error()`, `display_warning()`, `display_success()`, `display_info()` - Convenience methods - - `display_template_render_error()` - Detailed render error display - - `display_warning_with_confirmation()` - Interactive warning prompts - -4. **TableDisplayManager** - Handles table rendering - - `render_templates_table()` - Templates list with library indicators - - `render_status_table()` - Status tables with success/error indicators - - `render_config_tree()` - Configuration tree visualization - -**Usage Pattern:** -```python -# External code uses DisplayManager methods (backward compatible) -display = DisplayManager() -display.display_template(template, template_id) - -# Internally, DisplayManager delegates to specialized managers -# display.templates.render_template(template, template_id) -``` - -**Design Principles:** -- External code calls `DisplayManager` methods only -- `DisplayManager` delegates to specialized managers internally -- Each specialized manager has a single, focused responsibility -- Backward compatibility maintained through delegation methods -- All managers can access parent DisplayManager via `self.parent` - -## Templates - -Templates are directory-based. Each template is a directory containing all the necessary files and subdirectories for the boilerplate. - -### Template Rendering Flow - -**How templates are loaded and rendered:** - -1. **Discovery**: LibraryManager finds template directories containing `template.json` -2. **Parsing**: Template class loads and parses the template metadata and spec -5. **Variable Merging**: Template spec overrides are merged with module spec (precedence: module < template < user config < CLI) -6. **Collection Building**: VariableCollection is constructed with merged variables and sections -7. **Dependency Resolution**: Sections are topologically sorted based on `needs` constraints -8. **Variable Resolution**: Variables with `needs` constraints are evaluated for visibility -9. **Jinja2 Rendering**: Template files (`.j2`) are rendered with final variable values -10. **Sanitization**: Rendered output is cleaned (whitespace, blank lines, trailing newline) -11. **Validation**: Optional semantic validation (YAML structure, Docker Compose config, etc.) - -**Key Architecture Points:** -- Variable visibility is dynamic based on `needs` constraints (evaluated at prompt/render time) -- Jinja2 templates support `{% include %}` and `{% import %}` for composition -- Template loading can read configured libraries from disk, but the checked-in `library/` tree in this repo is legacy compatibility data, not the canonical `0.2.0+` library source - -### Template Structure - -Requires `template.json` with metadata and variables: - -```json -{ - "slug": "my-service", - "kind": "compose", - "metadata": { - "name": "My Service Template", - "description": "A template for a service.", - "author": "Your Name", - "date": "2024-01-01", - "version": { - "name": "v1.0.0" - } - }, - "variables": [ - { - "name": "general", - "title": "General", - "items": [ - { - "name": "service_name", - "type": "str", - "title": "Service name" - } - ] - } - ] -} -``` - -### Template Metadata Versioning - -**Template Version Field:** -The `metadata.version` field in `template.json` is a structured object and may be partial: -- `name` -- `source_dep_name` -- `source_dep_version` -- `source_dep_digest` -- `upstream_ref` -- `notes` - -**Rationale:** This separates the user-facing template version label from the tracked upstream dependency metadata used by the template snapshot. - -**Application Version Variables:** -- **IMPORTANT**: Application/image versions should be **hardcoded** in template files (e.g., `image: nginx:1.25.3`) -- Do NOT create template variables for application versions (e.g., no `nginx_version` variable) -- Users should update the template file directly when they need a different version -- This prevents version mismatches and ensures templates are tested with specific, known versions -- Exception: Only create version variables if there's a strong technical reason (e.g., multi-component version pinning) - - - - -```json -{ - "slug": "my-template", - "kind": "compose", - "metadata": { - "name": "My Template", - "version": { - "name": "v1.0.0" - } - }, - "variables": [ - { - "name": "general", - "title": "General", - "items": [] - } - ] -} -``` - -**How It Works:** - -**Behavior:** -- Older templates work with newer module versions (backward compatibility) -- Version comparison uses MAJOR.MINOR format (e.g., "1.0" < "1.2" < "2.0") - -**When to Use:** - -**Single-File Module Example:** -```python -class SimpleModule(Module): - name = "simple" - description = "Simple module" - spec = VariableCollection.from_dict({...}) # Single spec -``` - -```python -# cli/modules/modulename/__init__.py -class ExampleModule(Module): - name = "modulename" - description = "Module description" - - # Dynamically load the appropriate spec version - spec_module = importlib.import_module(f".{version_file}", package=__package__) - return spec_module.get_spec() -``` - -**Version Management:** -- CLI version is defined in `cli/__init__.py` as `__version__` -- pyproject.toml version must match `__version__` for releases -- GitHub release workflow validates version consistency - -### Template Files - -- **Jinja2 Templates (`.j2`)**: Rendered by Jinja2, `.j2` extension removed in output. Support `{% include %}` and `{% import %}`. -- **Static Files**: Non-`.j2` files copied as-is. -- **Sanitization**: Auto-sanitized (single blank lines, no leading blanks, trimmed whitespace, single trailing newline). -- **Shortcodes**: Template descriptions support emoji-style shortcodes (e.g., `:warning:`, `:info:`, `:docker:`) which are automatically replaced with Nerd Font icons during display. Add new shortcodes to `IconManager.SHORTCODES` dict. - -### Docker Compose Best Practices - -**Traefik Integration:** - -When using Traefik with Docker Compose, the `traefik.docker.network` label is **CRITICAL** for stacks with multiple networks. When containers are connected to multiple networks, Traefik must know which network to use for routing. - -**Implementation:** -- Review `archetypes/compose/` directory for reference implementations of Traefik integration patterns -- The `traefik.docker.network={{ traefik_network }}` label must be present in both standard `labels:` and `deploy.labels:` sections -- Standard mode and Swarm mode require different label configurations - check archetypes for examples - -### Variables - -**How Templates Inherit Variables:** - - -**When to Define Template Variables:** - -You only need to define variables in your template's `spec` section when: -1. **Overriding defaults**: Change default values for module variables (e.g., hardcode `service_name` for your specific app) - -**Variable Precedence** (lowest to highest): -1. Module `spec` (defaults for all templates of that kind) -2. Template `spec` (overrides module defaults) -3. User `config.yaml` (overrides template and module defaults) -4. CLI `--var` (highest priority) - -**Template Variable Override Rules:** -- **Override module defaults**: Only specify properties that differ from module spec (e.g., change `default` value) -- **Create new variables**: Define template-specific variables not in module spec -- **Minimize duplication**: Do NOT re-specify `type`, `description`, or other properties if they remain unchanged from module spec - -**Example:** -```yaml -# Template spec ONLY needs to override specific defaults: -spec: - general: - vars: - service_name: - default: whoami # Only override the default, type already defined in module -``` - -**Variable Types:** -- `str` (default), `int`, `float`, `bool` -- `email` - Email validation with regex -- `url` - URL validation (requires scheme and host) -- `hostname` - Hostname/domain validation -- `enum` - Choice from `options` list - -**Variable Properties:** -- `sensitive: true` - Masked in prompts/display (e.g., passwords) -- `autogenerated: true` - Auto-generates value if empty (shows `*auto` placeholder) -- `default` - Default value -- `description` - Variable description -- `prompt` - Custom prompt text (overrides description) -- `extra` - Additional help text -- `options` - List of valid values (for enum type) - -**Section Features:** -- **Toggle Settings**: Conditional sections via `toggle: "bool_var_name"`. If false, section is skipped. - - **IMPORTANT**: When a section has `toggle: "var_name"`, that boolean variable is AUTO-CREATED by the system - - Example: `ports` section with `toggle: "ports_enabled"` automatically provides `ports_enabled` boolean -- **Dependencies**: Use `needs: "section_name"` or `needs: ["sec1", "sec2"]`. Dependent sections only shown when dependencies are enabled. - -**Dependency Resolution Architecture:** - -Sections and variables support `needs` constraints to control visibility based on other variables. - -**Section-Level Dependencies:** -- Format: `needs: "section_name"` or `needs: ["sec1", "sec2"]` -- Section only appears when all required sections are enabled (their toggle variables are true) -- Automatically validated: detects circular, missing, and self-dependencies -- Topologically sorted: ensures dependencies are prompted/processed before dependents - -**Variable-Level Dependencies:** -- Format: `needs: "var_name=value"` or `needs: "var1=val1;var2=val2"` (semicolon-separated) -- Variable only visible when constraint is satisfied (e.g., `needs: "network_mode=bridge"`) -- Supports multiple values: `needs: "network_mode=bridge,macvlan"` (comma = OR) -- Evaluated dynamically at prompt and render time - -**Validation:** -- Circular dependencies: Raises error if A needs B and B needs A -- Missing dependencies: Raises error if referencing non-existent sections/variables -- Self-dependencies: Raises error if section depends on itself - -**Example Section with Dependencies:** - -```yaml -spec: - traefik: - title: Traefik - required: false - toggle: traefik_enabled - vars: - traefik_enabled: - type: bool - default: false - traefik_host: - type: hostname - - traefik_tls: - title: Traefik TLS/SSL - needs: traefik - toggle: traefik_tls_enabled - vars: - traefik_tls_enabled: - type: bool - default: true - traefik_tls_certresolver: - type: str - sensitive: false - default: myresolver -``` +- `docs/architecture.md` - repository layout, module system, and core runtime components +- `docs/templates.md` - template format, variables, rendering, and validation behavior +- `docs/libraries.md` - library configuration and canonical external template library guidance +- `docs/display.md` - DisplayManager and CLI output rules ## Validation -**Jinja2 Validation:** -- Templates validated for Jinja2 syntax errors during load -- Checks for undefined variables (variables used but not declared in spec) -- Built into Template class - -**Semantic Validation:** -- Validator registry system in `cli/core/validators.py` -- Extensible: `ContentValidator` abstract base class -- Built-in validators: `DockerComposeValidator`, `YAMLValidator` -- Validates rendered output (YAML structure, Docker Compose config, etc.) -- Triggered via `compose validate` command with `--semantic` flag (enabled by default) - -## Prompt - -Uses `rich` library for interactive prompts. Supports: -- Text input -- Password input (masked, for `sensitive: true` variables) -- Selection from list (single/multiple) -- Confirmation (yes/no) -- Default values -- Autogenerated variables (show `*auto` placeholder, generate on render) - -To skip the prompt use the `--no-interactive` flag, which will use defaults or empty values. - -## Commands - -**Standard Module Commands** (auto-registered for all modules): -- `list` - List all templates -- `search ` - Search templates by ID -- `show ` - Show template details -- `generate -o ` - Generate from template (supports `--dry-run`, `--var`, `--no-interactive`) -- `validate [template_id]` - Validate template(s) (Jinja2 + semantic). Omit template_id to validate all templates -- `defaults` - Manage config defaults (`get`, `set`, `rm`, `clear`, `list`) - -**Core Commands:** -- `repo sync` - Sync git-based libraries -- `repo list` - List configured libraries - -## Archetypes - -The `archetypes` package provides reusable, standardized template building blocks for creating boilerplates. Archetypes are modular Jinja2 snippets that represent specific configuration sections. - -### Purpose - -1. **Template Development**: Provide standardized, tested building blocks for creating new templates -2. **Testing & Validation**: Enable testing of specific configuration sections in isolation with different variable combinations - -### Usage +Default checks before finishing code changes: ```bash -# List available archetypes for a module -python3 -m archetypes compose list - -# Preview an archetype component -python3 -m archetypes compose generate - -# Test with variable overrides -python3 -m archetypes compose generate \ - --var traefik_enabled=true \ - --var swarm_enabled=true - -# Validate templates against archetypes -python3 -m archetypes compose validate # All templates -python3 -m archetypes compose validate