Productions

Productions

Concept Overview

Productions in AI Task represent a higher-level organizational structure that groups related pipeline executions into a cohesive unit. Similar to theatrical or musical productions, an AI Task production encompasses multiple “performances” (instances) of related task pipelines that follow a consistent pattern but may process different inputs.

New Directory Structure

A production now follows a standardized directory structure:

project-home/
├── productions/
│   ├── production_name/
│   │   ├── production_name_00001/
│   │   │   ├── step1/
│   │   │   │   └── document_00001_type1.txt
│   │   │   ├── step2/
│   │   │   │   └── document_00001_type2.json
│   │   │   └── step3/
│   │   │       └── document_00001_type3.pdf
│   │   ├── production_name_00002/
│   │   │   ├── step1/
│   │   │   ├── step2/
│   │   │   └── step3/
│   │   └── settings/
│   │       ├── production_config.yml
│   │       └── production_db.csv
│   └── another_production/
│       ├── another_production_00001/
│       └── settings/

Key Components

  1. Project Home: Top-level container for all AI Task project data
  2. Productions Directory: Central location for all productions
  3. Production Directory: Named after the production (e.g., profiles)
  4. Instance Directories: Individual execution instances with unique IDs (e.g., profile_00001)
  5. Step Directories: Organize documents by processing stage within each instance
  6. Document Files: Follow a standardized pattern (document_[ID]_[TYPE].[SUFFIX])
  7. Settings Directory: Contains metadata, diagnostics, and configuration for the production

Production Configuration

Productions are configured using a YAML file that defines their structure and behavior:

# Example production configuration
name: "profiles"
project_home: "./project-home"
production_dir: "productions"
id_format: "{name}_{id:05d}"
document_pattern: "document_{id}_{type}.{suffix}"
settings_dir: "settings"

# Define the steps (processing stages) for this production
steps:
  - "input"     # Initial data
  - "analysis"  # Analysis results
  - "summary"   # Final summaries
  - "export"    # Exportable content

# Additional production-specific settings
metadata:
  description: "Customer profile analysis production"
  version: "1.0"
  owner: "Marketing Team"

Configuration Options

Option Description Default
name Production name required
project_home Base directory for all projects ./project-home
production_dir Directory name for all productions productions
id_format Format for instance directory names {name}_{id:05d}
document_pattern Pattern for document filenames document_{id}_{type}.{suffix}
settings_dir Directory for production metadata settings
steps List of processing stages ['input', 'process', 'output']
metadata Additional descriptive information {}

ProductionManager API

The new ProductionManager class provides a comprehensive API for working with productions:

from ai_task import ProductionManager

# Create from configuration file
pm = ProductionManager("production_config.yml")

# Or create with dictionary
pm = ProductionManager(config_dict={
    "name": "profiles",
    "project_home": "./project-home",
    "steps": ["input", "analysis", "output"]
})

# Initialize directory structure
pm.initialize_structure()

# Create a new instance
instance_id = pm.create_instance(metadata={"source": "API"})

# Save a document
pm.save_document(
    instance_id=instance_id,
    document_type="raw",
    content="Sample content",
    suffix="txt",
    step="input"
)

# Find documents by type
documents = pm.find_documents(document_type="summary")

# Update database from filesystem
pm.update_productions()

# List all instances
instances_df = pm.list_instances()

Convenience Functions

AI Task also provides high-level convenience functions:

from ai_task import create_production, load_production, run_in_production

# Create a new production
pm = create_production("production_config.yml")

# Load existing production
pm = load_production("production_config.yml")

# Run pipeline in production
result, instance_id = run_in_production(
    pipeline_file="analyze.ai",
    production_config="production_config.yml",
    input_data="Sample input",
    document_type="analyzed",
    step="analysis"
)

Command-Line Interface

The command-line interface has been enhanced to support productions:

# Initialize a production
aitask --production-config production_config.yml --production-setup

# Create a new instance and run a pipeline
aitask analyze.ai "Input text" --production-config production_config.yml --production-new --document-type analyzed

# Run in existing instance
aitask analyze.ai "Input text" --production-config production_config.yml --production-instance 00001 --document-type analyzed --step analysis

# List all instances
aitask --production-config production_config.yml --production-list

Production Database

Each production maintains a database of its instances in a CSV file (production_db.csv) with the following columns:

Column Description
instance_id Unique ID for the instance
name Full instance name
created_at Creation timestamp
updated_at Last update timestamp
status Instance status (created, active, completed, etc.)
metadata Additional instance information
path Path to instance directory

This database is automatically maintained by the ProductionManager and can be queried using the list_instances() method.

Integration with Pipelines

Pipelines can be configured to work with a specific production by including production information in the pipeline configuration:

name: "Analysis Pipeline"
description: "Analyzes input data"

# Production configuration
production:
  name: "profiles"
  project_home: "./project-home"
  production_dir: "productions"

pipe:
  - type: llm
    name: "analyzer"
    tmpl: |
      Analyze the following data:
      
      {{pipe.pipein-text}}
    model: "gemini-2.0-flash"
    # Output can reference production context
    output_file: "{{production.instance_dir}}/analysis/results.txt"

Best Practices

  1. Consistent naming: Use clear, descriptive names for productions and document types
  2. Step organization: Define logical steps that represent the processing workflow
  3. Document types: Create a standardized set of document types for each production
  4. Metadata: Use metadata to track the source and purpose of each instance
  5. Regular updates: Call update_productions() to keep the database in sync with the filesystem
  6. Configuration management: Store production configurations in version control