Data Hub

Overview

The Data Hub action manages centralized data storage and retrieval within workflows. It allows you to store data from workflow steps, share data across workflows, and retrieve stored data for use in subsequent operations.

🗄️ Purpose

Use Data Hub to:

Store workflow outputs persistently
Share data across multiple workflows
Retrieve historical data for verification
Build data-driven workflows
Create shared datasets for teams

When to Use Data Hub

✅ Good Use Cases

Scenario

Reason

Persist Workflow Data

Store IDs/values for later use

Cross-Workflow Data

Share data between workflows

Historical Data

Compare with previous runs

Reference Data

Store test configuration data

Shared Test Data

Common data for multiple tests

Dynamic Test Data

Generate and store runtime data

Audit Trail

Track what changed and when

❌ Anti-Patterns (Don't Do)

❌ Store sensitive data (passwords, tokens)
❌ Store very large datasets
❌ Use as temporary variables (use session instead)
❌ Duplicate data unnecessarily
❌ Store without clear naming

Configuration

Basic Setup

Open your workflow
Click "Add Action" → Select "Data Hub"
Configure operation:
- Operation: Store, Retrieve, or Update
- Data Key: Identifier for data
- Data Value: What to store
- Scope: Private or Shared
Optional: Set expiration (optional)
Save

Configuration Options

Data Hub Action Configuration
├── Operation:
│   ├── Store: Save data
│   ├── Retrieve: Get data
│   ├── Update: Modify data
│   └── Delete: Remove data
├── Data Identifier:
│   ├── Key: Unique name
│   └── Version: Retrieve specific version
├── Data:
│   ├── Value: Data to store
│   ├── Type: JSON, String, Number, Boolean
│   └── Format: Structured or plain
├── Scope:
│   ├── Private: Only this workflow
│   └── Shared: Across workflows
└── Management:
    ├── Expiration: Auto-delete after time
    └── Tags: Categorize data

Data Operations

Store Operation

Save data to Data Hub:

Store Configuration:
├── Key: "user_123_profile"
├── Value: {
│   "user_id": "123",
│   "email": "[email protected]",
│   "created_at": "2024-01-15"
│ }
├── Scope: Shared
└── Expiration: 30 days

Use when:

First time saving this data
Overwriting existing data
Creating new reference data

Retrieve Operation

Fetch data from Data Hub:

Retrieve Configuration:
├── Key: "user_123_profile"
├── Scope: Shared
└── Version: Latest (or specific version)

Result:
├── Found: true/false
├── Data: {stored object}
├── Metadata: {
│   "created_at": "2024-01-15",
│   "last_updated": "2024-01-15",
│   "times_retrieved": 5
│ }
└── Status: "success"

Use when:

Need to access previously stored data
Comparing current vs previous values
Reusing reference data

Update Operation

Modify existing stored data:

Update Configuration:
├── Key: "user_123_profile"
├── Updates: {
│   "last_login": "2024-01-15 14:30",
│   "login_count": 42
│ }
├── Merge: true (preserve existing fields)
└── Scope: Shared

Use when:

Incrementing counters
Adding new fields
Updating timestamps

Delete Operation

Remove data from Data Hub:

Delete Configuration:
├── Key: "user_123_profile"
├── Scope: Shared
├── Confirm: true
└── Archive: Optional backup

Use when:

Cleaning up test data
Removing sensitive data
Archiving old data

Practical Examples

Example 1: Store User ID for Later Use

Workflow: Create and Verify User

Step 1: API - Create User
   Output: {
      "id": "usr_789",
      "email": "[email protected]"
   }

Step 2: Data Hub - Store
   Key: "test_user_id"
   Value: "usr_789"
   Scope: Shared
   Expiration: 7 days
   
Step 3: API - Send Welcome Email
   To: [email protected]
   
[Another workflow uses same user...]

Step 4: Data Hub - Retrieve
   Key: "test_user_id"
   
Step 5: API - Verify User Email
   User ID: [Retrieved from Data Hub]

Example 2: Track Test Metrics

Workflow: Performance Monitoring

Step 1: API - Get Performance Metrics
   Output: {
      "response_time": 245,
      "throughput": 1250
   }

Step 2: Data Hub - Store
   Key: "performance_baseline"
   Value: {metrics}
   Scope: Shared
   
Step 3: Wait metrics to accumulate

[Later workflow compares...]

Step 4: Data Hub - Retrieve
   Key: "performance_baseline"

Step 5: API - Get Current Metrics
   
Step 6: Compare
   IF current_response_time > baseline THEN
      [Alert: Performance degraded]

Workflow A: Setup Test Data

Step 1: API - Create Test Config
   Output: {
      "config_id": "cfg_123",
      "endpoints": [...],
      "credentials": {...}
   }

Step 2: Data Hub - Store
   Key: "shared_test_config"
   Value: config_object
   Scope: Shared
   
---

Workflow B: Use Shared Config

Step 1: Data Hub - Retrieve
   Key: "shared_test_config"

Step 2: Parse Retrieved Config
   Extract: endpoints, credentials
   
Step 3: API - Run Tests
   Using: Retrieved config
   
Step 4: Results: Tests use shared setup

Example 4: Versioned Data Management

Workflow: Track Configuration Changes

Step 1: API - Fetch Current Config
   Output: config_v1

Step 2: Data Hub - Store
   Key: "app_config"
   Value: config_v1
   Version: 1
   
[Configuration changes over time...]

Step 3: API - Fetch Updated Config
   Output: config_v2

Step 4: Data Hub - Update
   Key: "app_config"
   Value: config_v2
   Version: 2
   
Step 5: Data Hub - Retrieve
   Key: "app_config"
   Version: 1 (or 2)
   
Result: Compare V1 vs V2

Example 5: Counter Tracking

Workflow: Track Request Count

Step 1: Data Hub - Retrieve
   Key: "api_request_counter"
   Output: current_count

Step 2: Increment: current_count + 1

Step 3: Data Hub - Update
   Key: "api_request_counter"
   Value: incremented_count
   
Step 4: IF incremented_count > 1000 THEN
   [Alert: High usage]

Data Storage Best Practices

✅ Do

Use clear key names - Descriptive, organized naming
Store structured data - Use JSON objects for complex data
Include metadata - Track what data is and why stored
Set expiration - Auto-cleanup old data
Document data schema - Explain data structure
Version important data - Track changes over time
Use appropriate scope - Private vs Shared based on need
Encrypt sensitive data - If storing sensitive info
Tag related data - Organize with tags/categories

❌ Don't

Store passwords - Security risk
Store API keys - Should be in secrets manager
Store extremely large objects - Memory/performance impact
Store without naming convention - Leads to confusion
Forget to clean up - Old data accumulates
Share sensitive data - Use private scope
Store temporary values - Use workflow variables instead
Store duplicate data - Consolidate to single storage
Ignore expiration - Set TTL to auto-cleanup

Naming Conventions

Recommended Naming Pattern

{workflow_context}_{data_type}_{identifier}

Examples:
- "checkout_order_id_12345"
- "login_session_token_abc123"
- "performance_metrics_baseline"
- "user_config_standard_v1"
- "shared_test_data_credentials"

User Profile Data:
- "user_profile_uid_123"
- "user_preferences_uid_123"
- "user_history_uid_123"

Performance Data:
- "perf_baseline_get_users"
- "perf_baseline_search_orders"
- "perf_baseline_create_account"

Configuration:
- "config_api_endpoints_prod"
- "config_timeout_values"
- "config_retry_settings"

Working with Data Types

JSON Objects

Key: "user_profile"
Value: {
  "id": "usr_123",
  "email": "[email protected]",
  "created": "2024-01-15",
  "metadata": {
    "plan": "premium",
    "verified": true
  }
}

Arrays

Key: "test_ids"
Value: [
  "test_001",
  "test_002",
  "test_003"
]

Primitives

Key: "counter"
Value: 42

Key: "status"
Value: "active"

Key: "enabled"
Value: true

Data Lifecycle

Create → Store → Retrieve → Update → Archive → Delete

Timeline Example:

Day 1: Create Data
   [Generate test configuration]
   [Store in Data Hub]

Day 2: Use Data
   [Retrieve configuration]
   [Use in multiple workflows]

Day 5: Update Data
   [Modify configuration based on feedback]
   [Update in Data Hub]

Day 15: Review Data
   [Analyze usage patterns]
   [Archive if no longer needed]

Day 30: Cleanup
   [Auto-delete via expiration]
   OR
   [Manual deletion]

Performance Considerations

Storage Limits

Typical Data Hub Limits:
├── Single Entry Size: 1-10 MB
├── Total Storage: 100 MB - 1 GB
├── Retrieval Speed: <100ms
└── Concurrent Operations: 100+

Optimization Tips

✅ Compress large data before storing
✅ Use efficient data structures (nested JSON)
✅ Retrieve only needed fields
✅ Archive old data regularly
✅ Use versioning for historical data

Troubleshooting

Issue: Data not found on retrieval

Symptoms:

Retrieve returns "not found"
Data should be there

Causes:

Wrong key name
Data expired and deleted
Wrong scope (private vs shared)
Typo in key

Solutions:

Verify key name exactly
Check expiration settings
Verify scope (private/shared)
Look for similar keys
Re-store data if needed

Issue: Stored data changes unexpectedly

Symptoms:

Data different than stored
Modifications not from this workflow

Causes:

Another workflow modified data
Update operation accidentally changed it
Merge settings caused overwrite

Solutions:

Use versioning to track changes
Review update operation carefully
Use immutable storage for reference data
Check permissions on data

Issue: Performance degradation with Data Hub

Symptoms:

Workflow runs slower
Data Hub operations delayed

Causes:

Storing too much data
Large individual entries
Too many concurrent operations
Inefficient retrieval

Solutions:

Clean up old data
Compress large entries
Use targeted retrieval
Optimize data structure

Real-World Scenarios

E-Commerce: Cross-Workflow Order Processing

Workflow 1: Create Order
[API: Create order]
[Extract: order_id]
[Data Hub: Store order_id with expiration 24h]

Workflow 2: Process Payment
[Data Hub: Retrieve order_id]
[API: Process payment with order_id]

Workflow 3: Send Confirmation
[Data Hub: Retrieve order_id]
[Email: Send confirmation with order_id]

Workflow 4: Verify Delivery
[Data Hub: Retrieve order_id]
[API: Check delivery status]

Testing: Baseline Performance Tracking

Workflow 1: Establish Baseline
[Run API calls multiple times]
[Calculate average response time]
[Data Hub: Store as baseline]

Workflow 2: Monitor Performance
[Run same API calls]
[Collect response times]
[Data Hub: Retrieve baseline]
[Compare: Current vs Baseline]
[Alert if regression > 10%]

Multi-Team: Shared Test Data

Team A: Creates Configuration
[Generate test endpoints]
[Data Hub: Store in Shared scope]

Team B: Uses Configuration
[Data Hub: Retrieve shared endpoints]
[Run tests using retrieved config]

Team C: Validates Usage
[Data Hub: Track retrieval statistics]
[Verify all teams use latest config]

Integration with Other Actions

Data Hub + Retry

Reliable data operations:

[Data Hub - Store Action]
   Configuration:
   - Retry: 3 attempts
   - Backoff: Exponential

Ensures data is saved even with temporary failures

Data Hub + Condition

Conditional storage:

IF operation_successful THEN
    [Data Hub: Store result]
ELSE
    [Data Hub: Store error details]

Data Hub + Send Email

Include stored data in emails:

[Data Hub: Retrieve metrics]
[Send Email: Include metrics in body]

Data Propagation Feature - Data flow overview
Session Persistence Feature - State management
Workflows - Workflow basics
Action Types Overview - All actions
Run Configuration - Execution settings

Summary

Data Hub stores and retrieves workflow data persistently
Use for reference data shared across workflows
Set expiration to auto-cleanup old data
Use clear naming for easy identification
Choose appropriate scope (Private/Shared)
Combine with conditions for smart storage

Next: Learn about Comment Action for documentation.

PreviousSend Email NextComment

Last updated 12 days ago

hashtagOverview

hashtagWhen to Use Data Hub

hashtag✅ Good Use Cases

hashtag❌ Anti-Patterns (Don't Do)

hashtagConfiguration

hashtagBasic Setup

hashtagConfiguration Options

hashtagData Operations

hashtagStore Operation

hashtagRetrieve Operation

hashtagUpdate Operation

hashtagDelete Operation

hashtagPractical Examples

hashtagExample 1: Store User ID for Later Use

hashtagExample 2: Track Test Metrics

hashtagExample 3: Share Test Data Across Workflows

hashtagExample 4: Versioned Data Management

hashtagExample 5: Counter Tracking

hashtagData Storage Best Practices

hashtag✅ Do

hashtag❌ Don't

hashtagNaming Conventions

hashtagRecommended Naming Pattern

hashtagOrganizing Related Data

hashtagWorking with Data Types

hashtagJSON Objects

hashtagArrays

hashtagPrimitives

hashtagData Lifecycle

hashtagCreate → Store → Retrieve → Update → Archive → Delete

hashtagPerformance Considerations

hashtagStorage Limits

hashtagOptimization Tips

hashtagTroubleshooting

hashtagIssue: Data not found on retrieval

hashtagIssue: Stored data changes unexpectedly

hashtagIssue: Performance degradation with Data Hub

hashtagReal-World Scenarios

hashtagE-Commerce: Cross-Workflow Order Processing

hashtagTesting: Baseline Performance Tracking

hashtagMulti-Team: Shared Test Data

hashtagIntegration with Other Actions

hashtagData Hub + Retry

hashtagData Hub + Condition

hashtagData Hub + Send Email

hashtagRelated Topics

hashtagSummary

Overview

When to Use Data Hub

✅ Good Use Cases

❌ Anti-Patterns (Don't Do)

Configuration

Basic Setup

Configuration Options

Data Operations

Store Operation

Retrieve Operation

Update Operation

Delete Operation

Practical Examples

Example 1: Store User ID for Later Use

Example 2: Track Test Metrics

Example 3: Share Test Data Across Workflows

Example 4: Versioned Data Management

Example 5: Counter Tracking

Data Storage Best Practices

✅ Do

❌ Don't

Naming Conventions

Recommended Naming Pattern

Organizing Related Data

Working with Data Types

JSON Objects

Arrays

Primitives

Data Lifecycle

Create → Store → Retrieve → Update → Archive → Delete

Performance Considerations

Storage Limits

Optimization Tips

Troubleshooting

Issue: Data not found on retrieval

Issue: Stored data changes unexpectedly

Issue: Performance degradation with Data Hub

Real-World Scenarios

E-Commerce: Cross-Workflow Order Processing

Testing: Baseline Performance Tracking

Multi-Team: Shared Test Data

Integration with Other Actions

Data Hub + Retry

Data Hub + Condition

Data Hub + Send Email

Related Topics

Summary