🔧Functions Guide

Functions enable custom data transformations through Python code that can be reused across multiple validation jobs for consistent data processing.

Overview

Data transformation functions are powerful tools that preprocess and modify column data before validation or comparison. Reuse functions across jobs to maintain consistency and standardize data handling.

Perfect for:

• 🧹 Data cleaning and normalization

• 🔄 Custom transformation logic

• 📐 Format standardization

• 🛡️ Data masking and anonymization

• ✅ Validation rule enforcement

---

Key Benefits

Benefit	Impact
Reusability	Create once, use across unlimited jobs
Consistency	Standardized transformations everywhere
Maintainability	Update function in one place
Performance	Optimized, compiled transformations
Flexibility	Custom Python logic for any transformation

---

Function Types

Column Transformations

Transform individual column values

def transform_email(value):
    """Normalize email addresses"""
    return value.lower().strip()

Use for:

• Text normalization

• Date/time formatting

• Value calculations

• Type conversions

Data Cleaning

Aggregations

Security & Privacy

---

Quick Navigation

• Creating Functions – Build and deploy new transformation functions

• Using Functions – Apply functions to your validation jobs

---

Function Development Workflow

---

Common Use Cases

Email Normalization

def normalize_email(email):
    """Convert emails to lowercase, trim whitespace"""
    return email.lower().strip()

Applied to: Customer email validation and comparison

---

Date Standardization

def standardize_date(date_str):
    """Convert to ISO 8601 format"""
    from datetime import datetime
    dt = datetime.strptime(date_str, "%m/%d/%Y")
    return dt.strftime("%Y-%m-%d")

Applied to: Cross-source date field alignment

---

Phone Number Normalization

def format_phone(phone):
    """Extract digits and format"""
    digits = ''.join(c for c in phone if c.isdigit())
    if len(digits) == 10:
        return f"{digits[:3]}-{digits[3:6]}-{digits[6:]}"
    return digits

Applied to: Contact information validation

---

Currency Conversion

def usd_to_cents(amount):
    """Convert USD to cents"""
    return int(float(amount) * 100)

Applied to: Financial data comparison

---

Function Best Practices

Writing Effective Functions:

1. ✅ Keep functions focused on single transformation

2. ✅ Handle null/None values gracefully

3. ✅ Add input validation for type safety

4. ✅ Use descriptive names and docstrings

5. ✅ Test with edge cases and sample data

6. ✅ Document expected input/output formats

7. ✅ Optimize for performance with large datasets

---

Function Constraints & Limits

Constraint	Limit	Details
Function Size	64 KB	Maximum code length
Execution Time	5 seconds	Per value timeout
Memory	512 MB	Process memory limit
External Calls	Restricted	Network calls not allowed
File Access	Not allowed	No filesystem access

---

Debugging Functions

Test Your Function Locally

# Test locally before deployment
def clean_name(name):
    """Remove extra whitespace"""
    return ' '.join(name.split())

# Test cases
test_cases = [
    "John  Doe",
    "  Jane  Smith  ",
    "Bob"
]

for test in test_cases:
    print(f"'{test}' -> '{clean_name(test)}'")

View Function Logs

Check execution logs when applied in jobs to debug transformation issues.

Handle Errors Gracefully

def safe_transform(value):
    """Handle None and invalid values"""
    if value is None or value == "":
        return None
    try:
        return transform_logic(value)
    except Exception as e:
        # Log or return default value
        return value

---

Performance Tips

Optimization	Benefit
Avoid nested loops	Reduces O(n²) complexity
Use built-in functions	Faster than custom loops
Cache lookups	Avoid repeated calculations
Batch processing	Process multiple values together
Pre-compile regex	Speed up pattern matching

---

Related Documentation

• Creating Functions Guide

• Using Functions Guide

• Demo: Create Functions

• Demo: Use Functions

• Create Compare Job

---

FAQ

Q: Can I use external libraries? A: Standard Python libraries are supported. Third-party packages must be pre-approved.

Q: How many functions can I create? A: Unlimited functions. You can organize them by purpose or data domain.

Q: Can I edit a function after deploying? A: Yes, you can edit functions anytime. Changes apply to new job runs.

Q: What if my transformation fails? A: Implement error handling to return default values or null.