🗄️SQL Connectors

Set up SQL database connectors to retrieve data for comparison or evaluation jobs. Each connector requires database credentials, connection details, and optional keys for alignment.

Overview

SQL (Structured Query Language) connectors enable you to connect to relational databases for data validation. Whether you're using cloud-based solutions, enterprise databases, or open-source options, Data Testing supports comprehensive SQL integration.

All SQL connectors support custom SQL queries, pagination controls, and advanced filtering to accommodate your specific validation needs.

---

Available SQL Connectors

Cloud Data Warehouses

Modern, scalable cloud-native solutions:

Connector	Platform	Approach
Amazon Redshift	AWS	Column-store data warehouse
Snowflake	Multi-Cloud	Elastic cloud data platform

Amazon Redshift → Snowflake →

Enterprise Databases

Open Source

---

Common SQL Settings

All SQL connectors share common configuration patterns:

Connection Parameters

Parameter	Description	Required
Host/Hostname	Database server address or IP	✅ Yes
Port	Database service port	✅ Yes
Database Name	Specific database to connect to	✅ Yes
Username	Authentication username	✅ Yes
Password	Authentication password	✅ Yes

Optional Parameters

Parameter	Description
Schema	Specific schema within the database
Connection Pool	Connection pooling configuration
Query Timeout	Query execution timeout (seconds)
SSL/TLS	Secure connection options

---

SQL Query Configuration

Write your SQL queries to:

• Select specific columns you want to validate

• Filter to relevant data with WHERE clauses

• Use consistent ordering for comparisons

• Consider performance with LIMIT clauses

Example Query Format

SELECT 
  id,
  customer_name,
  email,
  created_at,
  status
FROM customers
WHERE created_at >= '2024-01-01'
  AND status = 'active'
ORDER BY id ASC
LIMIT 10000;

---

Advanced Features

Pagination & Limits

Enable pagination controls to handle large datasets efficiently:

• 📊 Limit - Restrict number of records retrieved

• 📍 Offset - Skip specific number of records

• 🔄 Batch Processing - Process data in chunks

Perfect for reducing resource usage and validating data in batches.

Alignment Keys

Specify keys for aligning data during comparison jobs:

• Use Primary Keys to match records across sources

• Support for Composite Keys (multiple columns)

• Automatic key detection available

---

Connection Best Practices

Security & Performance:

1. ✅ Use dedicated read-only database users

2. ✅ Restrict database access by network/IP

3. ✅ Enable TLS/SSL for connections

4. ✅ Rotate credentials regularly

5. ✅ Monitor query performance and resource usage

6. ✅ Keep database statistics up-to-date for optimal planning

---

Troubleshooting Connection Issues

Issue	Solution
Connection Timeout	Check host/port, verify firewall rules
Authentication Failed	Verify username/password, check user permissions
Query Errors	Validate SQL syntax, check table/column names
Performance Issues	Add query limits, optimize WHERE clauses

---

Quick Links

SQL Connectors:

• Amazon Redshift - AWS data warehouse

• MySQL - Open-source relational database

• Microsoft SQL Server - Enterprise SQL database

• IBM DB2 - Enterprise database system

• Oracle Database - Industry standard database

• Snowflake - Cloud data platform

Related Topics:

• Compare Jobs

• Evaluate Jobs

• Data Source Overview