💾Blob Storage & File Connectors

Configure blob or file storage connectors to validate file-based datasets stored in cloud platforms or on-premises systems.

Overview

Blob storage and file systems provide scalable, cost-effective storage for large datasets. Data Testing supports major cloud storage providers and file transfer protocols for comprehensive file-based data validation.

File-based connectors support various file formats (CSV, JSON, Parquet, etc.) and enable validation of data stored in cloud storage or on file servers.

---

Available Storage Connectors

Cloud Storage

Cloud-native object storage services:

Connector	Platform	Use Case
Amazon S3	AWS	AWS data lake storage
Azure Data Lake Storage Gen2	Microsoft Azure	Azure cloud storage

Ideal for:

• ✅ Scalable file storage

• ✅ Data lake architectures

• ✅ Cost-effective storage

• ✅ Multi-region replication

AWS S3 → Azure ADLS Gen2 →

File Transfer

---

Common File Connector Configuration

Connection Parameters

Parameter	Description	Required
Host/Endpoint	Storage endpoint or SFTP server address	✅ Yes
Port	Service port (S3: 443, SFTP: 22)	✅ Yes
Authentication	API keys, credentials, or certificates	✅ Yes
Bucket/Path	Storage location or directory path	✅ Yes

File Configuration

Parameter	Description
File Path/Prefix	Location of data files
File Format	CSV, JSON, Parquet, XML, etc.
Delimiter	Field separator for structured formats
Header Row	Whether file includes headers
Encoding	Character encoding (UTF-8, etc.)

---

📊 Amazon S3

Cloud-native object storage from AWS:

Amazon S3 is ideal for:

• AWS cloud data lakes

• Large-scale file storage

• Multi-region data distribution

• Integration with AWS analytics services

View Amazon S3 Connector →

---

🔷 Azure Data Lake Storage Gen2

Enterprise data lake on Azure:

ADLS Gen2 is ideal for:

• Azure cloud deployments

• Enterprise data lakes

• Hadoop file system compatibility

• Integration with Azure Synapse and Power BI

View ADLS Gen2 Connector →

---

🔐 SFTP

Secure file transfer protocol:

SFTP is ideal for:

• On-premises file servers

• Legacy system integration

• Secure file transfer

• Remote team collaboration

View SFTP Connector →

---

🔐 Security Best Practices

Essential Security Practices:

1. ✅ Use IAM roles instead of static credentials

2. ✅ Enable bucket policies for least-privilege access

3. ✅ Use SSH keys for SFTP instead of passwords

4. ✅ Enable encryption at rest and in transit

5. ✅ Enable versioning for data protection

6. ✅ Configure MFA delete protection

7. ✅ Audit access logs regularly

8. ✅ Implement network isolation

---

Supported File Formats

Format	Extension	Best For	Support
CSV	.csv	Tabular data	✅ Full
JSON	.json	Semi-structured data	✅ Full
Parquet	.parquet	Columnar storage	✅ Full
XML	.xml	Hierarchical data	⚠️ Limited
Text	.txt	Plain text	✅ Full
Excel	.xlsx/.xls	Spreadsheets	⚠️ Limited

---

Performance Considerations

Factor	Impact	Optimization
File Size	Memory usage	Use compression, split large files
File Count	Processing time	Batch files, use prefixes
Network	Transfer speed	Use regional endpoints
Format	Parse time	Use efficient formats (Parquet)
Encoding	Processing	Ensure consistent encoding

---

Storage Connector Comparison

Feature	Amazon S3	ADLS Gen2	SFTP
Cloud Provider	AWS	Azure	Any
Authentication	IAM/Access Keys	AD/SAS	SSH Keys/Password
Scalability	Unlimited	Unlimited	Server-dependent
Encryption	✅ Yes	✅ Yes	✅ Yes (TLS)
Versioning	✅ Yes	✅ Yes	Manual
Cost	Low	Low	Varies
Setup	Cloud-native	Cloud-native	Quick

---

🚀 Quick Start

1. Choose your storage platform (S3, ADLS Gen2, or SFTP)

2. Configure access credentials with minimal required permissions

3. Enable encryption for data security

4. Identify target files and their location/path

5. Configure file format and parsing options

6. Test the connection before creating jobs

7. Create validation jobs for data quality checks

---

Related Documentation

• Amazon S3 Connector

• Azure ADLS Gen2 Connector

• SFTP Connector

• Data Source Overview

• Create Compare Job