> For the complete documentation index, see [llms.txt](https://docs.qyrus.com/data-testing/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.qyrus.com/data-testing/blob-storage-and-file-systems/index.md).

# Blob Storage & File Connectors

> Configure blob or file storage connectors to validate file-based datasets stored in cloud platforms or on-premises systems.

## Overview

Blob storage and file systems provide scalable, cost-effective storage for large datasets. Data Testing supports major cloud storage providers and file transfer protocols for comprehensive file-based data validation.

{% hint style="info" %}
File-based connectors support various file formats (CSV, JSON, Parquet, etc.) and enable validation of data stored in cloud storage or on file servers.
{% endhint %}

***

## Available Storage Connectors

{% tabs %}
{% tab title="Cloud Storage" %}
**Cloud-native object storage services:**

| Connector                        | Platform        | Use Case              |
| -------------------------------- | --------------- | --------------------- |
| **Amazon S3**                    | AWS             | AWS data lake storage |
| **Azure Data Lake Storage Gen2** | Microsoft Azure | Azure cloud storage   |

Ideal for:

* ✅ Scalable file storage
* ✅ Data lake architectures
* ✅ Cost-effective storage
* ✅ Multi-region replication

[AWS S3 →](/data-testing/blob-storage-and-file-systems/index/amazon-s3.md) [Azure ADLS Gen2 →](/data-testing/blob-storage-and-file-systems/index/adls-gen2.md)
{% endtab %}

{% tab title="File Transfer" %}
**Network file transfer and on-premises systems:**

| Connector | Protocol          | Use Case            |
| --------- | ----------------- | ------------------- |
| **SFTP**  | SSH File Transfer | Remote file systems |

Ideal for:

* ✅ On-premises file servers
* ✅ Legacy system integration
* ✅ Secure file transfer
* ✅ Password & key authentication

[SFTP →](/data-testing/blob-storage-and-file-systems/index/sftp.md)
{% endtab %}
{% endtabs %}

***

## Common File Connector Configuration

### Connection Parameters

| Parameter          | Description                             | Required |
| ------------------ | --------------------------------------- | -------- |
| **Host/Endpoint**  | Storage endpoint or SFTP server address | ✅ Yes    |
| **Port**           | Service port (S3: 443, SFTP: 22)        | ✅ Yes    |
| **Authentication** | API keys, credentials, or certificates  | ✅ Yes    |
| **Bucket/Path**    | Storage location or directory path      | ✅ Yes    |

### File Configuration

| Parameter            | Description                            |
| -------------------- | -------------------------------------- |
| **File Path/Prefix** | Location of data files                 |
| **File Format**      | CSV, JSON, Parquet, XML, etc.          |
| **Delimiter**        | Field separator for structured formats |
| **Header Row**       | Whether file includes headers          |
| **Encoding**         | Character encoding (UTF-8, etc.)       |

***

## 📊 Amazon S3

Cloud-native object storage from AWS:

{% hint style="info" %}
Amazon S3 is ideal for:

* AWS cloud data lakes
* Large-scale file storage
* Multi-region data distribution
* Integration with AWS analytics services
  {% endhint %}

[**View Amazon S3 Connector →**](/data-testing/blob-storage-and-file-systems/index/amazon-s3.md)

***

## 🔷 Azure Data Lake Storage Gen2

Enterprise data lake on Azure:

{% hint style="info" %}
ADLS Gen2 is ideal for:

* Azure cloud deployments
* Enterprise data lakes
* Hadoop file system compatibility
* Integration with Azure Synapse and Power BI
  {% endhint %}

[**View ADLS Gen2 Connector →**](/data-testing/blob-storage-and-file-systems/index/adls-gen2.md)

***

## 🔐 SFTP

Secure file transfer protocol:

{% hint style="info" %}
SFTP is ideal for:

* On-premises file servers
* Legacy system integration
* Secure file transfer
* Remote team collaboration
  {% endhint %}

[**View SFTP Connector →**](/data-testing/blob-storage-and-file-systems/index/sftp.md)

***

## 🔐 Security Best Practices

{% hint style="warning" %}
**Essential Security Practices:**

1. ✅ Use IAM roles instead of static credentials
2. ✅ Enable bucket policies for least-privilege access
3. ✅ Use SSH keys for SFTP instead of passwords
4. ✅ Enable encryption at rest and in transit
5. ✅ Enable versioning for data protection
6. ✅ Configure MFA delete protection
7. ✅ Audit access logs regularly
8. ✅ Implement network isolation
   {% endhint %}

***

## Supported File Formats

| Format      | Extension  | Best For             | Support    |
| ----------- | ---------- | -------------------- | ---------- |
| **CSV**     | .csv       | Tabular data         | ✅ Full     |
| **JSON**    | .json      | Semi-structured data | ✅ Full     |
| **Parquet** | .parquet   | Columnar storage     | ✅ Full     |
| **XML**     | .xml       | Hierarchical data    | ⚠️ Limited |
| **Text**    | .txt       | Plain text           | ✅ Full     |
| **Excel**   | .xlsx/.xls | Spreadsheets         | ⚠️ Limited |

***

## Performance Considerations

| Factor         | Impact          | Optimization                       |
| -------------- | --------------- | ---------------------------------- |
| **File Size**  | Memory usage    | Use compression, split large files |
| **File Count** | Processing time | Batch files, use prefixes          |
| **Network**    | Transfer speed  | Use regional endpoints             |
| **Format**     | Parse time      | Use efficient formats (Parquet)    |
| **Encoding**   | Processing      | Ensure consistent encoding         |

***

## Storage Connector Comparison

| Feature            | Amazon S3       | ADLS Gen2    | SFTP              |
| ------------------ | --------------- | ------------ | ----------------- |
| **Cloud Provider** | AWS             | Azure        | Any               |
| **Authentication** | IAM/Access Keys | AD/SAS       | SSH Keys/Password |
| **Scalability**    | Unlimited       | Unlimited    | Server-dependent  |
| **Encryption**     | ✅ Yes           | ✅ Yes        | ✅ Yes (TLS)       |
| **Versioning**     | ✅ Yes           | ✅ Yes        | Manual            |
| **Cost**           | Low             | Low          | Varies            |
| **Setup**          | Cloud-native    | Cloud-native | Quick             |

***

## 🚀 Quick Start

1. **Choose your storage platform** (S3, ADLS Gen2, or SFTP)
2. **Configure access credentials** with minimal required permissions
3. **Enable encryption** for data security
4. **Identify target files** and their location/path
5. **Configure file format** and parsing options
6. **Test the connection** before creating jobs
7. **Create validation jobs** for data quality checks

***

## Related Documentation

* [Amazon S3 Connector](/data-testing/blob-storage-and-file-systems/index/amazon-s3.md)
* [Azure ADLS Gen2 Connector](/data-testing/blob-storage-and-file-systems/index/adls-gen2.md)
* [SFTP Connector](/data-testing/blob-storage-and-file-systems/index/sftp.md)
* [Data Source Overview](/data-testing/data-sources/index.md)
* [Create Compare Job](/data-testing/jobs-and-workflows/index/compare-job.md)