Amazon S3

Cloud object storage for large-scale data validation, perfect for AWS-based data lakes and analytics workloads.

Overview

Amazon Simple Storage Service (S3) is a scalable object storage service that enables you to store and retrieve any amount of data. The Data Testing connector integrates with S3 to validate file-based data stored in AWS.

Perfect for:

• ✅ AWS data lake validation

• ✅ Large dataset comparisons

• ✅ Export file validation

• ✅ Cross-region data checking

---

⚙️ Configuration Parameters

Parameter	Description	Required	Example
Access Key	AWS access key ID	✅ Yes	AKIAIOSFODNN7EXAMPLE
Secret Key	AWS secret access key	✅ Yes	wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Bucket Name	S3 bucket name	✅ Yes	my-data-lake
Role	IAM role (alternative to keys)	⚠️ Optional	arn:aws:iam::123456:role/S3Access
File Key	S3 object path/key	✅ Yes	data/exports/customers.csv
Rules File Key	Validation rules file location	⚠️ Optional	rules/validation_rules.json

Configuration Screenshot

---

🔐 Authentication Methods

Access Key & Secret

Programmatic credentials for API access

1. Create an IAM user or get credentials

2. Generate Access Key ID and Secret Access Key

3. Enter both keys in configuration

4. Click Test Connection

Most common method for applications and scripts.

IAM Role

---

📖 Getting Started

Step 1: Create AWS Credentials

Create an IAM user with S3 access:

Using AWS Management Console:
1. Go to IAM > Users > Create User
2. Enable programmatic access
3. Attach S3 read policy
4. Download credentials CSV

Step 2: Set Up S3 Bucket Access

Ensure your S3 bucket is accessible and contains your data files:

Bucket Name: my-data-lake
Folder Structure:
├── data/
│   ├── customers.csv
│   ├── orders.json
│   └── products.parquet
└── exports/
    └── 2024-03-10/

Step 3: Configure the Connector

1. Navigate to your job configuration

2. Select Amazon S3 as the data source

3. Enter AWS credentials or IAM role

4. Specify bucket name

5. Enter file path/key

6. Click Test Connection

7. Verify file format and preview data

---

📊 Supported File Formats

Format	Extension	Use Case	Support
CSV	.csv	Tabular data export	✅ Full
JSON	.json	Semi-structured data	✅ Full
Parquet	.parquet	Columnar analytics data	✅ Full
XML	.xml	Hierarchical data	✅ Full
Text	.txt	Plain text files	✅ Full

CSV Format Example

customer_id,customer_name,email,status
1,John Doe,[email protected],active
2,Jane Smith,[email protected],active
3,Bob Johnson,[email protected],inactive

JSON Format Example

[
  {
    "customer_id": 1,
    "customer_name": "John Doe",
    "email": "[email protected]",
    "status": "active"
  },
  {
    "customer_id": 2,
    "customer_name": "Jane Smith",
    "email": "[email protected]",
    "status": "active"
  }
]

---

🚀 S3-Specific Features

S3 Object Paths

S3 uses a key-based naming system:

Bucket: my-data-lake
Paths:
- data/customers.csv
- exports/2024-03-10/orders.csv
- archive/2023/backup.csv

Cross-Region Validation

Validate data across AWS regions:

Region: us-east-1  → Bucket 1
Region: eu-west-1 → Bucket 2
Region: ap-southeast-1 → Bucket 3

Versioning

S3 can track file versions for audit and recovery:

Enable S3 Versioning to:
- Track file changes over time
- Recover previous versions
- Compare historical data

---

⚠️ File Size Limitations

Maximum File Size: 5 GB per file

For larger files:

• Split into multiple smaller files

• Use partitioned approaches

• Archive older data separately

Handling Large Files

Large Dataset Strategy:
1. Partition by date: customers_2024_01.csv, customers_2024_02.csv
2. Partition by region: customers_us.csv, customers_eu.csv
3. Partition by status: customers_active.csv, customers_inactive.csv

---

🔐 Security Best Practices

Security Essentials:

1. ✅ Use IAM roles instead of long-term credentials

2. ✅ Enable bucket versioning

3. ✅ Enable S3 encryption (SSE-S3 or SSE-KMS)

4. ✅ Block public access to bucket

5. ✅ Enable access logging

6. ✅ Use VPC endpoints for private access

7. ✅ Implement bucket policies for least privilege

8. ✅ Rotate credentials every 90 days

S3 Bucket Policy Example

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:user/validator"
      },
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::my-data-lake",
        "arn:aws:s3:::my-data-lake/*"
      ]
    }
  ]
}

---

💡 Cost Optimization

Strategy	Benefit
Use storage classes	Store infrequent data in Glacier/Deep Archive
Enable versioning cleanup	Use lifecycle policies to delete old versions
Compress data	Use gzip or Parquet for smaller files
Batch operations	Process multiple files in single job
Archive	Move old data to cheaper storage

---

🐛 Troubleshooting

Issue	Solution
Access denied	Check IAM permissions, verify credentials
File not found	Verify bucket name and file path/key
File too large	Check file size (max 5 GB)
Invalid credentials	Regenerate access key/secret
Connection timeout	Check network/firewall, verify bucket exists
Permission denied	Ensure IAM user has S3:GetObject permission

---

Related Documentation

• Blob Storage Overview

• Azure ADLS Gen2 Connector

• SFTP Connector

• Data Source Overview

• AWS S3 Documentation