Azure Data Lake Storage Gen2

Enterprise data lake on Microsoft Azure with hierarchical namespace, perfect for large-scale analytics and big data workloads.

Overview

Azure Data Lake Storage (ADLS) Gen2 is a cloud-scale data lake solution that combines the capabilities of Azure Blob Storage with distributed file system semantics. It's ideal for organizations running analytics at scale with Hadoop, Spark, and other big data frameworks. The Data Testing connector integrates seamlessly with ADLS Gen2 for comprehensive file-based data validation.

Perfect for:

• ✅ Azure cloud data lake validation

• ✅ Enterprise analytics platforms

• ✅ Synapse Analytics integration

• ✅ Multi-tenant data storage

---

⚙️ Configuration Parameters

Parameter	Description	Required	Example
Account Name	Storage account name	✅ Yes	mydatalake
Account Key	Primary or secondary key	⚠️ Optional*	DefaultEndpointsProtocol=https...
Connection String	Complete connection string	⚠️ Optional*	key=value;
SAS Token	Shared Access Signature	⚠️ Optional*	sv=2020-08-04&...
Tenant ID	Azure AD tenant ID	⚠️ Optional*	xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Client ID	Azure AD app client ID	⚠️ Optional*	xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
Client Secret	Azure AD app secret	⚠️ Optional*	••••••••••
Filesystem/Container	Container name	✅ Yes	raw-data
File Path	Path to data file	✅ Yes	exports/customers.csv
Rules File Path	Validation rules location	⚠️ Optional	rules/validation.json

*Use one authentication method

Configuration Screenshot

ADLS Configuration

---

🔐 Authentication Methods

Account Key

Storage account connection using primary/secondary key

1. Go to Azure Portal → Storage Account

2. Navigate to Access Keys

3. Copy account name and key

4. Enter in configuration

5. Click Test Connection

Simple and direct for development/test environments.

Connection String

SAS Token

Azure AD (OAuth)

---

📖 Getting Started

Step 1: Set Up Azure Storage Account

Create a storage account with ADLS Gen2 enabled:

Azure Portal → Create Resource → Storage Account
- Account name: mydatalake
- Performance: Standard (or Premium for faster access)
- Redundancy: Choose based on requirements
- Enable hierarchical namespace: YES (required for Gen2)

Step 2: Create Container and Upload Data

Container: raw-data
Folder structure:
📦 raw-data/
├── 📁 customers/
│   ├── 📄 2024-01-01.csv
│   ├── 📄 2024-02-01.csv
│   └── 📄 2024-03-01.csv
├── 📁 orders/
│   └── 📄 orders.parquet
└── 📁 exports/
    └── 📄 backup.csv

Step 3: Configure Authentication

Choose your preferred authentication method (see tabs above).

Step 4: Configure the Connector

1. Navigate to your job configuration

2. Select Azure ADLS Gen2 as the data source

3. Enter storage account and authentication details

4. Specify container name

5. Enter file path

6. Click Test Connection

7. Verify file format and preview

---

📊 Supported File Formats

Format	Extension	Use Case	Support
CSV	.csv	Structured tabular data	✅ Full
Parquet	.parquet	Columnar analytics format	✅ Full
JSON	.json	Semi-structured data	✅ Full
Avro	.avro	Data serialization	✅ Full
ORC	.orc	Columnar format	✅ Full
Text	.txt	Plain text files	✅ Full

File Format Examples

CSV Format:

customer_id,name,email,country
1,John Doe,[email protected],US
2,Jane Smith,[email protected],UK

Parquet Format (Columnar):

Optimized for analytics queries
- High compression
- Efficient column selection
- Fast query performance

---

🚀 ADLS Gen2-Specific Features

Hierarchical Namespace

ADLS Gen2 provides a true directory structure:

Container: raw-data
Path: /2024/03/10/customers.csv
     ├── Year level
     ├── Month level
     └── Day level

Directory-Level Operations

Atomic operations on directories:
- Create/delete entire folder trees
- Rename directories atomically
- Set permissions at directory level

Access Control

Set permissions at file and directory level:

Container/Directory → Access Control (IAM)
- Owner: rwx (7)
- Group: r-x (5)
- Other: --- (0)

---

⚠️ File Size Limitations

Maximum File Size: 5 GB per file

Strategies for larger files:

• Split files by date/partition

• Use compressed formats (Parquet, ORC)

• Archive old data separately

• Use Blob tier for cold storage

Partitioning Strategy

Optimize for large datasets:
/data/2024/01/customers.csv
/data/2024/02/customers.csv
/data/2024/03/customers.csv

---

🔐 Security Best Practices

Security Essentials:

1. ✅ Use Azure AD authentication (OAuth) for production

2. ✅ Enable encryption at rest (Microsoft-managed or customer-managed)

3. ✅ Enable encryption in transit (TLS 1.2+)

4. ✅ Use private endpoints for network isolation

5. ✅ Remove public access to containers

6. ✅ Enable storage account firewalls

7. ✅ Audit access logs regularly

8. ✅ Rotate credentials every 90 days

Register Azure AD Application

# Register app for service principal auth
az ad sp create-for-rbac `
  --name "DataTestingValidator" `
  --role "Storage Blob Data Reader"

Storage Account Firewall

Azure Portal → Storage Account → Networking
- Firewall: Enabled
- Allowed services: (select as needed)
- Exceptions: Check "Allow Azure services"

---

💡 Integration Examples

Use with Azure Synapse Analytics

-- Query ADLS data from Synapse
SELECT 
  customer_id,
  COUNT(*) as orders
FROM OPENROWSET(
  BULK 'https://mydatalake.dfs.core.windows.net/data/customers.csv',
  FORMAT = 'CSV',
  PARSER_VERSION = '2.0'
) AS data
GROUP BY customer_id;

Use with Azure Databricks

# Read from ADLS in Databricks
df = spark.read.csv(
  "abfss://[email protected]/customers.csv",
  header=True
)
df.show()

---

💰 Cost Optimization

Strategy	Benefit
Storage tiers	Use hot/cool/archive based on access patterns
Lifecycle policies	Automatically move old data to cheaper tiers
Compression	Use Parquet/ORC for smaller storage footprint
Deduplication	Remove redundant data
Archive	Move cold data to Archive tier

---

🐛 Troubleshooting

Issue	Solution
Authentication failed	Verify account name, key, or service principal
Container not found	Check container name (case-sensitive)
File not found	Verify file path and container access
Permission denied	Check role assignments and SAS permissions
File too large	Verify file size is under 5 GB limit
Connection timeout	Check firewall rules, try using private endpoint

---

Related Documentation

• Blob Storage Overview

• Amazon S3 Connector

• SFTP Connector

• Data Source Overview

• Azure ADLS Documentation