Create Job

Evaluate jobs perform comprehensive validations on a single data source.

Step 1: Job Type Selection

• Select Job Type as Evaluate.

• Enter a Description for the job.

• Click Next.

Select Job Type

Step 2: Connection Configuration

1. Select the Data Source Connector from the dropdown.

2. Provide the required Connection Information.

3. Click Verify to test the connection.

4. Once verified successfully, click Next.

Connection Configuration

Step 3: Adding Verifications

Configure data validations for the selected data source.

Supported Verification Types

1. Row Count Verification – Validates total number of records.

2. Column Datatype Verification – Ensures columns match expected data types.

3. Column Duplicates Verification – Identifies duplicate records.

4. Column Not Null Verification – Checks for null values.

5. Column Verification using Regex – Validates values using regular expressions.

6. Column Verification using Python UDFs – Applies custom Python functions.

Verifications

Data Source Information appears at the top of the screen with column names and data types (available for database connectors).

Configuration Steps

1. Enable the desired verification option(s).

2. Add column name(s) for verification.

3. Provide additional parameters if required (e.g., Regex pattern, Python UDF).

Single vs. Combination Column Verification

The following verifications support both individual columns and column combinations:

• Column Datatype Verification.

• Column Duplicates Verification.

• Column Not Null Verification.

Configuration Examples

• Single column: Enter the column name and press Enter.

• Multiple columns: Enter column names separated by commas and press Enter.

Step 4: Save or Run the Job

Choose an execution option:

• Save – Saves job configuration for later execution.

• Save and Start – Saves and immediately starts the evaluation.

• Quick Run – Optimized execution for faster processing (supports up to 500,000 records).

Start Job