Cleanse

The Cleanse step enables you to validate Document field values and either repair or reject the Document before further processing. It is often helpful to use a Cleanse Step before a Map, so invalid values can be corrected before potentially causing errors while mapping.

The Cleanse step uses a Profile to determine what restrictions to validate. The restrictions are defined in the Profile at the field or Element level. Restrictions defined on repeating or "detail-level" Elements are automatically performed for each instance. The available restrictions differ slightly between different types of Profiles but include:

• Mandatory value
• Minimum and maximum length
• Date and numeric data type and formatting

For each Element, you can configure how to handle each restriction violation: reject the Document or repair the value. The repair options vary based on the type of restriction (see Repair Options below). It is important to understand that the entire Document will be rejected immediately upon the first violation--subsequent restrictions are not validated. Important: This also means that if your integration scenario involves a single batch file but requires individual records to be evaluated independently, you will need to use a Data Process Step to split the data into individual Documents before the Cleanse Step.

Documents are cleansed and then passed down one of two paths, "Clean" and "Rejected", to be processed accordingly. If a Document is rejected, the message is made available as a Document Property: "Standard" Document Property >> Meta Information >> Base >> Cleanse Result Message. This value can then be referenced in a Notify or Exception step to create an alert, for example. See Understanding Document Properties and Understanding Parameter Values for more information.

Note: Rejected Documents are processed before "clean" Documents. This allows you to halt the entire Process (using an End or Exception Shape) if a single Document is rejected, if required by your integration scenario.

Configuration

1. Select a Profile Type and choose a Profile from the dropdowns at the top. The Profile should already be configured with the necessary restrictions.
2. If you do not want to configure any repair actions, skip to the last step and click OK.
3. Click on an Element name in the Profile Tree on the left to display its Data Cleansing Details. Elements with a gray bullet ( ) do not have any restrictions defined in the Profile.
4. For each Cleansing Detail, determine if you want to configure a repair action. The Elements are set to reject by default, indicated by the stop icon ( ). If you want to repair the value, select the appropriate repair action from the dropdown and enter a repair value. The Cleansing Detail and Element will now show as repaired ( ). If an Element has multiple restrictions but only some are repaired, it will show as partially repaired ( ).
5. Repeat the above steps for each Element you want to repair.
6. When finished, click OK and connect the "Clean" and "Rejected" paths into the rest of the Process.

Repair Options

Restriction	Repair Options
Mandatory	Set a Default Value
Minimum Length	Prepend character to pad length Append character to pad length Enter the character to pad with.
Maximum Length	Remove excess leading characters Remove excess trailing characters
Data Type (numeric only)	Set a Default Value
Data Type Format (numeric and date)	Set a Default Value

Make sure the default values are still valid!

Example

This is example scenario, a batch CSV (comma-separated values) file of new orders is being picked up from your company's FTP server. The batch file will contain many orders. A number of restrictions are defined in the Flat File Profile "Order Extract CSV", including mandatory values, maximum lengths, and numeric and date formats. Properly formatted and populated orders should be inserted into the ordering database for fulfillment, but rejected orders should be sent back to the FTP site to be correct. In this scenario your Process might look like this: