Skip navigation
Toggle Sidebar

7 - Best Practices

Based on a variety of project implementations and user feedback, the Dell Boomi Client Services team has organized a list of technical Best Practices for use on the Dell Boomi AtomSphere platform. Review the below categories as a starting point to best build, deploy and manage your integrations.


Development/Testing

  • Start simple and iterate. Create a very simple Process that moves data from source to destination to make sure the connections, filters and basic mapping are correct. Then begin introducing validations, routing, advanced mapping/functions, custom error handling, logging, etc.
  • Start with the end points. When building a new Process from scratch, begin by configuring the end points: the source connection/operation in the Start step and the destination connection/operation in a Connector step. The operation configuration (for most Connectors) will automatically generate the profile components that you will use in the mapping, decisions or other steps.
  • Know what the document data looks like before and after every step. You can use Test mode to review the source data at every step in the Process.
  • Limit the amount of records. Do not select all 100,000 records from your database in the first test. During initial testing, mock up small test files or use filters to restrict the number of records being processed (e.g., select specific records by ID or name). As you get more comfortable with the Process, then introduce larger data sets.
  • Test the Start step by itself to verify connection credentials and that the filters and/or criteria you specified are returning what you expect.
  • Deploy a local Atom to your PC for development, regardless of where you will eventually deploy the integration for production.


Process Design

  • Ask yourself these initial questions:
    • What does the source data look like and what does the destination data need to look like in terms of fields, format, structure and delimiters?
    • What mapping, transformations, lookups, etc. need to be performed to get the data from source to destination?
  • Make sure the source and destination data profiles are configured to what the data should look like with respect to data types, value formats (e.g., dates), delimiters, etc. Use mapping functions to transform the values.
    • When mapping a date field to another date field, configure the source and destination profile elements with the respective date masks and let the Dell Boomi mapping engine reformat the values implicitly, rather than using a date Format map function.
  • Minimize the number of connections and Connector calls.
    • Look for opportunities to batch queries and requests. Note that many application connections batch requests automatically for you "behind the scenes".
    • If you need to route documents based on multiple values from a lookup, do one lookup in a map function and capture the results in user-defined document properties rather than doing the same Connector call in multiple Decision and/or Route steps. See Understanding Document Properties for more information.
    • Use cross reference table components for small, static sets of translation values rather than making Connector calls.
  • Think sequentially!
  • Consider where potential points of failure are and how you will capture and handle errors.
  • Use Decision and/or Route steps to address common business validation scenarios or ones that require a different work flow. Do not try to replicate all the business logic that occurs in the destination application.
  • Look to create common, reusable components, especially for things like map functions.
    • Create user-defined map functions for all but the simplest mapping transformations to easily reuse and reconfigure the function.
  • Use user-defined document properties to easily track arbitrary values for a given document without having to stage data somewhere or include it in the document's actual data. (Often this is a precluded by the Connector's profile.)
  • Ask yourself, "How often does a Process really need to run?" Few integrations actually need to be truly real-time, event-based or need to run every minute. Try to schedule Processes every 15-30 minutes, hourly or daily. This will make development, testing and, more importantly, monitoring easier.
  • Extract records incrementally. Avoid extracting all records every time.
    • Most systems capture the last modified date per record. Use Connector operation filters to select only records that have been modified since the last time the Process ran. (The preferred method is to query by the Process's last successful run date or use your own persisted Process property to capture the last modified date from the records themselves.). See Understanding Process Properties and this article for more information.
    • Another approach is to extract records based on some status or flag field and then update that field to another value after the records have been processed successfully. This can be a good option if your application allows custom or user-defined fields because it is easy for end users to revert the status to reprocess a record.


Component Management

  • Use a consistent naming convention for components.
  • Organize! Think about folder hierarchy, common components, connections, etc. before digging in.
  • Avoid saving components in the root folder. Use sub-folders to manage integration projects. You can easily move a component to a different folder by dragging and dropping it.
  • Give meaningful, self-describing names to components. For example, name a Process "Salesforce Won Opportunities to NetSuite Sales Orders" rather than "My Salesforce Process".
  • Delete or rename components that are no longer used to avoid confusion later.


Performance & High Volume

See High Volume Troubleshooting and review common questions about memory inefficiencies in the Atom.

  • Understand how using cached values in map functions and decisions can improve performance.
  • Stagger Process schedules to maintain performance levels.
  • Set the Allow Simultaneous Process Executions check box in Process Options accordingly to allow or prevent repeated executions of the Process within a certain time interval.
  • Limit the number of Connector calls -- performance, API limits.
  • Consider using the Combine and Split Documents steps in a Data Process step to group common data sets rather than sending these documents individually into logic steps that perform lookups or Connector calls.
  • Note that setting the Last Successful Run Date Start step parameter in a new Process will extract all records from the source application because the deployed Process has never before been executed.
  • If appropriate, consider extracting all the records from the destination application and storing them locally (e.g., in a database table) to perform quicker lookups. This is often helpful when integrating or migrating very large sets of data to/from an online application, because remote web service calls are much more time consuming than local database calls.


Mapping

  • Ensure that repeating elements are configured to loop in the profile configuration, especially for XML profiles.
  • To reformat dates, configure the source and destination profile elements as a Date/Time type and specify a date mask. Then simply map the two fields without a date format function. The date value will be reformatted automatically. The same applies for Number types.
  • Start by mapping the minimum number of required fields before you map everything.
    • When you are trying to determine which fields are required, temporarily hard code default values.
    • When you are trying to identify mapping problems, unmap fields until you find the one causing the problem.


Database Integration

  • Understand table schemas, relationships and key fields.
  • Develop and test queries using a SQL tool like Query Analyzer or SQuirreL. This is quicker and gives you the results in a easily readable format.


Change Management

  • Always provide descriptive comments for the Process deployment version. This will help you and other users track the changes that were made.
  • Before deploying a new Process version, use the Compare Versions feature to compare the most recently deployed version with the Current Development Process to see which components have been modified, added or removed.
  • Create a backup folder and make backups of major Process deployments.


Data Migration

Also review the Performance & High Volume section.

  • Export or import data to intermediate CSV files. This will enable you to easily massage, sort and "de-dupe" records as necessary and it will facilitate reprocessing.
  • Begin by importing a small set of records to verify your Process, mapping, validations, etc. Slowly increase the number of records.
  • Give consideration to how invalid records and errors will be reported. Use Connector operation response handling to help identify which records failed. Consider appending warnings to an arbitrary local log file.
  • Note that setting the Last Successful Run Date Start step parameter in a new Process will extract all records from the source application because the deployed Process has never before been executed.
  • Use the Notify step to write custom log messages to a local log file to get processing results in real time rather than waiting for results to be reported on the Manage menu's Process Reporting page at the completion of the Process.


Widgets

See Widget Design Tips for more information.

Adaptavist Theme Builder Powered by Atlassian Confluence