Michael Jones

Processing data in an Enterprise environment frequently involves very large files. In a traditional approach, processing these files can involve issues around memory and time as it involves first downloading the entire file, attempting to load the entire file into memory to work on, and then saving the whole file to output. Instead of taking this old-fashioned approach, we can utilized data streaming and deferred DataWeave processing in order to streamline this process.

Frequently when building integrations, especially batch jobs, we don't necessarily know the configuration for all of the systems we will be connecting to ahead of time; the configurations are typically (and should be) externalized. While it doesn't feel intuitive, we can create a global configuration and use scoped variables to dynamically create new connector configurations at runtime.