Ability to limit concurrency of Iterators

For interators, have a batch value option for concurrent processing to limit the maximum number of concurrent streams generated and active in a session. The reason for this is that Snowflake only allows a maximum of 20 concurrent operations to occur on the same table and with the iterator, where I may be spawning over 100 concurrent operations hitting the same logging table, I often hit this Snowflake limit and the pipeline errors. I have tried to work around this using sequential processing (much slower ) and the retry iterator (mostly successful but still sometimes errors) so batching up concurrent operations to a limit of 20 (or some arbitrary user define value) would be very valuable.

Additionally, we could consider offering different “modes”, for example:

• Batched - which could include “strict” batching where one batch much complete before the next finishes. This would allow the introduction of things like break on failure etc
• Concurrent capped - which is a cap on the number of concurrent tasks