Saturday, May 19, 2007

Large message processing

Typical Large message documents:

  • Large flat file documents with high volume (many records) and occasionally batched
  • Large flat file documents wrapped in a single CDATA section node in an XML
  • Large XML documents with thousands to millions of "rows" that were batched together
  • EDI interchanges where the file or data to be processed independently or in aggregate
  • Large flat document with a header and trailer at the starts and ends of the file with thousands to millions of records, each record need to be processed separately from the others, but the entire sequence must be processed in order to complete properly


Transforming a document with a map is a memory-intensive operation. BizTalk Server passes the message stream to the .Net XslTransform class, which then loads the document into a .NET XPathDocument object for processing in BizTalk 2006/2004, Where as DOM in the case of BizTalk 2002/2000. Loading the document into the .NET XPathDocument can potentially expand the original file size in memory by a factor of 10 or more.

XPathDocument caches information about the nodes of the XML along with the data itself to allow for faster access, but this result into high performance penalties because of the redundant data that sits in the objects. This is where 90%+ of the Out Of Memory (OOM) exceptions that cause orchestrations and receive/send ports to fail.

This expansion may be more pronounced when mapping flat files because flat files must be parsed into XML before they can be transformed

Note:
1 MB document may be enough with JITTed product and user code assemblies, other messages flowing through the process enough to blow the process to 200-500 MB in memory.
Since BizTalk converts the data into XML for internal processing we need to worry more with the flat files (Non-Xml files) thou they are designed to be as efficient as possible in order minimize cost, but XML explicitly stated this as a non-goal, with readability as a much higher priority.


The best recommendation not to send data that is more than 1MB into BizTalk, without some form of custom processing or large memory machines.
If possible try to transform the XML file before passing onto BizTalk Server Orchestration.

Other approach is to use distinguished fields or property promotion in our process. Orchestration does not load the data of the message stream unless required orchestration will fetch the right value without loading the whole message into memory and update the value this is a powerful means to manipulate key fields without loading the whole document into memory.

Adjust the message size threshold above which documents are buffered to the file system during mapping. To modify the size threshold, create a DWORD value named TransformThreshold in the BizTalk Server registry

HKLM\Software\Microsoft\BizTalk Server\3.0\Administration\TransformThreshold

Enter a decimal value with the number of bytes to set the new threshold to. E.g. 2097152 to increase the message size threshold to 2 MB (from the default of 1 MB). Increase this value on systems with a large amount of available memory to improve throughput. Buffering documents to disk conserves memory at a slight cost to overall throughput.

Wednesday, May 2, 2007

Typed/Untyped Messages & its Implementation

Let’s get into details what is a Typed Message?
Typed Message nothing but a strongly typed message conforms to a selected schema (XSD) or .NET class and the message inherits its properties from this schema or .NET class.

Where as Untyped Message nothing but a message which is configured to use System.Xml.Xmldocument as Message type
Or Non-typed Message, which is not tied to a specific schema

Let’s take simple scenario where by you need to receive PO’s (Purchase Orders) from different locations, systems or partners
Then implement predefine business logic (like check stocks, grade, delivery etc), since we are receiving from different sources PO details might vary.

As a developer we might need to define a generic fashion solution rather then customized solution to fit/resolve one issue!
Instead of building multiple solutions, you are now choose to implement Untyped Message process to receive different PO’s
Then implement predefine business logic, which gives you more manageability over the solutions.

Sunday, February 18, 2007

Great way of looking into BizTalk Assembly

As a developer we might wonder some time did I deployed right version of assembly!? Or thou my deployment went fine but I don’t see expected change or result at all!! Which version am I using!? Does my Assembly contain newly deployed schemas! etc

Answer for all this and without opening BizTalk MMC! One DLL registration away! Simply register BtsAsmExt.dll DLL and you are done!!



Regsvr32 "C:\Program Files\Microsoft BizTalk Server 2006\Developer Tools\BtsAsmExt.dll"

Thursday, January 25, 2007

Compensation Model with Exception Handling

BizTalk supports two types of transactions namely Atomic & Long Running (L-R).Atomic transaction by nature does support ACID & Long Running (L-R) Transaction doesn’t support Classic ACID transaction. Hoping that you guys know ACID, but let me recap quickly ACID (Atomicity, Consistency, Isolation and Durability)

Atomic = All or nothing, either all the actions happen or none.
Consistent = The transaction as a whole is a correct transformation of the database.
Isolated = Each transaction runs as though there are no concurrent transactions.
Durable = The effects of committed transactions survive failures.

And it was expressed in very elegant mathematical theory of serializability and the Influence of the theory and the practical mechanism was recognized in the year 1998 by turning award to Jim Gray (70s he was responsible for some of the most fundamental work on database theory and practice)

Back to L-R Transaction, within the context of BizTalk Server an L-R transaction supports Consistency and Durability but not Atomicity and Isolation.This is true in a way since L-R decomposable into a number of inner transactions in which some or all may be ACID transactions and failed transaction can be rolled back in the normal way, but if previous ACID transactions might already been committed then it is too late to roll back.Which can be mitigated by the concept called compensation handlers, In which each inner transaction is provided with its own compensation handler. In the event of a failure the inner transaction's compensation executed and transaction will be rolled back, since we can’t roll back committed transactions so each compensation handler must do some action which can cancels or reverses the effect of its associated transaction which is nothing but backwards recovery. In some scenarios we can’t fully reverse the effect like email out so compensation doesn’t guarantee 100% recovery.

Assume that we have Transaction A, Transaction B, Transaction C & Transaction D and hit with error at Transaction C, now compensation come into picture to perform backwards recovery. Now forward recovery begins at transaction B what if error again hit at Transaction C it will go into backwards recovery again, which might lead into endless loop of recovery.To avoid this we need to implement exception handling saying that after N number of tries stop the recovery and exit from the loop.

Friday, December 22, 2006

Commerce Server Catalog Manager Encoding (UTF-8/UTF-16) issue

While working on Automated Catalog process using BizTalk Server Adapters for Commerce Server Catalog Subsystem found some thing interesting with Xml file (Processed by Catalog Manager) I am using Aggregator Pattern in BizTalk to generate final XML file which grabbed and processed by the Catalog Web service.

First I wasn't successful, thought that issue with encoding in the XML file, so decided to deploy Pipeline to set the encoding UTF-16 instead of default UTF-8 in XML assembly but again the end results wasn't fruitful (thou the file grow twice because of encoding).

While comparing the xml file got an idea!!. How about open the file in the Explorer copy and past the content into notepad and save the file as xml (removed -, + then Change "&" to "&" etc) then pass it to Catalog Web service also failed

Then I realized to use hex editors or Unicode editors. Since UTF-8 is the byte-oriented encoding form of Unicode. Quickly googled out and found few wonderful freeware did the same as above this time it worked with out any glitch.

Then I open up both file in Binary Editor found the difference, before it was 3E 3C where as with the Unicode editor (since open up the file in explorer) saved file 3E 20 0D 0A 3C

20 --> Space; 0D --> CR; 0A --> NL
With this can I say that Catalog Manager Parser expecting a well formatted XML file. Or it’s not well written to handle above scenario!?? at this point I don't know yet it to be consider as bug!!

Thursday, December 14, 2006

Need Well Formatted XML File for uploading Catalog


MSCommerceCatalogCollection2 version 3.0 need to be well formatted and encode if it’s contains special characters like “Tracker™ “other wise upload won’t be successful and system won’t show any errors and display as if it was successfully uploaded. In other words we will see this kind of behavior/result in the Catalog Manager Status paint for all unformatted xml file.

BizTalk Adapter For MSCS2007 Catalog System

Be aware that If you won’t set the Parent Category tag for Base Category in the MSCommerceCatalogCollection2 xml file While uploading through Catalog Manger or through BizTalk Adapters for CS2007 using Catalog web Service
You will end up having sub categories as base, hence no base category exist for sub categories Simply because base wasn’t created prior to inserting sub categories
So Set some value in the Parent Category tag Like
< ParentCategory >Same As Base Category Display Value </ParentCategory >