BTS2006 performance (batch processing)

[andrew]

I have a debatch orchestration that takes a batch request and breaks it up
into individual single requests. For each single request I start up a
processing orchestration that calls an internal webservice which in turn
calls an external webservice.

Scenario 1: small batch request (3 single requests)
- I get fairly decent response times between BizTalk and my internal
webservice (< 3 seconds for each request according to BAM).

Scenario 2: large batch request (1600 single requests)
- I get brutal response times between BizTalk and my internal webservice (as
high as 20 seconds according to BAM).

I know that the hop between my internal webservice and the external
webservice isn't the problem... when I insert trace statements to monitor I
consistently get approx. 1.5 seconds per request even when I'm running the
large batch.

Does anyone have any suggestions? For this thing to work I really need for
the response time to be < 5 seconds all the time.

SIDE NOTE: I have built throttling into the debatch orchestration with a
delay every X messages... which does seem to help a bit... but doesn't get me
anywhere near the performance I need.

[Benjamin Goeltz]

Andrew,
Stephen Thomas wrote an excellent article on the performance
considerations of a number of debatching options within BizTalk:
http://www.geekswithblogs.net/sthomas/archive/2004/12/12/17373.aspx
Hope that helps.
--
Cheers,
Benjamin

Post by andrew
I have a debatch orchestration that takes a batch request and breaks it up
into individual single requests. For each single request I start up a
processing orchestration that calls an internal webservice which in turn
calls an external webservice.
Scenario 1: small batch request (3 single requests)
- I get fairly decent response times between BizTalk and my internal
webservice (< 3 seconds for each request according to BAM).
Scenario 2: large batch request (1600 single requests)
- I get brutal response times between BizTalk and my internal webservice (as
high as 20 seconds according to BAM).
I know that the hop between my internal webservice and the external
webservice isn't the problem... when I insert trace statements to monitor I
consistently get approx. 1.5 seconds per request even when I'm running the
large batch.
Does anyone have any suggestions? For this thing to work I really need for
the response time to be < 5 seconds all the time.
SIDE NOTE: I have built throttling into the debatch orchestration with a
delay every X messages... which does seem to help a bit... but doesn't get me
anywhere near the performance I need.

[andrew]

I wrote myself a little test application that creates 10 threads and calls my
internal webservice which then calls the external webservice... essentially
removing biztalk from the equation so I could compare the performance times.

Turns out even my windows app was getting fairly poor performance as well.
A co-worker I was talking to seems to think the problem is that my biztalk
machine is on a VM... however I put my little test app on my physical machine
and saw similar poor performance. I'm not quite sure what to do about it at
this point.

Post by Benjamin Goeltz
Andrew,
Stephen Thomas wrote an excellent article on the performance
http://www.geekswithblogs.net/sthomas/archive/2004/12/12/17373.aspx
Hope that helps.
--
Cheers,
Benjamin

[Benjamin Goeltz]

Andrew,
You've likely thought of this already, but I would isolate the internal
and external web services to see if a majority of the latency is being caused
by one or the other. If the latency is on the internal web service, you
could look at load balancing to improve performance. If the latency is on
the external web service, you might look at improving bandwidth (optimistic,
I know)?
--
Cheers,
Benjamin

Post by andrew
I wrote myself a little test application that creates 10 threads and calls my
internal webservice which then calls the external webservice... essentially
removing biztalk from the equation so I could compare the performance times.
Turns out even my windows app was getting fairly poor performance as well.
A co-worker I was talking to seems to think the problem is that my biztalk
machine is on a VM... however I put my little test app on my physical machine
and saw similar poor performance. I'm not quite sure what to do about it at
this point.

Post by Benjamin Goeltz
Andrew,
Stephen Thomas wrote an excellent article on the performance
http://www.geekswithblogs.net/sthomas/archive/2004/12/12/17373.aspx
Hope that helps.
--
Cheers,
Benjamin

[andrew]

Yeah I put in tracing to time the external webservice before I did anything
else... and it is consistently fast...< 2 seconds per request.

By looking at load balancing are you talking about IIS settings? At one
point I increased the Max # of worker processes for my app pool from 1 to
10... but that didn't seem to affect anything.

Post by Benjamin Goeltz
Andrew,
You've likely thought of this already, but I would isolate the internal
and external web services to see if a majority of the latency is being caused
by one or the other. If the latency is on the internal web service, you
could look at load balancing to improve performance. If the latency is on
the external web service, you might look at improving bandwidth (optimistic,
I know)?
--
Cheers,
Benjamin

[Benjamin Goeltz]

Andrew,
When I said load balancing, I was actually talking about having multiple
IIS servers providing the internal web service functionality - i.e. have web
server 1 and web server 2 both available to call for the internal web
service. This can be implemented in a number of ways (some complex and
expensive, others relatively simple and cheap) - the right choice would
depend on how critical your solution is.
Based on the tests you've already run, it sounds like the latency is
introduced when calling the internal web service, but is not due to the call
to the external web service. Does your internal web service do anything else
than call the external web service?
--
Cheers,
Benjamin

Post by andrew
Yeah I put in tracing to time the external webservice before I did anything
else... and it is consistently fast...< 2 seconds per request.
By looking at load balancing are you talking about IIS settings? At one
point I increased the Max # of worker processes for my app pool from 1 to
10... but that didn't seem to affect anything.

Post by Benjamin Goeltz
Andrew,
You've likely thought of this already, but I would isolate the internal
and external web services to see if a majority of the latency is being caused
by one or the other. If the latency is on the internal web service, you
could look at load balancing to improve performance. If the latency is on
the external web service, you might look at improving bandwidth (optimistic,
I know)?
--
Cheers,
Benjamin

[andrew]

Yeah my internal website doesn't really do much other than call the external
one. I was able to improve my average performance a LOT when I changed the
debatch throttling from delay 20 seconds every 10th message to delay 3
seconds every 2nd message.

Post by Benjamin Goeltz
Andrew,
When I said load balancing, I was actually talking about having multiple
IIS servers providing the internal web service functionality - i.e. have web
server 1 and web server 2 both available to call for the internal web
service. This can be implemented in a number of ways (some complex and
expensive, others relatively simple and cheap) - the right choice would
depend on how critical your solution is.
Based on the tests you've already run, it sounds like the latency is
introduced when calling the internal web service, but is not due to the call
to the external web service. Does your internal web service do anything else
than call the external web service?
--
Cheers,
Benjamin

Post by andrew
Yeah I put in tracing to time the external webservice before I did anything
else... and it is consistently fast...< 2 seconds per request.
By looking at load balancing are you talking about IIS settings? At one
point I increased the Max # of worker processes for my app pool from 1 to
10... but that didn't seem to affect anything.

Post by Benjamin Goeltz
Andrew,
You've likely thought of this already, but I would isolate the internal
and external web services to see if a majority of the latency is being caused
by one or the other. If the latency is on the internal web service, you
could look at load balancing to improve performance. If the latency is on
the external web service, you might look at improving bandwidth (optimistic,
I know)?
--
Cheers,
Benjamin