Discussion:
Problem With BCP and -b option - Open-Client 12.5
(too old to reply)
Pablo Ibarra Duprat
2004-04-22 14:23:14 UTC
Permalink
Hi Sybase Team

I have a Sybase 12.5.0.3 EBF 11443 - ESD #4 and
Open-Client version ( using bcp -v ) : Sybase CTBCP Utility/12.5/P-EBF11490
ESD #14/DRV.12.5.0.6/RS6000/AIX 4.3.3/BUILD125-032/OPT/Tue Oct 7 23:00:44
2003.
When Use the bcp utility to load a data in tables,
BCP return with error "Fatal Error: Memory Allocation Failed" and must use
the -b parameters with a litle number associated (i.e -b10). I believe that
installing the last EBF for SDK this problem will be dissapear but still
here.
The platform is IBM AIX 5.1 32-bit!

¿Any Recommendation or suggest? any will be welcome
Thanks In Advance
PID
SybaseNeal
2004-04-22 16:14:55 UTC
Permalink
Hello,

There is a special note in EBF#14:
BCP and memory allocation
-------------------------
As a result of a fix to several issues related to memory allocation
and bcp (including CR 313067), you may see a change with bcp under
certain conditions.
When trying to bcp into a table that has many columns (> 100), bcp
might allocate substantial amounts of memory ( > 200 MB ). If you
experience the conditions described above, it is recommended that
you use smaller batch sizes (< 40) to avoid potential out-of-memory
errors.

So if your table has enough columns, it could cause bcp to require
upwards of 200MB of memory. You should be able to adjust your "ulimit"
parameters so that the operating system gives bcp the memory it requires.

I ran many tests with my ulimit set to unlimited so I could see how
much memory bcp required at different batch sizes (10, 100, 1000, 10000).
If I recall, it never required much more than 200MB for my table with
the most columns (I don't remember how many columns or what datatypes
they were).

The datatypes of your columns could affect memory usage for bcp as well.
If your table has text/image columns that contain more than 32K you have
to use the -T parameter with bcp to specify a larger default size. The
-T parameter can increase bcp's memory usage.

If you take all of this into account and test with your ulimit memory
parameters set to unlimited and see that bcp is using a lot more than
200 MB, you should log a case with Sybase tehcnical support.

Thanks,
Neal
Post by Pablo Ibarra Duprat
Hi Sybase Team
I have a Sybase 12.5.0.3 EBF 11443 - ESD #4 and
Open-Client version ( using bcp -v ) : Sybase CTBCP Utility/12.5/P-EBF11490
ESD #14/DRV.12.5.0.6/RS6000/AIX 4.3.3/BUILD125-032/OPT/Tue Oct 7 23:00:44
2003.
When Use the bcp utility to load a data in tables,
BCP return with error "Fatal Error: Memory Allocation Failed" and must use
the -b parameters with a litle number associated (i.e -b10). I believe that
installing the last EBF for SDK this problem will be dissapear but still
here.
The platform is IBM AIX 5.1 32-bit!
¿Any Recommendation or suggest? any will be welcome
Thanks In Advance
PID
Pablo Ibarra Duprat
2004-04-23 20:02:26 UTC
Permalink
Thanks for your response....!

I see the values for ulimit configuration and obtain the next data:

$ ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 131072
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) 2097151
nofiles(descriptors) 2000

When you talk about : "You should be able to adjust your "ulimit" parameters
so that the operating system gives bcp the memory it requires"
Your speak about the value for "memory(kbytes) or another ?
Thanks in advance
PID
Post by SybaseNeal
Hello,
BCP and memory allocation
-------------------------
As a result of a fix to several issues related to memory allocation
and bcp (including CR 313067), you may see a change with bcp under
certain conditions.
When trying to bcp into a table that has many columns (> 100), bcp
might allocate substantial amounts of memory ( > 200 MB ). If you
experience the conditions described above, it is recommended that
you use smaller batch sizes (< 40) to avoid potential out-of-memory
errors.
So if your table has enough columns, it could cause bcp to require
upwards of 200MB of memory. You should be able to adjust your "ulimit"
parameters so that the operating system gives bcp the memory it requires.
I ran many tests with my ulimit set to unlimited so I could see how
much memory bcp required at different batch sizes (10, 100, 1000, 10000).
If I recall, it never required much more than 200MB for my table with
the most columns (I don't remember how many columns or what datatypes
they were).
The datatypes of your columns could affect memory usage for bcp as well.
If your table has text/image columns that contain more than 32K you have
to use the -T parameter with bcp to specify a larger default size. The
-T parameter can increase bcp's memory usage.
If you take all of this into account and test with your ulimit memory
parameters set to unlimited and see that bcp is using a lot more than
200 MB, you should log a case with Sybase tehcnical support.
Thanks,
Neal
Post by Pablo Ibarra Duprat
Hi Sybase Team
I have a Sybase 12.5.0.3 EBF 11443 - ESD #4 and
Open-Client version ( using bcp -v ) : Sybase CTBCP
Utility/12.5/P-EBF11490
Post by SybaseNeal
Post by Pablo Ibarra Duprat
ESD #14/DRV.12.5.0.6/RS6000/AIX 4.3.3/BUILD125-032/OPT/Tue Oct 7 23:00:44
2003.
When Use the bcp utility to load a data in tables,
BCP return with error "Fatal Error: Memory Allocation Failed" and must use
the -b parameters with a litle number associated (i.e -b10). I believe that
installing the last EBF for SDK this problem will be dissapear but still
here.
The platform is IBM AIX 5.1 32-bit!
¿Any Recommendation or suggest? any will be welcome
Thanks In Advance
PID
Neal Stack
2004-04-23 20:13:28 UTC
Permalink
Hello,

Yes, "memory" is the primary one. You may
need to adjust "stack" and/or "data" but try
setting "memory" to unlimited once just so you
can see how much it actually uses.
Post by Pablo Ibarra Duprat
Thanks for your response....!
$ ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 131072
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) 2097151
nofiles(descriptors) 2000
When you talk about : "You should be able to adjust your "ulimit" parameters
so that the operating system gives bcp the memory it requires"
Your speak about the value for "memory(kbytes) or another ?
Thanks in advance
PID
Post by SybaseNeal
Hello,
BCP and memory allocation
-------------------------
As a result of a fix to several issues related to memory allocation
and bcp (including CR 313067), you may see a change with bcp under
certain conditions.
When trying to bcp into a table that has many columns (> 100), bcp
might allocate substantial amounts of memory ( > 200 MB ). If you
experience the conditions described above, it is recommended that
you use smaller batch sizes (< 40) to avoid potential out-of-memory
errors.
So if your table has enough columns, it could cause bcp to require
upwards of 200MB of memory. You should be able to adjust your "ulimit"
parameters so that the operating system gives bcp the memory it requires.
I ran many tests with my ulimit set to unlimited so I could see how
much memory bcp required at different batch sizes (10, 100, 1000, 10000).
If I recall, it never required much more than 200MB for my table with
the most columns (I don't remember how many columns or what datatypes
they were).
The datatypes of your columns could affect memory usage for bcp as well.
If your table has text/image columns that contain more than 32K you have
to use the -T parameter with bcp to specify a larger default size. The
-T parameter can increase bcp's memory usage.
If you take all of this into account and test with your ulimit memory
parameters set to unlimited and see that bcp is using a lot more than
200 MB, you should log a case with Sybase tehcnical support.
Thanks,
Neal
Post by Pablo Ibarra Duprat
Hi Sybase Team
I have a Sybase 12.5.0.3 EBF 11443 - ESD #4 and
Open-Client version ( using bcp -v ) : Sybase CTBCP
Utility/12.5/P-EBF11490
Post by SybaseNeal
Post by Pablo Ibarra Duprat
ESD #14/DRV.12.5.0.6/RS6000/AIX 4.3.3/BUILD125-032/OPT/Tue Oct 7
23:00:44
Post by SybaseNeal
Post by Pablo Ibarra Duprat
2003.
When Use the bcp utility to load a data in
tables,
Post by SybaseNeal
Post by Pablo Ibarra Duprat
BCP return with error "Fatal Error: Memory Allocation Failed" and must
use
Post by SybaseNeal
Post by Pablo Ibarra Duprat
the -b parameters with a litle number associated (i.e -b10). I believe
that
Post by SybaseNeal
Post by Pablo Ibarra Duprat
installing the last EBF for SDK this problem will be dissapear but
still
Post by SybaseNeal
Post by Pablo Ibarra Duprat
here.
The platform is IBM AIX 5.1 32-bit!
¿Any Recommendation or suggest? any will be welcome
Thanks In Advance
PID
Pablo Ibarra Duprat
2004-04-23 20:25:12 UTC
Permalink
Thank again Neal...

But you can give me a document that can explain me what's parameters over
Unix AIX are important for BCP operations ?
Or tell me what's document I can read to learn more please.

Thanks in Advance
PID
Post by SybaseNeal
Hello,
Yes, "memory" is the primary one. You may
need to adjust "stack" and/or "data" but try
setting "memory" to unlimited once just so you
can see how much it actually uses.
Post by Pablo Ibarra Duprat
Thanks for your response....!
$ ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 131072
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) 2097151
nofiles(descriptors) 2000
When you talk about : "You should be able to adjust your "ulimit" parameters
so that the operating system gives bcp the memory it requires"
Your speak about the value for "memory(kbytes) or another ?
Thanks in advance
PID
Post by SybaseNeal
Hello,
BCP and memory allocation
-------------------------
As a result of a fix to several issues related to memory allocation
and bcp (including CR 313067), you may see a change with bcp under
certain conditions.
When trying to bcp into a table that has many columns (> 100), bcp
might allocate substantial amounts of memory ( > 200 MB ). If you
experience the conditions described above, it is recommended that
you use smaller batch sizes (< 40) to avoid potential out-of-memory
errors.
So if your table has enough columns, it could cause bcp to require
upwards of 200MB of memory. You should be able to adjust your "ulimit"
parameters so that the operating system gives bcp the memory it requires.
I ran many tests with my ulimit set to unlimited so I could see how
much memory bcp required at different batch sizes (10, 100, 1000, 10000).
If I recall, it never required much more than 200MB for my table with
the most columns (I don't remember how many columns or what datatypes
they were).
The datatypes of your columns could affect memory usage for bcp as well.
If your table has text/image columns that contain more than 32K you have
to use the -T parameter with bcp to specify a larger default size. The
-T parameter can increase bcp's memory usage.
If you take all of this into account and test with your ulimit memory
parameters set to unlimited and see that bcp is using a lot more than
200 MB, you should log a case with Sybase tehcnical support.
Thanks,
Neal
Post by Pablo Ibarra Duprat
Hi Sybase Team
I have a Sybase 12.5.0.3 EBF 11443 - ESD #4 and
Open-Client version ( using bcp -v ) : Sybase CTBCP
Utility/12.5/P-EBF11490
Post by SybaseNeal
Post by Pablo Ibarra Duprat
ESD #14/DRV.12.5.0.6/RS6000/AIX 4.3.3/BUILD125-032/OPT/Tue Oct 7
23:00:44
Post by SybaseNeal
Post by Pablo Ibarra Duprat
2003.
When Use the bcp utility to load a data in
tables,
Post by SybaseNeal
Post by Pablo Ibarra Duprat
BCP return with error "Fatal Error: Memory Allocation Failed" and must
use
Post by SybaseNeal
Post by Pablo Ibarra Duprat
the -b parameters with a litle number associated (i.e -b10). I believe
that
Post by SybaseNeal
Post by Pablo Ibarra Duprat
installing the last EBF for SDK this problem will be dissapear but
still
Post by SybaseNeal
Post by Pablo Ibarra Duprat
here.
The platform is IBM AIX 5.1 32-bit!
¿Any Recommendation or suggest? any will be welcome
Thanks In Advance
PID
Neal Stack
2004-04-23 20:50:17 UTC
Permalink
Hello,

We do not have any guide, just the warning from the
Post by Pablo Ibarra Duprat
Post by SybaseNeal
Post by SybaseNeal
BCP and memory allocation
-------------------------
As a result of a fix to several issues related to memory allocation
and bcp (including CR 313067), you may see a change with bcp under
certain conditions.
When trying to bcp into a table that has many columns (> 100), bcp
might allocate substantial amounts of memory ( > 200 MB ). If you
experience the conditions described above, it is recommended that
you use smaller batch sizes (< 40) to avoid potential out-of-memory
errors.
bcp uses the default stack size for any given operating system.
You can double check your operating system documentation but I
would bet that the ulimit you have set for the stack is OK
(otherwise, bcp wouldn't run for other tables).

The data (data segment or heap) is also probably OK.

The virtual memory looks very low. For example, I'm using
Netscape right now and it is using 52,472Kb which is way
more than you are allowing bcp (32768Kb). Sybase was faced
with a choice of having bcp run very slowly and use a smaller
amount of memory OR running very fast and using lots of memory.
They chose to make it run fast and use lots of memory.

If you choose, you can make bcp run slower and use less memory
by selecting a smaller batch size.
Post by Pablo Ibarra Duprat
Thank again Neal...
But you can give me a document that can explain me what's parameters over
Unix AIX are important for BCP operations ?
Or tell me what's document I can read to learn more please.
Thanks in Advance
PID
Post by SybaseNeal
Hello,
Yes, "memory" is the primary one. You may
need to adjust "stack" and/or "data" but try
setting "memory" to unlimited once just so you
can see how much it actually uses.
Post by SybaseNeal
Thanks for your response....!
$ ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 131072
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) 2097151
nofiles(descriptors) 2000
When you talk about : "You should be able to adjust your "ulimit"
parameters
Post by SybaseNeal
Post by SybaseNeal
so that the operating system gives bcp the memory it requires"
Your speak about the value for "memory(kbytes) or another ?
Thanks in advance
PID
Post by SybaseNeal
Hello,
BCP and memory allocation
-------------------------
As a result of a fix to several issues related to memory allocation
and bcp (including CR 313067), you may see a change with bcp under
certain conditions.
When trying to bcp into a table that has many columns (> 100), bcp
might allocate substantial amounts of memory ( > 200 MB ). If you
experience the conditions described above, it is recommended that
you use smaller batch sizes (< 40) to avoid potential out-of-memory
errors.
So if your table has enough columns, it could cause bcp to require
upwards of 200MB of memory. You should be able to adjust your "ulimit"
parameters so that the operating system gives bcp the memory it
requires.
Post by SybaseNeal
Post by SybaseNeal
Post by SybaseNeal
I ran many tests with my ulimit set to unlimited so I could see how
much memory bcp required at different batch sizes (10, 100, 1000,
10000).
Post by SybaseNeal
Post by SybaseNeal
Post by SybaseNeal
If I recall, it never required much more than 200MB for my table with
the most columns (I don't remember how many columns or what datatypes
they were).
The datatypes of your columns could affect memory usage for bcp as well.
If your table has text/image columns that contain more than 32K you have
to use the -T parameter with bcp to specify a larger default size. The
-T parameter can increase bcp's memory usage.
If you take all of this into account and test with your ulimit memory
parameters set to unlimited and see that bcp is using a lot more than
200 MB, you should log a case with Sybase tehcnical support.
Thanks,
Neal
Post by Pablo Ibarra Duprat
Hi Sybase Team
I have a Sybase 12.5.0.3 EBF 11443 - ESD #4 and
Open-Client version ( using bcp -v ) : Sybase CTBCP
Utility/12.5/P-EBF11490
Post by SybaseNeal
Post by Pablo Ibarra Duprat
ESD #14/DRV.12.5.0.6/RS6000/AIX 4.3.3/BUILD125-032/OPT/Tue Oct 7
23:00:44
Post by SybaseNeal
Post by Pablo Ibarra Duprat
2003.
When Use the bcp utility to load a data in
tables,
Post by SybaseNeal
Post by Pablo Ibarra Duprat
BCP return with error "Fatal Error: Memory Allocation Failed" and must
use
Post by SybaseNeal
Post by Pablo Ibarra Duprat
the -b parameters with a litle number associated (i.e -b10). I believe
that
Post by SybaseNeal
Post by Pablo Ibarra Duprat
installing the last EBF for SDK this problem will be dissapear but
still
Post by SybaseNeal
Post by Pablo Ibarra Duprat
here.
The platform is IBM AIX 5.1 32-bit!
¿Any Recommendation or suggest? any will be welcome
Thanks In Advance
PID
SybaseNeal
2004-05-05 13:48:52 UTC
Permalink
Hello,

I was reviewing the list of bugs fixed in 12.5.1 ESD#2
(scheduled to be released this week) and found one bug
that was fixed that addressed excessive memory allocation
in bcp.

Bug 325371 reduces the amount of memory bcp allocates for
character columns. So, if your table contains a large
number of character columns, 12.5.1 ESD#2 should probably help.

- Neal
Post by Pablo Ibarra Duprat
Thank again Neal...
But you can give me a document that can explain me what's parameters over
Unix AIX are important for BCP operations ?
Or tell me what's document I can read to learn more please.
Thanks in Advance
PID
Post by SybaseNeal
Hello,
Yes, "memory" is the primary one. You may
need to adjust "stack" and/or "data" but try
setting "memory" to unlimited once just so you
can see how much it actually uses.
Post by Pablo Ibarra Duprat
Thanks for your response....!
$ ulimit -a
time(seconds) unlimited
file(blocks) unlimited
data(kbytes) 131072
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) 2097151
nofiles(descriptors) 2000
When you talk about : "You should be able to adjust your "ulimit"
parameters
Post by SybaseNeal
Post by Pablo Ibarra Duprat
so that the operating system gives bcp the memory it requires"
Your speak about the value for "memory(kbytes) or another ?
Thanks in advance
PID
Post by SybaseNeal
Hello,
BCP and memory allocation
-------------------------
As a result of a fix to several issues related to memory allocation
and bcp (including CR 313067), you may see a change with bcp under
certain conditions.
When trying to bcp into a table that has many columns (> 100), bcp
might allocate substantial amounts of memory ( > 200 MB ). If you
experience the conditions described above, it is recommended that
you use smaller batch sizes (< 40) to avoid potential out-of-memory
errors.
So if your table has enough columns, it could cause bcp to require
upwards of 200MB of memory. You should be able to adjust your "ulimit"
parameters so that the operating system gives bcp the memory it
requires.
Post by SybaseNeal
Post by Pablo Ibarra Duprat
Post by SybaseNeal
I ran many tests with my ulimit set to unlimited so I could see how
much memory bcp required at different batch sizes (10, 100, 1000,
10000).
Post by SybaseNeal
Post by Pablo Ibarra Duprat
Post by SybaseNeal
If I recall, it never required much more than 200MB for my table with
the most columns (I don't remember how many columns or what datatypes
they were).
The datatypes of your columns could affect memory usage for bcp as well.
If your table has text/image columns that contain more than 32K you have
to use the -T parameter with bcp to specify a larger default size. The
-T parameter can increase bcp's memory usage.
If you take all of this into account and test with your ulimit memory
parameters set to unlimited and see that bcp is using a lot more than
200 MB, you should log a case with Sybase tehcnical support.
Thanks,
Neal
Post by Pablo Ibarra Duprat
Hi Sybase Team
I have a Sybase 12.5.0.3 EBF 11443 - ESD #4 and
Open-Client version ( using bcp -v ) : Sybase CTBCP
Utility/12.5/P-EBF11490
Post by SybaseNeal
Post by Pablo Ibarra Duprat
ESD #14/DRV.12.5.0.6/RS6000/AIX 4.3.3/BUILD125-032/OPT/Tue Oct 7
23:00:44
Post by SybaseNeal
Post by Pablo Ibarra Duprat
2003.
When Use the bcp utility to load a data in
tables,
Post by SybaseNeal
Post by Pablo Ibarra Duprat
BCP return with error "Fatal Error: Memory Allocation Failed" and must
use
Post by SybaseNeal
Post by Pablo Ibarra Duprat
the -b parameters with a litle number associated (i.e -b10). I believe
that
Post by SybaseNeal
Post by Pablo Ibarra Duprat
installing the last EBF for SDK this problem will be dissapear but
still
Post by SybaseNeal
Post by Pablo Ibarra Duprat
here.
The platform is IBM AIX 5.1 32-bit!
¿Any Recommendation or suggest? any will be welcome
Thanks In Advance
PID
Loading...