(sorry if this is a repost, I sent it before
confirming my subscription)
I'm having a memory problem with Perl in Panther that
I didn't have in Jaguar. When I try to read a large
text file using the line input operator (<>), I get an
"out of memory" error:
*** malloc: vm_allocate(size=8421376) failed (error
code=3)
*** malloc[5576]: error: Can't allocate region
Out of memory!
The strange thing is that I'm not slurping up the
whole file, I'm reading it line by line. But Perl
seems to be slurping the whole thing into memory
before even processing the first line. But even so,
the file isn't large enough to use up all physical and
virtual memory. I know it's not an issue of using the
wrong linefeed character, as this script works fine in
Jaguar and processes the same text file line-by-line
without a problem. Here's the structure of my code:
open (FILE1, "file1.txt"); # 300MB Mac text file
$/ = "\r";
foreach my $line (<FILE1>) {
chomp $line;
(do something...)
}
close FILE1;
open (FILE2, "file2.txt"); # 300MB Mac text file
$/ = "\r";
foreach my $line (<FILE2>) {
chomp $line;
(do something)
}
close FILE2;
Here's how top displays Perl's memory usage after
opening the first 300MB file, on a machine with 2 gigs
of RAM and a hundred gigs of free HD space:
PID COMMAND %CPU TIME #TH #PRTS #MREGS RPRVT
RSHRD RSIZE VSIZE
5576 perl 63.2% 2:56.60 1 13 364
1.38G- 1.37M 397M+ 1.57G
After processing all of the lines of the first file
(during which the memory usage increases only
slightly), the script tries to read the second file
and quits with the memory error before it processes
the first line of the second file.
Is this a bug in Perl 5.8.1? Is there a way to force
Perl to not slurp up the whole file at once, or at
least to release the memory used up by the first file
before reading the second?
Thanks!
XB
Jarkko Hietaniemi - 07 May 2004 07:41 GMT
> *** malloc: vm_allocate(size=8421376) failed (error
> code=3)
[quoted text clipped - 3 lines]
> The strange thing is that I'm not slurping up the
> whole file,
Oh but you are...
> I'm reading it line by line. But Perl
> seems to be slurping the whole thing into memory
> before even processing the first line. But even so,
> foreach my $line (<FILE1>) {
This reads in the whole file first because the (...) of
for each enforces list context, and the <FH> in list context
does read in the whole handle. You meant
while (my $line = <FILE1>) {
my $line = $_;
or something similar.
Xavier Broccoli - 07 May 2004 07:54 GMT
Oh, you are so right. I guess Panther is just
stingier with its memory, and I never got bitten by
this before!
Thanks!!
XB
> > *** malloc: vm_allocate(size=8421376) failed
> (error
[quoted text clipped - 24 lines]
>
> or something similar.
Sherm Pendley - 07 May 2004 07:58 GMT
> The strange thing is that I'm not slurping up the
> whole file
The code you posted *does* slurp the whole file.
> foreach my $line (<FILE1>) {
> chomp $line;
> (do something...)
> }
"foreach" is documented in perlsyn as "LABEL foreach VAR (LIST) BLOCK".
That means that the contents of the parenthesis is evaluated in list
context. In list context, <FILE1> returns an array of all the
previously-unread lines in the file. In other words, what happens above
is that the entire contents of the file referred to by FILE1 is slurped
into a temporary array, and then each item in the array is assigned to
$line in turn.
To read the file a line at a time, you need to use <FILE1> in scalar
context, something like this:
while (my $line = <FILE1>) {
chomp $line;
(do stuff...)
}
> Is there a way to force Perl to not slurp up the whole file at once
Yes - don't ask it to. ;-)
sherm--
Chuck Rice - 07 May 2004 20:29 GMT
How do I set up a perl program such as the one below to check mail
without hard coding the password in the code? I want to run the job
automatically via cron, but I I do not like the idea of leaving the
password lying around in clear text. I am pretty new to Perl as you
can see from the quality of the code, but it seems the right choice
for this application, except for the passwords. -Chuck-
#!/usr/bin/perl
#------------------------------------------------------
# Read LetterRip Daily Statistics email and process
# bounce addresses.
#
# Written April 2004 by Chuck Rice
#------------------------------------------------------
use Mail::Mailer;
use Net::POP3;
use strict;
my $mail_server = "mail.wildrice.com";
my $username = "xxxxxxxx";
my $password = "xxxxxxxx";
my $pop;
my $line;
my @bouncer;
my $listMsgCnt;
my @fields;
my $mailer;
my $from_address = "Chuck\@WildRice.com";
my $to_address = "requests\@lists.WildRice.com,Chuck\@WildRice.com";
my $subject = "Unsubscribe Bouncers";
my $body = "user chuck\@wildrice.com\rpass xxxxxxxx\n";
$pop = Net::POP3->new($mail_server)
or die "Can't open connection to $mail_server : $!\n";
defined ($pop->login($username, $password))
or die "Can't authenticate: $!\n";
my $messages = $pop->list
or die "Can't get list of undeleted messages: $!\n";
foreach my $msgid (keys %$messages) {
print $msgid . "\r";
my $message = $pop->get($msgid);
unless (defined $message) {
warn "Couldn't fetch $msgid from server: $!\n";
next;
}
# $message is a reference to an array of lines
$pop->delete($msgid);
foreach $_ (@$message) {
if (/Message posts/) {
@fields = split(/\s+/, $_);
$listMsgCnt = $fields[2];
}
if (/unsub \"/) {
my @fields = split(/\s+/, $_);
$fields[$#fields+1] = 0;
$fields[$#fields+1] = 0;
$fields[$#fields+1] = 0;
$fields[$#fields+1] = 0;
$fields[$#fields+1] = 0;
$fields[$#fields+1] = 0;
my $failCnt = $fields[5];
if ($listMsgCnt <= 0 . $failCnt) {
$bouncer[$#bouncer+1] = $_;
}
}
}
}
foreach $line (@bouncer) {
print $line;
$body .= $line;
}
$mailer = Mail::Mailer->new("sendmail");
$mailer->open({ From => $from_address,
To => $to_address,
Subject => $subject
})
or die "Can't open: $!\n";
print $mailer $body;
$mailer->close();
print "Done";

Signature
Fight Spam! Join CAUCE! == http://www.cauce.org/
Sherm Pendley - 07 May 2004 22:46 GMT
> How do I set up a perl program such as the one below to check mail
> without hard coding the password in the code?
If you're using Panther, the "security" command-line tool looks like it
might be useful. You can use it to access the keychain from a script.
"man security" for more info.
You'll need to open it as a child task using both input and output; for
more about doing that from Perl, see "perldoc IPC::Open2" and perldoc
IPC::Open3".
sherm--
Chuck Rice - 08 May 2004 00:35 GMT
>>How do I set up a perl program such as the one below to check mail
>>without hard coding the password in the code?
[quoted text clipped - 8 lines]
>
>sherm--
Thanks. I did not know that that existed. I have some reading to do. -Chuck-

Signature
Fight Spam! Join CAUCE! == http://www.cauce.org/
Chuck Rice - 11 May 2004 21:35 GMT
>>How do I set up a perl program such as the one below to check mail
>>without hard coding the password in the code?
[quoted text clipped - 8 lines]
>
>sherm--
Not sure about the child task stuff, but this code seems to get the
password from the keychain, (assuming that you have a keychain entry
called emailpassword). Pretty cryptic, and I am sure beginnerish,
but it works. -Chuck-
#!/usr/bin/perl
my $temp;
my $pass;
$temp = `security find-internet-password -a emailpassword -g 2>&1`;
$temp =~ /password: "([A-Za-z]*)"/;
$pass = $1;
print "$pass";

Signature
Fight Spam! Join CAUCE! == http://www.cauce.org/
Joseph Alotta - 12 May 2004 16:49 GMT
>>> How do I set up a perl program such as the one below to check mail
>>> without hard coding the password in the code?
[quoted text clipped - 22 lines]
>
> --
I would think that you can have digits in your password. I would
change it to be
$temp =~ /password: "([A-Za-z0-9]"/;
Joe.
------------------------------------------------------------------------
--------------
Joseph Alotta, Principal
(630) 969-2628
Open Door Investment Advisors, Inc.
www.opendoorinvestments.com
409 North Washington Street
certifiedfinancialplanner_at_earthlink_dot_net
Westmont, IL 60559 USA qualified,
professional, friendly
------------------------------------------------------------------------
--------------
Chuck Rice - 12 May 2004 17:19 GMT
>>>>How do I set up a perl program such as the one below to check
>>>>mail without hard coding the password in the code?
[quoted text clipped - 26 lines]
>change it to be
>$temp =~ /password: "([A-Za-z0-9]"/;
Yes, and special characters too. But is the '*' not needed? -Chuck-

Signature
Fight Spam! Join CAUCE! == http://www.cauce.org/
Chris Nandor - 13 May 2004 21:50 GMT
> If you're using Panther, the "security" command-line tool looks like it
> might be useful. You can use it to access the keychain from a script.
> "man security" for more info.
Am I missing something, or does the security(1) password prompt not hide
your typed password? Yeesh!
$ security unlock-keychain
password to unlock default: adsdasd^C

Signature
Chris Nandor pudge@pobox.com http://pudge.net/
Open Source Development Network pudge@osdn.com http://osdn.com/
Morbus Iff - 13 May 2004 21:51 GMT
>Am I missing something, or does the security(1) password
>prompt not hide your typed password? Yeesh!
>
> $ security unlock-keychain
> password to unlock default: adsdasd^C
Next thing we know, this'll be possible:
$ security unlock-keychain adsdasd

Signature
Morbus Iff ( i put the demon back in codemonkey )
Culture: http://www.disobey.com/ and http://www.gamegrene.com/
Spidering Hacks: http://amazon.com/exec/obidos/ASIN/0596005776/disobeycom
icq: 2927491 / aim: akaMorbus / yahoo: morbus_iff / jabber.org: morbus