Home | Contact Us | FAQ | Search & Site Map | Link to Us
Sign In | Join | Other 45 Sites in Network
Home
Discussion Groups
General
GeneralPortable MacsHardwareNetworking
Applications
Mac ApplicationsEudoraFirefox / MozillaInternet ExplorerOutlook ExpressMS OfficeEntourageExcelPowerPointWordVirtual PCMedia PlayerOther MS Products
Programming
Mac ProgrammingCodeWarriorPerl
Country Specific
Australian Mac GroupUK Mac Group

Mac Forum / Programming / Perl / May 2004



Tip: Looking for answers? Try searching our database.

"out of memory" error in Panther, not in Jaguar?

Thread view: 
Enable EMail Alerts  Start New Thread
Thread rating: 
Xavier Broccoli - 07 May 2004 07:32 GMT
(sorry if this is a repost, I sent it before
confirming my subscription)

I'm having a memory problem with Perl in Panther that
I didn't have in Jaguar.  When I try to read a large
text file using the line input operator (<>), I get an
"out of memory" error:

*** malloc: vm_allocate(size=8421376) failed (error
code=3)
*** malloc[5576]: error: Can't allocate region
Out of memory!

The strange thing is that I'm not slurping up the
whole file, I'm reading it line by line.  But Perl
seems to be slurping the whole thing into memory
before even processing the first line. But even so,
the file isn't large enough to use up all physical and
virtual memory. I know it's not an issue of using the
wrong linefeed character, as this script works fine in
Jaguar and processes the same text file line-by-line
without a problem.  Here's the structure of my code:

open (FILE1, "file1.txt");   # 300MB Mac text file
$/ = "\r";
foreach my $line (<FILE1>) {
  chomp $line;
  (do something...)
}
close FILE1;

open (FILE2, "file2.txt");  # 300MB Mac text file
$/ = "\r";
foreach my $line (<FILE2>) {
   chomp $line;
   (do something)
}
close FILE2;

Here's how top displays Perl's memory usage after
opening the first 300MB file, on a machine with 2 gigs
of RAM and a hundred gigs of free HD space:

PID COMMAND      %CPU   TIME   #TH #PRTS #MREGS RPRVT
RSHRD  RSIZE  VSIZE
5576 perl        63.2%  2:56.60   1    13   364
1.38G- 1.37M   397M+ 1.57G

After processing all of the lines of the first file
(during which the memory usage increases only
slightly), the script tries to read the second file
and quits with the memory error before it processes
the first line of the second file.

Is this a bug in Perl 5.8.1?  Is there a way to force
Perl to not slurp up the whole file at once, or at
least to release the memory used up by the first file
before reading the second?

Thanks!

XB

   
       
Jarkko Hietaniemi - 07 May 2004 07:41 GMT
> *** malloc: vm_allocate(size=8421376) failed (error
> code=3)
[quoted text clipped - 3 lines]
> The strange thing is that I'm not slurping up the
> whole file,

Oh but you are...

> I'm reading it line by line.  But Perl
> seems to be slurping the whole thing into memory
> before even processing the first line. But even so,

> foreach my $line (<FILE1>) {

This reads in the whole file first because the (...) of
for each enforces list context, and the <FH> in list context
does read in the whole handle.  You meant

while (my $line = <FILE1>) {
   my $line = $_;

or something similar.
Xavier Broccoli - 07 May 2004 07:54 GMT
Oh, you are so right.  I guess Panther is just
stingier with its memory, and I never got bitten by
this before!

Thanks!!

XB

> > *** malloc: vm_allocate(size=8421376) failed
> (error
[quoted text clipped - 24 lines]
>
> or something similar.

   
       
Sherm Pendley - 07 May 2004 07:58 GMT
> The strange thing is that I'm not slurping up the
> whole file

The code you posted *does* slurp the whole file.

> foreach my $line (<FILE1>) {
>    chomp $line;
>    (do something...)
> }

"foreach" is documented in perlsyn as "LABEL foreach VAR (LIST) BLOCK".
That means that the contents of the parenthesis is evaluated in list
context. In list context, <FILE1> returns an array of all the
previously-unread lines in the file. In other words, what happens above
is that the entire contents of the file referred to by FILE1 is slurped
into a temporary array, and then each item in the array is assigned to
$line in turn.

To read the file a line at a time, you need to use <FILE1> in scalar
context, something like this:

while (my $line = <FILE1>) {
    chomp $line;
    (do stuff...)
}

> Is there a way to force Perl to not slurp up the whole file at once

Yes - don't ask it to. ;-)

sherm--
Chuck Rice - 07 May 2004 20:29 GMT
How do I set up a perl program such as the one below to check mail
without hard coding the password in the code? I want to run the job
automatically via cron, but I I do not like the idea of leaving the
password lying around in clear text. I am pretty new to Perl as you
can see from the quality of the code, but it seems the right choice
for this application, except for the passwords. -Chuck-

#!/usr/bin/perl

#------------------------------------------------------
# Read LetterRip Daily Statistics email and process
#   bounce addresses.
#
# Written April 2004 by Chuck Rice
#------------------------------------------------------

use Mail::Mailer;
use Net::POP3;
use strict;

my $mail_server = "mail.wildrice.com";
my $username = "xxxxxxxx";
my $password = "xxxxxxxx";
my $pop;
my $line;
my @bouncer;
my $listMsgCnt;
my @fields;
my $mailer;
my $from_address = "Chuck\@WildRice.com";
my $to_address = "requests\@lists.WildRice.com,Chuck\@WildRice.com";
my $subject = "Unsubscribe Bouncers";
my $body = "user chuck\@wildrice.com\rpass xxxxxxxx\n";

$pop = Net::POP3->new($mail_server)
    or die "Can't open connection to $mail_server : $!\n";
   
defined ($pop->login($username, $password))
    or die "Can't authenticate: $!\n";
   
my $messages = $pop->list
    or die "Can't get list of undeleted messages: $!\n";
   
foreach my $msgid (keys %$messages) {
    print $msgid . "\r";
    my $message = $pop->get($msgid);
    unless (defined $message) {
        warn "Couldn't fetch $msgid from server: $!\n";
        next;
    }
    # $message is a reference to an array of lines
    $pop->delete($msgid);

    foreach $_ (@$message) {

       if (/Message posts/) {
          @fields = split(/\s+/, $_);
          $listMsgCnt = $fields[2];
          }    
     
       if (/unsub \"/) {
          my @fields = split(/\s+/, $_);
          $fields[$#fields+1] = 0;
          $fields[$#fields+1] = 0;
          $fields[$#fields+1] = 0;
          $fields[$#fields+1] = 0;
          $fields[$#fields+1] = 0;
          $fields[$#fields+1] = 0;
          my $failCnt = $fields[5];
          if ($listMsgCnt <= 0 . $failCnt) {
              $bouncer[$#bouncer+1] = $_;
              }
          }  
    }
}
foreach $line (@bouncer) {
    print $line;
    $body .= $line;
    }

$mailer = Mail::Mailer->new("sendmail");
$mailer->open({ From    => $from_address,
                To      => $to_address,
                Subject => $subject
              })
    or die "Can't open: $!\n";
print $mailer $body;
$mailer->close();
print "Done";
Signature

Fight Spam! Join CAUCE! == http://www.cauce.org/

Sherm Pendley - 07 May 2004 22:46 GMT
> How do I set up a perl program such as the one below to check mail
> without hard coding the password in the code?

If you're using Panther, the "security" command-line tool looks like it
might be useful. You can use it to access the keychain from a script.
"man security" for more info.

You'll need to open it as a child task using both input and output; for
more about doing that from Perl, see "perldoc IPC::Open2" and perldoc
IPC::Open3".

sherm--
Chuck Rice - 08 May 2004 00:35 GMT
>>How do I set up a perl program such as the one below to check mail
>>without hard coding the password in the code?
[quoted text clipped - 8 lines]
>
>sherm--

Thanks. I did not know that that existed. I have some reading to do. -Chuck-

Signature

Fight Spam! Join CAUCE! == http://www.cauce.org/

Chuck Rice - 11 May 2004 21:35 GMT
>>How do I set up a perl program such as the one below to check mail
>>without hard coding the password in the code?
[quoted text clipped - 8 lines]
>
>sherm--

Not sure about the child task stuff, but this code seems to get the
password from the keychain, (assuming that you have a keychain entry
called emailpassword).  Pretty cryptic, and I am sure beginnerish,
but it works. -Chuck-

#!/usr/bin/perl  
my $temp;
my $pass;
$temp = `security find-internet-password -a emailpassword -g  2>&1`;
$temp =~ /password: "([A-Za-z]*)"/;
$pass = $1;
print "$pass";    

Signature

Fight Spam! Join CAUCE! == http://www.cauce.org/

Joseph Alotta - 12 May 2004 16:49 GMT
>>> How do I set up a perl program such as the one below to check mail  
>>> without hard coding the password in the code?
[quoted text clipped - 22 lines]
>
> --  

I would think that you can have digits in your password.  I would  
change it to be
$temp =~ /password: "([A-Za-z0-9]"/;

Joe.

------------------------------------------------------------------------
--------------
Joseph Alotta, Principal                                                
(630) 969-2628
Open Door Investment Advisors, Inc.                        
www.opendoorinvestments.com
409 North Washington Street              
certifiedfinancialplanner_at_earthlink_dot_net
Westmont, IL 60559 USA                               qualified,  
professional, friendly
------------------------------------------------------------------------
--------------
Chuck Rice - 12 May 2004 17:19 GMT
>>>>How do I set up a perl program such as the one below to check
>>>>mail without hard coding the password in the code?
[quoted text clipped - 26 lines]
>change it to be
>$temp =~ /password: "([A-Za-z0-9]"/;

Yes, and special characters too. But is the '*' not needed? -Chuck-

Signature

Fight Spam! Join CAUCE! == http://www.cauce.org/

Chris Nandor - 13 May 2004 21:50 GMT
> If you're using Panther, the "security" command-line tool looks like it
> might be useful. You can use it to access the keychain from a script.
> "man security" for more info.

Am I missing something, or does the security(1) password prompt not hide
your typed password?  Yeesh!

 $ security unlock-keychain
 password to unlock default: adsdasd^C

Signature

Chris Nandor                      pudge@pobox.com    http://pudge.net/
Open Source Development Network    pudge@osdn.com     http://osdn.com/

Morbus Iff - 13 May 2004 21:51 GMT
>Am I missing something, or does the security(1) password
>prompt not hide your typed password?  Yeesh!
>
>  $ security unlock-keychain
>  password to unlock default: adsdasd^C

Next thing we know, this'll be possible:

  $ security unlock-keychain adsdasd

Signature

Morbus Iff ( i put the demon back in codemonkey )
Culture: http://www.disobey.com/ and http://www.gamegrene.com/
Spidering Hacks: http://amazon.com/exec/obidos/ASIN/0596005776/disobeycom
icq: 2927491 / aim: akaMorbus / yahoo: morbus_iff / jabber.org: morbus

 
Sign In
Join
My Latest Posts
My Monitored Threads
My Blog
My Photo Gallery
My Profile
My Homepage

Start New Thread
Enable EMail Alerts
Rate this Thread



©2009 Advenet LLC   Privacy Policy - Terms of Use
This website includes both content owned or controlled by Advenet as well as content owned or controlled by third parties.