Script Hat

Here is an example for using perl and curl to retrieve files from a website with authentication and a self signed certificate (curl -k). It also uses the Date::Manip module to figure out how to get todays file from the directory it is downloading from (the files have the date as part of their naming convention). Date Manip is not necessary for this but it does a good job and makes figuring dates easy. This script is ran from a cronjob daily and pulls a compressed log file in the format of filename020410.log.gz.

use strict;
use warnings;
use Date::Manip;

my $site = 'https://username:password@192.168.0.7/logs/'; # page to check with username and password
my $fileName = "Filename";                                           # file names to look for

my $today = ParseDate('today');
print "Looking for new files for Today ($today)\n";
my $day = (substr$today, 6,2);
my $mo = (substr$today, 4,2);
my $year = (substr$today, 2,2);
my $i = 0;
my $file;
my @origCurl = `curl -k $site 2>&1`;                              # Get a list of files
foreach (@origCurl)  {
        $i++;
        if ($_ =~ />($fileName.*\.log\.gz)<\/a>/) {              # If we get a match download it
                $file = $1;
                if ($file =~ /$mo$day$year/)  {                      # Today's files
                        print "Getting $file\n";
                        my $Curl2nd = `curl -k -O $site$file`;
                }
        }
}