Auto Download Latest Linux Code

In Linux, there are a number of good package managers. From a build it yourself standpoint, its very difficult to stay up-to-date on all the newest packages. One of the nice features about the http://ftp.gnu.org/gnu is it uses a very similar folder scheme for all of its applications. With a few excepts, the below code (which is still being changed and modified on a daily basis) will create an output file with the download links for many of the main linux applications. It is not complete, but provides a very nice foundation for anyone who does not want to spend a lot of their time looking for new updates. At the bottom of the page is another script which will download all the files from the output file generated from the first script.

#!/bin/bash
### ——————————————————————– ###

# This script creates a text file with a list of the latest
# source downloads for Linux.

# Location: <a href="http://virtualparadise.wordpress.com/2010/04/29/auto-download-latest-linux-code">http://virtualparadise.wordpress.com/2010/04/29/auto-download-latest-linux-code</a>
# Date Created: 4/29/2010
# Date Modified: 4/29/2010

# Permissions Required: Non-root
# Notes: Not every link will be valid or the latest, double
# check before relying on the code

### ——————————————————————– ###

cd ~/Desktop

function getFile {
SORTV="?C=M;O=D"
SITE=http://ftp.gnu.org/gnu/$1/
PAGE=""
FNAME=""

# GNU site, download is in the child folder
if [[ $2 = 1 || $2 = 2 ]]; then
wget -q "$SITE$SORTV"
PAGE=$(cat $HTM | cut -d""" -f 6 | grep $1 | grep "tar.bz2" | grep -v ".sig" | head -n 1)
COUNTV=$(cat $HTM | cut -d""" -f 6 | grep $1 | grep "tar.bz2" | grep -v ".sig" | head -n 1 | grep "." -c)
if [ $COUNTV = "0" ]; then
PAGE=$(cat $HTM | cut -d""" -f 6 | grep $1 | grep "tar.gz" | grep -v ".sig" | head -n 1)
fi

FNAME=""
fi

# GNU site, download is in the grandchild folder
if [ $2 = 2 ]; then
PAGE=$(cat *.htm* | grep "/"" | cut -d""" -f 6 | grep $1 | head -n 1)
FNAME=$(echo $PAGE | cut -d"/" -f 1)
FNAME="$FNAME.tar.bz2"
fi

if [[ $2 = 1 || $2 = 2 ]]; then
rm $HTM
fi

# Site is passed in variable,
if [ $2 = 3 ]; then
wget -q $3
HTM="*.htm*"

cat $HTM | tr "[:upper:]" "[:lower:]" &gt; $HTM

if [ $(cat $HTM | grep ".tar.gz" -c) = 0 ]; then
SITE=$(cat $HTM | grep ".tar.bz2"" | head -n 1)
else
SITE=$(cat $HTM | grep ".tar.gz"" | head -n 1)
fi
SITEINDEX=$(awk -v a="$SITE" -v b="href" 'BEGIN{print index(a,b)}')
SITE=${SITE:$SITEINDEX:100}
SITE=$(echo $SITE head -n 1 | cut -d""" -f 2)
# SITE=$(cat $HTM | grep ".tar.gz"" | head -n 1 | cut -d""" -f 2)

if [[ ${SITE:0:1} = "/" || ${SITE:0:4} != "http" ]]; then
SITEREV=$(echo $3 | rev)
SITEINDEX=$(expr index "$SITEREV" "/")
SITELEN=$(expr length $3)
ROOTSITE=${3:0:SITELEN-SITEINDEX+1}
SITE=$ROOTSITE$SITE
fi

rm $HTM
fi

if [ $2 = 4 ]; then
SITE=$3
fi

echo "$SITE$PAGE$FNAME" &gt;&gt; link.txt
}

&gt; link.txt

function blah {
echo "p"
}
# Latest Stable Kernel
wget -q http://www.kernel.org
head index.html -n 100 | grep ".tar.bz2" | cut -d """ -f2 | head -n 1 &gt;&gt; link.txt
HTM="*.htm*"
rm $HTM

##NOTE TO SELF, remove the GLIBC locals that I don't want in the scripts
## Look at adjusting the toolchains in the LFS book for a better understanding of the libs!!
# Check out packages.qa.debian.org!!!
# Check bison internationalization settings - turn off

## Very minimal Linux Distrobution
# Ability to boot into commandline and connect to the internet

# Compile only the needed core packages
# Download the already compiled packages
# Weight the size vs the amount of usage
# Maybe follow the LSB Requirements

## Packages
# Linux Kernel
# GRUB
# UDev
# Boot scripts
# Bash
# Apt-Get
# Logging

################
### LSB Core ###
################

# Bash shell
getFile bash 1

# Linker and assembler
getFile binutils 1

# Utilities to show and set the system characteristics
getFile coreutils 1

# Utilities to show file differences
getFile diffutils 1

# Determines file types
#getFile file 3 ftp://ftp.astron.com/pub/file/ ## USE TAIL FOR THIS ONE

# Programs to find files
getFile findutils 1

# Utilites to manipulate text files
getFile gawk 1

# Searches through files
getFile grep 1

# Compression utilities
getFile gzip 1

# Macro processor
getFile m4 1

### NEED man-db

# Lbbraries for terminal-independent handling of characters
getFile ncurses 1

# Process monitoring
getFile procps 3 http://procps.sourceforge.net/download.html

### NEED psmisc

# Stream editor
getFile sed 1

### NEED shadow

# Tape archives
getFile tar 1

# File system handling, consoles, paritions, and messages
getFile util-linux-ng 3 http://www.kernel.org/pub/linux/utils/util-linux/?O=D

# Compression and decompression routines
getFile zlib 3 http://www.zlib.net/

###############
### LSB C++ ###
###############

# GNU C and C++ Compilers
getFile gcc 2

# Math libraries - required by GCC
getFile gmp 1
# Math libraries - required by GCC
getFile mpfr 1

#####################
### LSB Languages ###
#####################

# Practical Extraction and Report Language
#getFile perl 4 http://www.cpan.org/src/latest.tar.bz2

###################
### Test Suites ###
###################
# TCL Shell
getFile tcl 3 http://www.tcl.tk/software/tcltk/download.html
# Carries out scripted dialogues with other applications
getFile expect 4 http://expect.nist.gov/expect.tar.gz
# Framework of tests for applications (needs Expect)
getFile dejagnu 1
# GNU generic library support script - needed by test suites
getFile libtool 1

##############
### I WANT ###
##############

# Main C Library
getFile glibc 1

# Good compression libraries
getFile bzip2 3 http://bzip.org/downloads.html

# Network services and protocol information
getFile iana-etc 3 http://sethwklein.net/projects/iana-etc/downloads/?S=D
# Basic networking utilities
getFile inetutils 1

# Command-line editing - used by Bash
getFile readline 1

# Handles the file systems ext2, ext3, and ext4
##getFile e2fsprogs ? no chance # ?????? e2fsprogs.sourceforge.net

################
### LFS Need ###
################

# Used to make packages
getFile make 1
# Used to patch applications
getFile patch 1
# Passes the include and library pathes during the configure and make execution
getFile pkg-config 3 http://pkgconfig.freedesktop.org/releases/?C=M;O=D

# Utilities for localization
#getFile gettext 1

# Used to read and write info pages
#getFile texinfo 1

# Manual application and pages
#getFile man-pages http://www.cpan.org/src/latest.tar.bz2 ??

# Parser generator for text files
getFile bison 1

# GNU Database Manager - disk file format, needed by Man-DB
#getFile GDBM 1

# Produces shell scripts - requires M4
#getFile autoconf
# Generates make files for use with Autoconf
#getFile automake

cat link.txt
echo "Total Packages:"; cat link.txt | grep -c ""

Below is the download script. Remove –spider to actually download the file as opposed to just checking if it is available.

#!/bin/bash

#mkdir packages

COUNTER=0

for FILE2 in $(cat link.txt)
do
    wget $FILE2 --spider -a output.txt
    COUNTER=`expr $COUNTER + 1`
done

echo "Total Downloaded: "$(cat output.txt | grep "Remote" -c)
echo "Total Files: "$COUNTER