JIGDO - create SME Server ISOs

	Skill level: Developer
	Risk of inconsistencies with Koozali SME Server methodology, upgrades & functionality is high. One must be knowledgeable about how changes impact their Koozali SME Server. Significant risk of irreversible harm.

jigdo

Introduction

jigdo is a method of downloading CD and DVD images over the Internet (and other large files) and is used by Debian and Ubuntu when new releases are created. jigdo files contain a list of the files that need to be included in the new CD or DVD ISO image and you can get the files from local sources and over the Internet. This means when you want to create ISO images for updated versions of Debian, you only need to download the changed files. This saves both bandwidth and and time.

jigdo's approach is unique due to the way it identifies the small pieces that the large file consists of: For instance, if the file is a CD image, the individual files on the CD are contained somewhere within the image file. jigdo-file is capable of finding these files, so if copies of the individual files on the CD are stored separately on the server, jigdo-lite can be made to download these files one by one, and assemble the CD image on the fly on the user's computer. This scheme has several advantages:

• The large image does not need to be stored on the server, instead only the many small files contained in the image. This works with CD images, DVD images (both ISO9660 and UDF format), uncompressed zip files, tar archives... jigdo-file is format independent.
• In spite of the above, jigdo creates a bit-exact copy of the original image on the user's machine. (To achieve this, the directory data, boot block etc. of the image is stored in a special .template file which is distributed alongside the .jigdo file.)
• There is full control over where jigdo-lite will download the individual parts. It is possible to define mirrors so users can choose their nearest mirror.
• jigdo relies on standard HTTP/FTP, no support for special protocols like rsync, and no installation of special software is required on the mirrors.
• jigdo-lite supports resuming aborted downloads, or continuing the download with another mirror if the current one proves slow.
• It is possible to "upgrade" a CD image: If a new version of an image is released, jigdo-lite can download only the data that has changed, the remaining data is read from the outdated version.
• It is possible to release modified versions: If someone distributes a CD image and you have made a few small modifications, e.g. added some files, then you only need to upload the data for your modifications to your webspace. All the data from the original CD image is still fetched from the original site.

If you have read this far, you may be confused now, asking yourself what jigdo does and doesn't do. :-) It seems that the jigdo concept is a bit difficult to grasp at first. If you're still interested, have a look at the respective section of the HOWTO and at the Examples section of the jigdo-file manual.

Installing jigdo on CentOS and RHEL

You need to download and install the correct RPM file from http://dag.wieers.com/rpm/packages/jigdo/ . The choices currently available are for RHEL/CentOS 1 through 5 (they're the ones labelled eg jigdo-0.7.3-1.el5.rf.i386.rpm where the text in bold indicates Enterprise Linux 5), Red Hat 9, and Fedora 1 through 3. There are also source RPMs available.

Download the file first, and then use yum from the command line to install it running either as the root user, or using the sudo command:

Downloading files for CentOs 5 workstation: you can see this folder http://pkgs.repoforge.org/jigdo/

wget http://dag.wieers.com/rpm/packages/jigdo/jigdo-0.7.3-1.el5.rf.i386.rpm
yum install jigdo-0.7.3-1.el5.rf.i386.rpm
[Note: download the el4 files for CentOs 4 (SME7) if you do the job on this platform...]
ok

Generic Use of jigdo by mirroring repository on SME Server

This is generic instruction when you need to build an iso for test purpose on a SME Server (what else). Before formally releasing an iso on SME Server mirrors, the developers work on a pre-release to test and debug it, so you have to use jigdo to create the iso, the aim being to test it .

First you have to verify that jigdo is installed on your SME server, see instructions above or below for debian.

• If you want to build an iso of our marvellous work, you need to download packages from our mirrors. Firstly, make a couple of folders to organize your work (choose the right SME Server needed) :

mkdir -p /root/jigdo/sme9

or for SME Server 8

mkdir -p /root/jigdo/sme8

• Download all contents from the repos (see below), best to make a cup of coffee, it can take some time...This is an example of what you have to do to build a sme9.0beta3, you should take a look to the repository to adapt a new url. This command Line can update rpm in your folder, there is no need to download again which is already downloaded.

rsync --progress -aHS --partial --delete-after rsync://mir1.ovh.ca/smeserver/releases/testing/9.0beta3/ /root/jigdo/sme9

or for SME Server 8

rsync --progress -aHS --partial --delete-after rsync://mir1.ovh.ca/smeserver/releases/testing/8.1beta3/ /root/jigdo/sme8

• Once finished you need to go to the right folder

cd /root/jigdo/sme9/iso/x86_64/

or for SME Server 8

cd /root/jigdo/sme8/iso/x86_64/

• Perform this command

jigdo-lite --noask --scan ../../smeos/x86_64/ *.jigdo

of course you have to adapt the architecture which you want to build (i386 or x86_64). A known issue is that jigdo fails with a code 3 if an ISO is already present in its current folder.

Getting SME 8.1 Beta 2 on Debian

The mini-HOWTO for jigdo describes it's format and justification for use.

• Download the jigdo file:

mkdir -p /home/dluser/smeserver
wget http://mirror.contribs.org/releases/testing/8.1beta2/smeos/i386/smeserver-8.1beta2-i386.jigdo

• Since it is only beta, edit the Servers section for the SMEmirror parameter to be:

SMEmirror=http://mirror.contribs.org/releases/testing/8.1beta2/smeos/i386/

• When it becomes final, the above step will not be necessary as the /8.1beta2/ folder will become /8/.
• Trying to get it in debian will entail the install of jigdo-file

apt-get install jigdo-file

• Get the ISO using jigdo-lite (yes, not jigdo-file)

jigdo-lite smeserver-8.1beta2-i386.jigdo

• On 2013-10-15, when the ISO was built, it was 727,961,600 bytes in length and had the following MD5:

a509d21d476f66b9d27135599b3e6936  smeserver-8.1beta2-i386.iso

• The end log was

...
...
FINISHED --2013-10-15 16:41:30--
Total wall clock time: 18s
Downloaded: 3 files, 718K in 14s (49.7 KB/s)
Found 3 of the 3 files required by the template
Successfully created `smeserver-8.1beta2-i386.iso'

-----------------------------------------------------------------
Finished!
The fact that you got this far is a strong indication that `smeserver-8.1beta2-i386.iso'
was generated correctly. I will perform an additional, final check,
which you can interrupt safely with Ctrl-C if you do not want to wait.

OK: Checksums match, image is good!

GET SME 8

Create JIGDO directory and move latest ISO there as root:

[root@canopus JIGDO]# pwd
/home/chris/JIGDO
[root@canopus JIGDO]# ls
smeserver-8.0beta7-i386.iso

SME latest jigdo files are here:

http://distro.ibiblio.org/pub/linux/distributions/smeserver/releases/testing/8.0rc1/iso/i386/smeserver-8.0rc1-i386.jigdo

Lets create a mount point:

[root@canopus /]# cd mnt/
[root@canopus mnt]# ls
[root@canopus mnt]# mkdir iso

and mount

[chris@canopus JIGDO]$ mount -o loop smeserver-8.0beta7-i386.iso /mnt/iso

[root@canopus JIGDO]# jigdo-lite

Jigsaw Download "lite"
Copyright (C) 2001-2005  |  jigdo@
Richard Atterer          |  atterer.net
-----------------------------------------------------------------

To resume a half-finished download, enter name of .jigdo file. To start a new download, enter URL of .jigdo file. You can also enter several URLs/filenames, separated with spaces, or enumerate in {}, e.g. `http://server/cd-{1_NONUS,2,3}.jigdo'

Provide URL when requested:

http://distro.ibiblio.org/pub/linux/distributions/smeserver/releases/testing/8.0rc1/iso/i386/smeserver-8.0rc1-i386.jigdo

You can also enter a single digit from the list below to select the respective entry for scanning:

1: /home/chris/JIGDO/smeserver-8.0beta7-i386.template
2: /home/chris/JIGDO/smeserver-8.0beta6-i386.template
3: mnt/iso
4: /mnt/iso

Files to scan: 

Note: Not working because jidgo file points to 8, not 8rc1. Editing the file smeserver-8.0rc1-i386.jigdo for correct path for SMEmirror under the Servers section allows completion of ISO downloading. Should work for final release....