Page Comparison

Install squid caching server to reduce build time

Building using mock will trigger downloads of many RPMs from CentOS and Fedora mirrors each time the chroot initialized, which is often enough. If you install a caching proxy server, then RPMs are only downloaded once then cached on your local network. This will speed build times up substantially after running through a complete build once.

yum install squid

I recommend these settings in file /etc/squid/squid.conf for optimal caching. You can place these settings at the top of the config file.

# 10G Cache
cache_dir ufs /var/spool/squid 10000 16 256
# Cache large files
maximum_object_size 512000 KB

Then call

service squid start
chkconfig squid on

Finding a proper mirror site

yum is normally configured to selected the optimal site for downloading packages based on speed and on spreading the traffic equally among several mirror sites. When using a download proxy, you'll want to choose a single site for downloading packages that has the proper cache settings, is up reliably and is relatively quick. Be sure you understand why this is true otherwise you risk decreasing your build times and filling your disk cache quickly.

Here are the steps for selecting the proper mirror site.

1.) Check the CentOS mirror sites page for a single site that is in your area. Also consider reliability of the company or institution providing the mirror. Do some investigation on speed of the download speeds from the site.

2.) Test the mirror's cache settings

For a squid server : localhost
CentOS mirror site : http://mirrors.rit.edu/centos

You would run this command twice :

time http_proxy=http://localhost:3128 \
  wget -S http://mirrors.rit.edu/centos/5.5/os/i386/repodata/repomd.xml 2>&1 \
  | grep X-Cache

Where you'll see on the first run

  X-Cache: MISS from swift.hubler.us
  X-Cache-Lookup: MISS from swift.hubler.us:3128

Then on the second run

  X-Cache: HIT from swift.hubler.us
  X-Cache-Lookup: HIT from swift.hubler.us:3128

If you do not see HIT, select a different mirror.

3.) Follow steps 1 and 2 again to select a good fedora mirror fedora mirror list. You'll need to do this even if you're not planning for build on Fedora because the build system will require you to specify a fedora mirror url.
Example:

time http_proxy=http://localhost:3128 \
  wget -S http://ftp.linux.ncsu.edu/pub/fedora/linux/releases/14/Fedora/i386/os/repodata/repomd.xml  2>&1 \
  | grep X-Cache

Building with a download proxy

To tell sipXecs build scripts to use your caching proxy, just add this configure switch when preparing to build sipXecs

./configure ... \
   --with-yum-proxy=http://swift.hubler.us:3128 \
   CENTOS_BASE_URL=http://mirrors.rit.edu/centos \
   FEDORA_BASE_URL=http://mirrors.rit.edu/fedora/linux

You should verify caching is working by your second build by looking for the string TCP_MEM_HIT in /var/log/squid/access.log.

Clearing squid's cache.

If you switch mirror sites or you suspect the caching is causing build issues, you'll want to clear squid's cache. There is no easy way to clear a portion of the cache, but this set of commands will clear all of squid's cache.

service squid stop
rm -rf /var/spool/squid/*
service squid start

Advanced: Using your download proxy for testing installations

You can also use your download proxy server to install systems faster. To use the proxy, edit /etc/yum.conf and add the setting proxy=http://your-squid-server:3128 to the [main] section.

Next you'll want to edit as many of the repository files as you can to use a fixed location for downloading. To do this, comment out the lines starting with mirrorlist= and uncomment out lines #baseurl. To automate editing of repos files to use base urls instead or mirror lists, this command should work.
sed -i -e 's/^mirrorlist/#mirrorlist/g' -e 's/^#baseurl/baseurl/g' /etc/yum.repos.d/*.repo