Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

Install squid caching server to reduce build time

Building using mock will trigger downloads of many RPMs from CentOS and Fedora mirrors each time the chroot initialized, which is often enough. If you install a caching proxy server, then RPMs are only downloaded once then cached on your local network. This will speed build times up substantially after running through a complete build once.

yum install squid

I recommend these settings in file /etc/squid/squid.conf for optimal caching 

# 10G Cache
cache_dir ufs /var/spool/squid 10000 16 256
# Cache large files
maximum_object_size 512000 KB

Then call

service squid start
chkconfig squid on

Finding a proper mirror site

yum is normally configured to selected the optimal site for downloading packages based on speed and on spreading the traffic equally among several mirror sites. When using a download proxy, you'll want to choose a single site for downloading packages that has the proper cache settings, is up reliably and is relatively quick. Be sure you understand why this is true otherwise you risk decreasing your build times and filling your disk cache quickly.

Here are the steps for selecting the proper mirror site.

1.) Check the CentOS mirror sites page for a single site that would be best for a mirror site for you.

2.) Test the mirror's cache settings for that site are set properly.

For a squid server : localhost
CentOS mirror site : http://mirrors.rit.edu/centos

You would run this command twice :

http_proxy=http://localhost:3128 \
  wget -S http://mirrors.rit.edu/centos/5.5/os/i386/repodata/repomd.xml 2>&1 \
  | grep X-Cache

Where you'll see on the first run

  X-Cache: MISS from swift.hubler.us
  X-Cache-Lookup: MISS from swift.hubler.us:3128

Then on the second run

  X-Cache: HIT from swift.hubler.us
  X-Cache-Lookup: HIT from swift.hubler.us:3128

If you do not see HIT, select a different mirror.

3.) Follow steps 1 and 2 again to select a good fedora mirror fedora mirror list. You'll need to do this even if you're not planning for build on Fedora because the build system will require you to specify a fedora mirror url.

Building with a download proxy

To tell sipXecs build scripts to use your caching proxy, just add this configure switch when preparing to build sipXecs 

./configure ... \
   --with-yum-proxy=http://swift.hubler.us:3128 \
   CENTOS_BASE_URL=http://mirrors.rit.edu/centos \
   FEDORA_BASE_URL=http://mirrors.rit.edu/fedora/linux

You should verify caching is working by your second build by looking for the string TCP_MEM_HIT in /var/log/squid/access.log.

You can use your new caching server to install systems faster as well. This is handy if you maintain a lot of systems, or test installations a lot. To use the proxy, edit /etc/yum.conf and add the setting proxy=http://your-squid-server:3128 to the [main] section.

Next you'll want to edit as many of the repository files as you can to use a fixed location for downloading. To do this, comment out the lines starting with mirrorlist= and uncomment out lines #baseurl. To automate editing of repos files to use base urls instead or mirror lists, this command should work.
sed -i -e 's/^mirrorlist/#mirrorlist/g' -e 's/^#baseurl/baseurl/g' /etc/yum.repos.d/*.repo

The reason you should use baseurls is so that you are consistently going to the same source for files and squid proxy will know to cache.
There is one issue, you must pick a mirror site that uses cache headers so squid knows caching is valid.

  • No labels