How to Cache CRAM Reference Sequences using Squid
To reduce network traffic originating from the use of the CRAM Reference Registry we recommend using locally cached reference sequences. In addition to local caches supported by Samtools it is possible to cache sequences using an HTTP proxy.
In the example below, the Squid is used as a reverse proxy to cache reference sequences retrieved from the CRAM Reference Registry.
Setting up Squid reverse proxy
Download and install Squid.
If a binary package is not available for download then Squid can also be compiled from source. We recommend that you install version 3.1 or later.
Create directories for the cache files, log files and the Squid configuration file.
In the example below we have created directories under /data/squid but any directory accessible to the user running Squid can be used after it has been defined in the Squid configuration file.
Used to store the local squid cache
Used to store the Squid log files
Used to store the Squid configuration file
Create the Squid configuration file.
The following configuration file in /data/squid/conf will cache locally reference sequences retrieved from the CRAM Reference Registry. Parameter values that should be configured locally are bolded.
maximum_object_size_in_memory 512 MB
maximum_object_size 1024 MB
http_port <port> accel defaultsite=<hostname> vhost
cache_peer www.ebi.ac.uk parent 80 0 no-query originserver name=myAccel
cache_dir ufs /data/squid/cache 65536 16 256
acl mysites dstdomain ebi.ac.uk
acl all src all
http_access allow mysites
http_access allow all
cache_peer_access myAccel allow mysites
cache_peer_access myAccel allow all
The HTTP port the Squid server is configured to use.
Please note that ports below 1024 require root privileges.
Must match the hostname of the machine running the Squid server.
The amount of memory reserved for Squid cache. It is
recommended that this does not exceed 70% of the available
physical memory. If the number of objects that is being cached
exceeds the available memory then the cache will write the
retrieved reference sequences to the Squid cache directory.
Execute the following command:
squid -f /data/squid/conf/squid.conf -z
Start up Squid.
Execute the following command:
squid -f /data/squid/conf/squid.conf
This will start the squid process as a daemon so that it runs in the background. If you wish, you can execute squid in the foreground by adding the parameter -N.
Configuring Samtools to use Squid.
In Samtools, the REF_CACHE environment variable is used to indicate that any downloaded reference sequences are stored locally in the specified directory in order to avoid subsequent downloads.
If you wish SAMTools to cache retrieved sequences in addition to caching them in Squid then you should define both REF_PATH and REF_CACHE environment variables:
where <hostname> and <port> are the ones used by the Squid server, and the <dir> is the directory path used by SAMTools for its cache. The %2s/%2s/%s syntax informs SAMTools to store the sequences in sub-directories first using the first two digits and then using the second two digits of the MD5 checksum. This helps to avoid one large directory with thousands of files in it.
If you wish to cache reference sequences only in the Squid cache then only the REF_PATH environment variable should be defined:
Configuring CRAMToolkit to use Squid.
CRAMTools can be instructed to use the Squid cache by defining the following Java property:
where <hostname> and <port> are the ones used by the Squid server.
Execute the following command to test the cache:
The computed md5 checksum should match the requested entry. The cache status of the request can be determined by checking the /data/squid/log/access.log file. The cache status is available in the fourth column of the access log and can be one of the following:
Shut down Squid.
To gracefully shut down the squid daemon process, execute the following command:
squid -k shutdown