How to Cache CRAM Reference Sequences using Squid

To reduce network traffic originating from the use of the CRAM Reference Registry we recommend using locally cached reference sequences. In addition to local caches supported by Samtools it is possible to cache sequences using an HTTP proxy.

In the example below, the Squid is used as a reverse proxy to cache reference sequences retrieved from the CRAM Reference Registry.

Setting up Squid reverse proxy

  1. Download and install Squid.

If a binary package is not available for download then Squid can also be compiled from source. We recommend that you install version 3.1 or later.

  1. Create directories for the cache files, log files and the Squid configuration file.

In the example below we have created directories under /data/squid but any directory accessible to the user running Squid can be used after it has been defined in the Squid configuration file.

Directory

Description

/data/squid/cache

Used to store the local squid cache

/data/squid/log

Used to store the Squid log files

/data/squid/conf

Used to store the Squid configuration file

  1. Create the Squid configuration file.

The following configuration file in /data/squid/conf will cache locally reference sequences retrieved from the CRAM Reference Registry. Parameter values that should be configured locally are bolded.

max_filedesc 4096
cache_mem <memory>
maximum_object_size_in_memory 512 MB
maximum_object_size 1024 MB
http_port <port> accel defaultsite=<hostname> vhost
#forwarded_for on
cache_peer www.ebi.ac.uk parent 80 0 no-query originserver name=myAccel
cache_dir ufs /data/squid/cache 65536 16 256
acl mysites dstdomain ebi.ac.uk
acl all src all
http_access allow mysites
http_access allow all
cache_peer_access myAccel allow mysites
cache_peer_access myAccel allow all
access_log /data/squid/log/access.log
cache_store_log /data/squid/log/store.log
cache_log /data/squid/log/cache.log

Parameter

Example

<port>

8080
The HTTP port the Squid server is configured to use.
Please note that ports below 1024 require root privileges.

<hostname>

Must match the hostname of the machine running the Squid server.

<memory>

10048 MB
The amount of memory reserved for Squid cache. It is
recommended that this does not exceed 70% of the available
physical memory. If the number of objects that is being cached
exceeds the available memory then the cache will write the
retrieved reference sequences to the Squid cache directory.

  1. Initialize Squid.

Execute the following command:

squid -f /data/squid/conf/squid.conf -z

  1. Start up Squid.

Execute the following command:

squid -f /data/squid/conf/squid.conf

This will start the squid process as a daemon so that it runs in the background. If you wish, you can execute squid in the foreground by adding the parameter -N.

  1. Configuring Samtools to use Squid.

In Samtools, the REF_CACHE environment variable is used to indicate that any downloaded reference sequences are stored locally in the specified directory in order to avoid subsequent downloads.

If you wish SAMTools to cache retrieved sequences in addition to caching them in Squid then you should define both REF_PATH and REF_CACHE environment variables:

where <hostname> and <port> are the ones used by the Squid server, and the <dir> is the directory path used by SAMTools for its cache. The %2s/%2s/%s syntax informs SAMTools to store the sequences in sub-directories first using the first two digits and then using the second two digits of the MD5 checksum. This helps to avoid one large directory with thousands of files in it.

If you wish to cache reference sequences only in the Squid cache then only the REF_PATH environment variable should be defined:

export REF_PATH=http://<hostname>:<port>/ena/cram/md5/%s

  1. Configuring CRAMToolkit to use Squid.

CRAMTools can be instructed to use the Squid cache by defining the following Java property:

java -DREF_URL_TEMPLATE=http://<hostname>:<port>/ena/cram/md5/%s

where <hostname> and <port> are the ones used by the Squid server.

  1. Testing Squid.

Execute the following command to test the cache:

The computed md5 checksum should match the requested entry. The cache status of the request can be determined by checking the /data/squid/log/access.log file. The cache status is available in the fourth column of the access log and can be one of the following:

  1. Shut down Squid.

To gracefully shut down the squid daemon process, execute the following command:

squid -k shutdown