Debugging Mirrorlist

Note: if you’re not interrested in why I wrote my check_metalink script, and just want to use it, go here .

A little while ago, we were having some issues with the Fedora Project’s mirrorlist servers.

A great writeup of what metalink does was written by Kevin , but the very short summary is that it tells yum/dnf what mirrors it can use, and what the checksum is for the latest version of the core repo metadata file (repomd.xml).

The issues we were having presented themselves to the user because the checksum we published in the metalink didn’t match the checksum of the repomd on the master mirror and any of the current mirrors (the checksum in the metalink was outdate). This caused yum/dnf to refuse each of the mirrors, and in the end return an error saying “No mirrors left”, and refusing to run any operations.

Because this was very hard to debug manually every time (it’s a lot of redirections from metalink to mirror and to check checksums), I wrote a small script to automatically check the metalink that our servers provide to users against the current contents of the master mirror (it was a modification of a script I wrote earlier to find mirrors shipping invalid repo files).

It is this script I would like to give some overview on, as it is also useful for other people to run when reporting issues to us at Fedora Infrastructure, since it’ll give a lot of information that we can use to debug your issues.

Using it

So, how do you use this script?

First off, grab the script itself: just git clone https://github.com/puiterwijk/check_metalink .

Next, make sure you have python-requests installed, and if you want to check FTP mirrors, also python-requests-ftp (not available on EPEL for now). I might remove this dependency at some point in the future, but for now it was the easiest way for me to get this working.

Now you are ready to run the script! So just cd into the directory, and run ./check_metalink.

If you run it just like this, with no additional arguments, it will start testing all mirrors for all repos for your currently installed system. It will report for each mirror whether or not it is “Working” or “Failed”. Working means that the script was able to retrieve repomd.xml from that mirror, and the checksum matches one of the checksums in the metalink (in other words, yum/dnf would be happy to use this mirror). Failed means that either it was unable to get repomd.xml, or the checksum didn’t match.

After testing all of the mirrors, it will print a short overview of OK, Bad, and “Exception” metalinks.

• OK means that at least one mirror passed, and you should be able to update.
• Bad means that all of the mirrors failed (please report this to the infra team).
• “Exception” means that something went wrong when checking the repo, most likely the metalink failed.

The script will by default be in “info verbosity” mode, which will make it also print the appserver that returned the metalink used, the date when it retrieved it according to the server, and your own remote internet IP, these points of data are very useful for the Fedora infrastructure team to debug your specific issue, so please make sure to include them when reporting an issue.

In case the script marks all of the repos as OK, but you are still unable to update, that might mean some of the mirrors have a valid repomd, but have errors in the rest of the repo metadata (this happens for all sorts of weird things, don’t ask). The script can even check that! Just add –check-files on the end of the command line, and it will check a lot more thorough whether all of the files referenced by repomd.xml are valid.

So, next time you report any issues at Fedora Infrastructure regarding being unable to update, it would be much appreciated if you could run this script at least one time and give us the output of it, as it will make it a lot easier for us to determine whether the issue is related to your local mirror, or if we need to get to work!