Serve rpm repository from AWS S3

Some notes on how packages.zonalivre.org was setup to be serve custom rpm packages from an AWS S3 bucket, mostly based on http://blog.celingest.com/en/2014/09/17/create-your-own-yum-rpm-repository-using-amazon-s3/ .

Make sure the aws cli is setup on the workstation:

mkdir -p ~/.aws
cat <<EOF >~/.aws/config
[default]
region = us-east-1
EOF

cat <<EOF >~/.aws/credentials
[default]
aws_access_key_id = AWS_ACCESS_KEY
aws_secret_access_key = AWS_SECRET_ACCESS_KEY
EOF

From the workstation, create a new S3 bucket. If you intend to use your own domain name, it’s important that the bucket has the same name as the domain you intend to use.

Also create a user (to upload files to the bucket) and give this user access to your S3 buckets.

aws s3 mb s3://packages.zonalivre.org
aws iam create-user --user-name zonalivre-rpm-repo
aws iam create-access-key --user-name zonalivre-rpm-repo
cat <<EOF >zonalivre-rpm-repo_policy.json
{
    "Version": "2012-10-17",
    "Statement": [
    {
        "Effect": "Allow",
        "Action": "s3:*",
        "Resource": ["arn:aws:s3:::packages.zonalivre.org/*", "arn:aws:s3:::packages.zonalivre.org"]
    },
    {
        "Effect": "Allow",
        "Action": "s3:ListAllMyBuckets",
        "Resource": "*",
        "Condition": {}
    }
    ]
}
EOF
aws iam put-user-policy --user-name zonalivre-rpm-repo --policy-name zonalivre-rpm-repo-bucket-access --policy-document file://zonalivre-rpm-repo_policy.json

From the workstation, check user is configured correctly

aws iam list-user-policies --user-name zonalivre-rpm-repo
{
    "PolicyNames": [
        "zonalivre-rpm-repo-bucket-access"
    ]
}

On the S3 console, click “Properties”->”Static Website Hosting”->”Enable static website hosting”. You’ll also need to provide an Index document. Just enter index.html.

Create a DNS alias to point to the new bucket. After this is setup and propagated, it should look like:

host packages.zonalivre.org
packages.zonalivre.org is an alias for packages.zonalivre.org.s3.amazonaws.com.
packages.zonalivre.org.s3.amazonaws.com is an alias for s3-directional-w.amazonaws.com.
s3-directional-w.amazonaws.com is an alias for s3-1-w.amazonaws.com.
s3-1-w.amazonaws.com has address 54.231.1.137

Install required packages on build server. In this particular case, packages are installed as per chef recipe https://github.com/joaocosta/chef-repo/blob/master/cookbooks/fx-build/recipes/default.rb .

The important one here is s3cmd, available as part of the epel repo.

On the build server, configure s3cmd

s3cmd --configure
...
s3cmd ls
2016-01-24 09:03  s3://packages.zonalivre.org

Still on the build server, generate a public/private GPG key pair.
See https://technosorcery.net/blog/2010/10/10/pitfalls-with-rpm-and-gpg/ for a number of gotchas around using gpg keys to sign rpms. Some of the bugs described in the article may have been fixed by now.

gpg --gen-key
# Choose signing only RSA key
# Your key cannot have any subkeys
# Your key must be > 1024-bit (i used 2048)

gpg needs randomness to generate keys. The more entropy there is in your system, the quicker the keys will be generated. You can check how much entropy is available with:

watch -n 1 cat /proc/sys/kernel/random/entropy_avail

There are multiple ways of generating additionally entropy so that key generation happens quicker, such as moving your mouse around, hitting the keyboard, or using specialized software. A quick and dirty way of generating entropy is to run this in a separate shell:

find / -type f | egrep -v "(/dev|/proc|/sys/kernel)" | xargs md5sum

Once key generation completes, list generated keys:

gpg --list-keys
/root/.gnupg/pubring.gpg
------------------------
pub   2048R/859031CB 2016-01-24
uid                  Builder <packages@zonalivre.org>

And export the public key:

gpg --output ~/RPM-GPG-KEY-zonalivre-rpm-repo --armor --export 859031CB

Add gpg rpm macros and remember to replace YOUR_GPG_KEY_ID .
In my case, this is 859031CB.

cat <<EOF >~/.rpmmacros
%signature gpg
%_gpg_path $HOME/.gnupg
%_gpg_name <YOUR_GPG_KEY_ID>
%_gpg_bin /usr/bin/gpg
%packager Builder <packages@zonalivre.org>
%_topdir $HOME/rpmbuild
EOF

Now import GPG public key into rpm

rpm --import ~/RPM-GPG-KEY-zonalivre-rpm-repo

Confirm key has been imported into rpm

rpm -q gpg-pubkey --qf '%{name}-%{version}-%{release} --> %{summary}\n' | grep zona
gpg-pubkey-859031cb-56a4c669 --> gpg(Builder <packages@zonalivre.org>)

Create the .repo file

cat <<EOF >~/zonalivre-rpm.repo
[zonalivre-rpm-repo]
name=name=Extra Packages from Zonalivre RPM Repository -
baseurl=https://packages.zonalivre.org/zonalivre/$releasever/$basearch/
enabled=1
gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zonalivre-rpm-repo
gpgcheck=1
EOF

Create rpmbuild tree

rpmdev-setuptree

To package this repo as an rpm, create a spec file

cat <<EOF >~/rpmbuild/SPECS/zonalivre-rpm.spec
# Zonalivre RPM Repository configuration files and GPG key
%define name zonalivre-rpm-repo
%define version 1
%define release 0.1
%define buildroot %{_topdir}/%{name}-%{version}-root
BuildArch:  noarch
BuildRoot:  %{buildroot}
Summary:    Zonalivre RPM Repository
License:    MIT
Name:       %{name}
Version:    %{version}
Release:    %{release}
Group:      Development/Tools

%description
Package containing Zonalivre RPM Repository configuration files and GPG key.

%prep
exit 0

%build

%install
rm -rf $RPM_BUILD_ROOT
mkdir -p $RPM_BUILD_ROOT%{_sysconfdir}/yum.repos.d/
mkdir -p $RPM_BUILD_ROOT%{_sysconfdir}/pki/rpm-gpg/
cp -p ~/zonalivre-rpm.repo $RPM_BUILD_ROOT%{_sysconfdir}/yum.repos.d/
cp -p ~/RPM-GPG-KEY-zonalivre-rpm-repo $RPM_BUILD_ROOT%{_sysconfdir}/pki/rpm-gpg/

%clean
rm -rf $RPM_BUILD_ROOT

%files
%defattr(-,root,root,-)
/etc/yum.repos.d/zonalivre-rpm.repo
/etc/pki/rpm-gpg/RPM-GPG-KEY-zonalivre-rpm-repo

%changelog
* Sun Jan 24 2016 Builder <builder@zonalivre.org> 1.0.1
- First release.
EOF

Create RPM

cd ~/rpmbuild/SPECS/
rpmbuild -v -ba --sign --clean zonalivre.spec

Verify package signature

rpm --checksig ~/rpmbuild/RPMS/noarch/zonalivre-rpm-repo-1-0.1.noarch.rpm 
/root/rpmbuild/RPMS/noarch/zonalivre-rpm-repo-1-0.1.noarch.rpm: rsa sha1 (md5) pgp md5 OK

Create and populate the final repository structure

mkdir -vp ~/packages.zonalivre.org/zonalivre/7/{x86_64,noarch}
cp ~/RPM-GPG-KEY-zonalivre-rpm-repo ~/packages.zonalivre.org/zonalivre/
cp -rv ~/rpmbuild/RPMS/* ~/packages.zonalivre.org/zonalivre/7/
for repo in ~/packages.zonalivre.org/zonalivre/7/{x86_64,noarch}; do
  createrepo -v --deltas ${repo}/
done

Sync the repository structure to AWS S3:

s3cmd -P sync ~/packages.zonalivre.org/ s3://packages.zonalivre.org/

And finally, our repo can be installed in a client:

yum localinstall http://packages.zonalivre.org/zonalivre/7/noarch/zonalivre-rpm-repo-1-0.1.noarch.rpm

References:

Serve rpm repository from AWS S3