Engineering Log 2020-06-22

June 22, 2020

These notes were compiled before I chose an accurate date system. Some entries here may predate the time listed above.

Checking nested salt highstate data in python

myjson contains json fomatted output from highstate

>>> for state_id, state_info in myjson.items():
...     for key in state_info:
...         print('key result', state_info[key]['result'])


{
    "opsdevrob99.example.com": {
        "service_|-minion deploy config_|-salt-minion_|-running": {
            "comment": "The service salt-minion is already running", 
            "name": "salt-minion", 
            "start_time": "18:23:17.285884", 
            "result": true, 
            "duration": 33.658, 
            "__run_num__": 2, 
            "__sls__": "minion", 
            "changes": {}, 
            "__id__": "minion deploy config"
        }
    }
}

Centos 8 imaging failures in Koji

This issue was fixed temporarily by setting install tree directly to upstream mirror leaseweb. Anaconda requires relative links for Appstream to work.

06:31:16,027 INFO anaconda:packaging: enabled repo: '' - ['http://repos.example.com/cobbler/repo_mirror/loc-base8/'] and got repomd
06:31:16,027 DEBUG anaconda:packaging: repo koji-override-0: mirrorlist None, baseurl http://opskojihub11.example.com/kojifiles/repos/image-centos-8-build/33/x86_64, metalink False
06:31:16,029 INFO anaconda:packaging: added repo: 'koji-override-0' - http://opskojihub11.example.com/kojifiles/repos/image-centos-8-build/33/x86_64
06:31:16,029 DEBUG anaconda:packaging: repo AppStream: mirrorlist None, baseurl http://repos.example.com/cobbler/repo_mirror/loc-base8/../../../AppStream/x86_64/os/, metalink False
06:31:16,030 INFO anaconda:packaging: added repo: 'AppStream' - http://repos.example.com/cobbler/repo_mirror/loc-base8/../../../AppStream/x86_64/os/
06:31:16,031 DEBUG anaconda:dnf: repo: downloading from remote: koji-override-0
06:31:16,050 INFO anaconda:packaging: enabled repo: '' - ['http://opskojihub11.example.com/kojifiles/repos/image-centos-8-build/33/x86_64'] and got repomd
06:31:16,053 DEBUG anaconda:dnf: repo: downloading from remote: AppStream
06:31:16,063 DEBUG anaconda:dnf: error: Status code: 404 for http://repos.example.com/AppStream/x86_64/os/repodata/repomd.xml (http://repos.example.com/AppStream/x86_64/os/repodata/repomd.xml).
06:31:16,064 DEBUG anaconda:dnf: Cannot download 'http://repos.example.com/cobbler/repo_mirror/loc-base8/../../../AppStream/x86_64/os/': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried.
06:31:16,065 ERR anaconda:dnf: Failed to download metadata for repo 'AppStream'
06:31:16,065 DEBUG anaconda:packaging: repo: '' - ['http://repos.example.com/cobbler/repo_mirror/loc-base8/../../../AppStream/x86_64/os/'] failed to load repomd
06:31:16,066 ERR anaconda:anaconda: payload: PayloadError: Failed to download metadata for repo 'AppStream'

output from failed image build.

This happens after the fix for appstream above

WARNING anaconda:packaging: 
Problem 1: package kernelshark-2.7-8.el8.x86_64 requires libfreetype.so.6()(64bit), but none of the providers can be installed  
	 - cannot install the best candidate for the job  
	 - package freetype-2.9.1-4.el8.x86_64 is excluded

Problem 2: package firewalld-0.7.0-5.el8_1.1.noarch requires firewalld-filesystem = 0.7.0-5.el8_1.1, but none of the providers can be installed
	 - cannot install the best candidate for the job
	 - package firewalld-filesystem-0.7.0-5.el8_1.1.noarch is excluded

Problem 3: package bind-utils-32:9.11.4-26.P2.el8.x86_64 requires libbind9.so.160()(64bit), but none of the providers can be installed  
	 - package bind-utils-32:9.11.4-26.P2.el8.x86_64 requires liblwres.so.160()(64bit), but none of the providers can be installed  
	 - package bind-utils-32:9.11.4-26.P2.el8.x86_64 requires bind-libs(x86-64) = 32:9.11.4-26.P2.el8, but none of the providers can be installed  
	 - package bind-libs-32:9.11.4-26.P2.el8.x86_64 requires bind-license = 32:9.11.4-26.P2.el8, but none of the providers can be installed  
	 - cannot install the best candidate for the job  
	 - package bind-license-32:9.11.4-26.P2.el8.noarch is excluded 

Problem 4: package grub2-pc-1:2.02-78.el8_1.1.x86_64 requires grub2-tools-extra = 1:2.02-78.el8_1.1, but none of the providers can be installed  
	 - package grub2-tools-extra-1:2.02-78.el8_1.1.x86_64 requires libfreetype.so.6()(64bit), but none of the providers can be installed  
	 - cannot install the best candidate for the job  - package freetype-2.9.1-4.el8.x86_64 is excluded

Example kickstart file from CentOS public koji system

#version=DEVEL
# Keyboard layouts
keyboard 'us'
# Root password
rootpw --iscrypted thereisnopasswordanditslocked
# Use network installation
url --url="http://qa.centos.org/centos/8/BaseOS/x86_64/os/"
# System language
lang en_US.UTF-8
# Firewall configuration
firewall --enabled --service=ssh
repo --name="koji-override-0" --baseurl=http://qa.centos.org/centos/8/BaseOS/x86_64/os/
repo --name="koji-override-1" --baseurl=http://qa.centos.org/centos/8/AppStream/x86_64/os/
# Shutdown after installation
shutdown
# Network information
network  --bootproto=dhcp --device=link --activate
network  --bootproto=dhcp --hostname=localhost.localdomain
# System timezone
timezone UTC --isUtc
# Use text mode install
text
# System authorization information
auth --enableshadow --passalgo=sha512
firstboot --disable
# SELinux configuration
selinux --enforcing

# System services
services --disabled="kdump" --enabled="NetworkManager,sshd,rsyslog,chronyd,cloud-init,cloud-init-local,cloud-config,cloud-final,rngd"
ignoredisk --only-use=vda
# System bootloader configuration
bootloader --append="console=ttyS0,115200n8 no_timer_check crashkernel=auto net.ifnames=0" --location=mbr --timeout=1 --boot-drive=vda
reqpart
# Clear the Master Boot Record
zerombr
# Partition clearing information
clearpart --all --initlabel
# Disk partitioning information
part / --fstype="xfs" --ondisk=vda --size=8000

%post --erroronfail
passwd -d root
passwd -l root

# pvgrub support
echo -n "Creating grub.conf for pvgrub"
rootuuid=$( awk '$2=="/" { print $1 };'  /etc/fstab )
mkdir /boot/grub
echo -e 'default=0\ntimeout=0\n\n' > /boot/grub/grub.conf
for kv in $( ls -1v /boot/vmlinuz* |grep -v rescue |sed s/.*vmlinuz-//  ); do
  echo "title CentOS Linux 8 ($kv)" >> /boot/grub/grub.conf
  echo -e "\troot (hd0)" >> /boot/grub/grub.conf
  echo -e "\tkernel /boot/vmlinuz-$kv ro root=$rootuuid console=hvc0 LANG=en_US.UTF-8" >> /boot/grub/grub.conf
  echo -e "\tinitrd /boot/initramfs-$kv.img" >> /boot/grub/grub.conf
  echo
done
ln -sf grub.conf /boot/grub/menu.lst
ln -sf /boot/grub/grub.conf /etc/grub.conf

# setup systemd to boot to the right runlevel
rm -f /etc/systemd/system/default.target
ln -s /lib/systemd/system/multi-user.target /etc/systemd/system/default.target
echo .

dnf -C -y remove linux-firmware

# Remove firewalld; it is required to be present for install/image building.
# but we dont ship it in cloud
dnf -C -y remove firewalld --setopt="clean_requirements_on_remove=1"
dnf -C -y remove avahi\* 
sed -i '/^#NAutoVTs=.*/ a\
NAutoVTs=0' /etc/systemd/logind.conf

cat > /etc/sysconfig/network << EOF
NETWORKING=yes
NOZEROCONF=yes
EOF

# For cloud images, 'eth0' _is_ the predictable device name, since
# we don't want to be tied to specific virtual (!) hardware
rm -f /etc/udev/rules.d/70*
ln -s /dev/null /etc/udev/rules.d/80-net-name-slot.rules

# simple eth0 config, again not hard-coded to the build hardware
cat > /etc/sysconfig/network-scripts/ifcfg-eth0 << EOF
DEVICE="eth0"
BOOTPROTO="dhcp"
ONBOOT="yes"
TYPE="Ethernet"
USERCTL="yes"
PEERDNS="yes"
IPV6INIT="no"
PERSISTENT_DHCLIENT="1"
EOF

echo "virtual-guest" > /etc/tuned/active_profile

# generic localhost names
cat > /etc/hosts << EOF
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

EOF
echo .

systemctl mask tmp.mount

cat <<EOL > /etc/sysconfig/kernel
# UPDATEDEFAULT specifies if new-kernel-pkg should make
# new kernels the default
UPDATEDEFAULT=yes

# DEFAULTKERNEL specifies the default kernel package type
DEFAULTKERNEL=kernel
EOL

# make sure firstboot doesn't start
echo "RUN_FIRSTBOOT=NO" > /etc/sysconfig/firstboot

# centos cloud user
echo -e 'centos\tALL=(ALL)\tNOPASSWD: ALL' >> /etc/sudoers
sed -i 's/name: cloud-user/name: centos/g' /etc/cloud/cloud.cfg

dnf clean all

# XXX instance type markers - MUST match CentOS Infra expectation
echo 'genclo' > /etc/yum/vars/infra

# change dhcp client retry/timeouts to resolve #6866
cat  >> /etc/dhcp/dhclient.conf << EOF

timeout 300;
retry 60;
EOF


rm -rf /var/log/yum.log
rm -rf /var/lib/yum/*
rm -rf /root/install.log
rm -rf /root/install.log.syslog
rm -rf /root/anaconda-ks.cfg
rm -rf /var/log/anaconda*

rm -f /var/lib/systemd/random-seed

cat /dev/null > /etc/machine-id

echo "Fixing SELinux contexts."
touch /var/log/cron
touch /var/log/boot.log
mkdir -p /var/cache/yum
/usr/sbin/fixfiles -R -a restore

# reorder console entries
sed -i 's/console=tty0/console=tty0 console=ttyS0,115200n8/' /boot/grub2/grub.cfg

true

%end

%packages
@core
NetworkManager
centos-release
chrony
cloud-init
cloud-utils-growpart
cockpit-system
cockpit-ws
dhcp-client
dnf
dnf-utils
dracut-config-generic
dracut-norescue
firewalld
grub2
kernel
nfs-utils
python3-jsonschema
qemu-guest-agent
rng-tools
rsync
tar
yum
yum-utils
-aic94xx-firmware
-alsa-firmware
-alsa-lib
-alsa-tools-firmware
-biosdevname
-iprutils
-ivtv-firmware
-iwl100-firmware
-iwl1000-firmware
-iwl105-firmware
-iwl135-firmware
-iwl2000-firmware
-iwl2030-firmware
-iwl3160-firmware
-iwl3945-firmware
-iwl4965-firmware
-iwl5000-firmware
-iwl5150-firmware
-iwl6000-firmware
-iwl6000g2a-firmware
-iwl6000g2b-firmware
-iwl6050-firmware
-iwl7260-firmware
-langpacks-*
-langpacks-en
-libertas-sd8686-firmware
-libertas-sd8787-firmware
-libertas-usb8388-firmware
-plymouth

%end

Setting up Salt-API for simple sandbox deployments

Due to errors in the ldap setup we are using pam as the auth method This means:

must manage external_auth.conf file in /etc/salt/master.d
must use setfacl -m u:slt800:r /etc/shadow for pam to work
a. /etc/shadow perms are reset anytime someone runs passwd
external_auth.conf must have restrictions on which modules can be run by api user.
since vmware cloning and deploying can take a while, use runner_async.
It will return jid that the caller can then reference for completion.
must create local user isv000 for all api calls from iServer.

external_auth.conf example:
external_auth:
  pam:
    isv000:
      - '*':
        - test.ping
      - '@runner':
        - 'manage.*'
        - 'cloud.profile'
        - 'jobs.print_job'
        - 'jobs.exit_success'

ops-master docker setup for testing runner modules

run kitchen converge to set up prod copy /tmp/kitchen/etc/salt/minion to /etc/salt/master.d/fileserver.conf remove reference to local remove root_dir reference line add auto_accept: True or open_mode: True echo “master: localhost” > /etc/salt/minion.d/saltconf.conf systemctl start salt-master salt-minion salt-api etcd

Runner call with asynchronous method

In the below, the “low” parameter is an empty dict. This works and provides a jid for retreval run_client.asynchronous('cloud.list_images', {})

This provides options to the called fun.

Was able to call all other images from other deployements, but not opsvcenter run_client.asynchronous('cloud.list_images', {'provider': 'opsvcenter'})