as migrating lago* jobs seems to be successful, I propose to do the full migration on Sunday(10-04-2016) evening, TLV time.

Few notes:

• users - will have to enrol again. I didn't find any solution other than setting up a proper ldap/sso solution without introducing all sort of 'hacks' to the environment. the reason is that Jenkins encrypts passwords using a private key that is found on the master. this implies that we have no way of automating users creation without storing plain-text passwords somewhere(hiera, foreman or locally), also replicating the encryption process is not trivial. a more in-depth review of Jenkins password storage mechanism(somewhat outdated, now jbcrypt is also used by default)

• users personal views - if someone ask I can just migrate them manually.

• none-yamlized jobs - I think we should not migrate them. they will be available in jenkins-old if someone requires them. unless there is something critical I am not aware of(moving them technically is not a problem, just copying the jobs XMLs).

• I propose to bundle the migration with trying to improve gerrit performances(this and this), I think its a good time to do it while Jenkins is down. if we could get the memory increased by then would be even better.

Steps left before migration:

• merge puppet code, yaml code(this, this)

• change VM environment to production.

• test live storage increasing.

• increase VM memory size to 32gb.

• increase storage to 1TB(we have enough available last I checked)

• ensure icinga alerts are working.

Migration process:

• send alert email.

• block both Jenkins's for out-side access.

• change DNS: jenkins current ip -> jenkins-old.ovirt.org, jenkins.ovirt.org -> jenkins new ip, iirc @Anton Marchukov had access to it(to *.ovirt.org)

• ensure there are no jenkins.phx.ovirt.org leftovers in the configuration(including lago @ github)

• run script to move all slaves.

• trigger JJB.

• smoke-test publishers/build_artifacts/check_merged(this would cover most things).

Enhancements left:

• add puppet jenkins lvm module.

• configure backups using backups.phx.ovirt.org

• change JVM memory parameter to be relative to the VM memory size(hard-coded in the manifest right now).

• version the config.xml file.

your thoughts? also you think any of the enhancements must be done before the migration?

update:
1. gerrit triggers are working.
2. infra-puppet jobs migrated and working.
http://jenkins.phx.ovirt.org/job/system_update_puppet/
http://jenkins.phx.ovirt.org/job/infra-puppet_production_check-puppet-manifests_gerrit/
though they are not yamlized yet, should open another ticket on that.
3. merged the main puppet jenkins manifest https://gerrit.ovirt.org/53309, and hiera patch https://gerrit.ovirt.org/#/c/54718/
4. https://gerrit.ovirt.org/#/c/55363/ - another puppet patch under work(packages, site logo, etc)
5. jjb is working, theoretically we could load all yaml jobs with http://jenkins.phx.ovirt.org/job/jenkins_master_deploy-configs_merged/, but better not for now(would duplicate everything).

had some progress on slaves authentication via puppet with exported resources, still working on it.
took fc21-vm07.phx.ovirt.org offline for the migration.
plan is to first attempt running http://jenkins.ovirt.org/job/ovirt-node-ng_master_build-artifacts-fc22-x86_64/ and later migrate @Former user's jobs.

I think there are things we can do, such as add a few slaves to the new jenkins and for e.g, applying the "oVirt" UI and logo to the new server..

I cannot move forward with this without sitting with him at least once

On 16 March 2016 at 10:46, eyal edri [Administrator] (oVirt JIRA)

–
Barak Korren
bkorren@redhat.com
RHEV-CI Team

Barak, see if you can move this forward while Nadav isn't here, so we can hasten the migration.

Actions needed for migration:

• Copy master configuration to new server

• Enable new slaves on the new server

• Deploy 'jenkins job builder' deploy job and create all currently yamlized jobs on new master

• Move backups to new backup.phx.ovirt.org server.

• Change DNS entry for jenkins new master to jenkins.ovirt.org and old once to jenkins-old.ovirt.org.

Verification needed:

• Verify gerrit integration works

• Verify user logic

• Verify JJB deploy/disable

• Verify publishers works

• Verify backup works

• Verify puppet classes are working

@Anton Marchukov @Barak Korren @Former user please add any AI / Verification you think is missing.