Re: [ovirt-users] Public Glance repository for oVirt

Description

On Tue, Oct 12, 2021 at 6:45 PM <p.staniforth@leedsbeckett.ac.uk> wrote:
>
> Hello,
> can anyone get to the public openstack image repository ovirt-image-repository?
>
> Also when I try browse the site I get
>
> This site can’t be reached
> https://glance.ovirt.org/ is unreachable.

Thanks for the report.

Adding infra/infra-support. Any clue? This is affecting also CI,
breaking at least
test_undo_preview_snapshot_when_mac_used_reassigns_a_new_mac in
ovirt-system-tests.

Thanks and best regards,

Didi

Activity

Show:

Former user October 13, 2021 at 12:15 PM

Pre-reboot mem usage and processes according to top:

 

top - 05:49:55 up 63 days,  7:16,  1 user,  load average: 0.02, 0.04, 0.09
Tasks: 123 total,   2 running, 121 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.5 us,  0.0 sy,  0.0 ni, 99.5 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  1782800 total,   165440 free,  1306576 used,   310784 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   101944 avail Mem  

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
7860 keystone  20   0  611020  81440      0 S   0.0  4.6   0:10.75 httpd
7861 keystone  20   0  328900   4416      0 S   0.0  0.2   0:07.00 httpd
7862 keystone  20   0  328900   4416      0 S   0.0  0.2   0:06.95 httpd
7863 keystone  20   0  394436   4416      0 S   0.0  0.2   0:06.98 httpd
7864 keystone  20   0  328900   4416      0 S   0.0  0.2   0:06.82 httpd
7865 keystone  20   0  328900   4416      0 S   0.0  0.2   0:06.76 httpd
7866 apache    20   0  238860   4480     12 S   0.0  0.3   0:00.07 httpd
7867 apache    20   0  238860   4480     12 S   0.0  0.3   0:00.07 httpd
7868 apache    20   0  238860   4480     12 S   0.0  0.3   0:00.06 httpd
7869 apache    20   0  238860   4480     12 S   0.0  0.3   0:00.04 httpd
7870 apache    20   0  238860   4480     12 S   0.0  0.3   0:00.05 httpd
7960 apache    20   0  238860   4480     12 S   0.0  0.3   0:00.59 httpd
10361 root      20   0       0      0      0 S   0.0  0.0   0:00.11 kworker/u8:2
10845 root      20   0       0      0      0 S   0.0  0.0   0:01.02 kworker/u8:1
11508 apache    20   0  238860   4480     12 S   0.0  0.3   0:00.04 httpd
11593 apache    20   0  238860   4480     12 S   0.0  0.3   0:00.04 httpd
11795 glance    20   0  431172 107928   1552 S   0.0  6.1   0:24.25 glance-api
14914 root      20   0       0      0      0 R   0.0  0.0   0:01.96 kworker/0:0
18651 root      20   0       0      0      0 S   0.0  0.0   0:00.19 kworker/1:0
20789 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/0:2
20962 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/1:2
21174 postfix   20   0   90508   4192   3100 S   0.0  0.2   0:00.01 pickup
21202 root      20   0  156936   5840   4408 S   0.0  0.3   0:00.40 sshd
21205 root      20   0  115644   2196   1668 S   0.0  0.1   0:00.04 bash
21305 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/1:1
21364 root      20   0  112940   4284   3260 S   0.0  0.2   0:00.00 sshd
21366 root      20   0  162104   2324   1600 R   0.0  0.1   0:00.02 top

 

Post-reboot equivalents:

top - 08:14:15 up  2:23,  1 user,  load average: 0.01, 0.03, 0.05
Tasks: 119 total,   2 running, 117 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.7 us,  0.3 sy,  0.0 ni, 99.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  1782800 total,   524928 free,   962676 used,   295196 buff/cache
KiB Swap:        0 total,        0 free,        0 used.   592456 avail Mem  

 PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 999 glance    20   0  421700 104872   7872 R   1.7  5.9   1:45.13 glance-api
 682 root      20   0  474276  12732   6644 S   1.0  0.7   0:56.94 NetworkManager
   9 root      20   0       0      0      0 S   0.3  0.0   0:11.49 rcu_sched
   1 root      20   0  128088   6748   4216 S   0.0  0.4   0:02.23 systemd
   2 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kthreadd
   3 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0
   4 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/0:0H
   5 root      20   0       0      0      0 S   0.0  0.0   0:00.10 kworker/u8:0
   6 root      20   0       0      0      0 S   0.0  0.0   0:00.49 ksoftirqd/0
   7 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 migration/0
   8 root      20   0       0      0      0 S   0.0  0.0   0:00.00 rcu_bh
  10 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 lru-add-drain
  11 root      rt   0       0      0      0 S   0.0  0.0   0:00.03 watchdog/0
  12 root      rt   0       0      0      0 S   0.0  0.0   0:00.03 watchdog/1
  13 root      rt   0       0      0      0 S   0.0  0.0   0:00.00 migration/1
  14 root      20   0       0      0      0 S   0.0  0.0   0:00.01 ksoftirqd/1
  16 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kworker/1:0H
  18 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kdevtmpfs
  19 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 netns
  20 root      20   0       0      0      0 S   0.0  0.0   0:00.00 khungtaskd
  21 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 writeback
  22 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kintegrityd
  23 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset
  24 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset
  25 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 bioset
  26 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kblockd
  27 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 md
  28 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 edac-poller
  29 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 watchdogd
  36 root      20   0       0      0      0 S   0.0  0.0   0:00.00 kswapd0
  37 root      25   5       0      0      0 S   0.0  0.0   0:00.00 ksmd
  38 root      39  19       0      0      0 S   0.0  0.0   0:00.07 khugepaged
  39 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 crypto
  47 root       0 -20       0      0      0 S   0.0  0.0   0:00.00 kthrotld

Former user October 13, 2021 at 9:52 AM

The service is back up for now, will review if we need to grow RAM on this host or solve this in some other way as it’s worked for several years up to this point with no issues.

Former user October 13, 2021 at 9:50 AM
Edited

Looks like the box is running out of memory:

 

[5380587.626351] Out of memory: Kill process 11388 (mysqld) score 62 or sacrifice child
[5380587.627302] Killed process 11388 (mysqld), UID 27, total-vm:1725896kB, anon-rss:110984kB, file-rss:0kB, shmem-rss:0kB

I’ve rebooted the system to ensure it is back to clean state.

Former user October 13, 2021 at 9:47 AM

Looks like mysql is down on the glance server.

Checking why this is happening to restore the service.

Details

Assignee

Reporter

Priority

Created October 13, 2021 at 6:42 AM
Updated October 13, 2021 at 12:15 PM