OST is failing - Last successful run was Dec-13-2019

Description

Seems we have NFS permissions issue for el8 vdsm in some of the runs.

Example from
https://jenkins.ovirt.org/view/Amit/job/ovirt-system-tests_manual/6302/artifact/exported-artifacts/test_logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-host-1/_var_log/vdsm/vdsm.log
:

2020-01-03 12:07:34,169-0500 INFO (MainThread) [vds] (PID: 1264) I am the
actual vdsm 4.40.0.1458.git1fca84350 lago-basic-suite-master-host-1
(4.18.0-80.11.2.el8_0.x86_64) (vdsmd:152)...
2020-01-03 12:50:29,662-0500 ERROR (check/loop) [storage.Monitor] Error
checking path /rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata
(monitor:501)
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line
499, in _pathChecked
delay = result.delay()
File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line 391,
in delay
raise exception.MiscFileReadException(self.path, self.rc, self.err)
vdsm.storage.exception.MiscFileReadException: Internal file read failure:
('/rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata',
1, bytearray(b"/usr/bin/dd: failed to open
\'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata\':
Operation not permitted\n"))
2020-01-03 12:50:30,112-0500 DEBUG (jsonrpc/7) [jsonrpc.JsonRpcServer]
Calling 'StoragePool.disconnect' in bridge with {'storagepoolID':
'c90b137f-6e1f-4b9a-9612-da58910a2439', 'hostID': 2, 'scsiKey':
'c90b137f-6e1f-4b9a-9612-da58910a2439'} (_init_:329)
2020-01-03 12:50:30,114-0500 INFO (jsonrpc/7) [vdsm.api] START
disconnectStoragePool(spUUID='c90b137f-6e1f-4b9a-9612-da58910a2439',
hostID=2, remove=False, options=None) from=::ffff:192.168.201.4,38786,
flow_id=8d05a1, task_id=95573498-d1c7-41ad-ad33-28f2192b2b60 (api:48)

Probably need to set NFS server export options as in
https://bugzilla.redhat.com/show_bug.cgi?id=1776843#c7

Activity

Show:
Martin Perina
January 6, 2020, 9:49 AM

On Sun, Jan 5, 2020 at 10:08 AM Amit Bawer <abawer@redhat.com> wrote:

> Seems we have NFS permissions issue for el8 vdsm in some of the runs.
>
> Example from
> https://jenkins.ovirt.org/view/Amit/job/ovirt-system-tests_manual/6302/artifact/exported-artifacts/test_logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-host-1/_var_log/vdsm/vdsm.log
> :
>
>
> 2020-01-03 12:07:34,169-0500 INFO (MainThread) [vds] (PID: 1264) I am the
> actual vdsm 4.40.0.1458.git1fca84350 lago-basic-suite-master-host-1
> (4.18.0-80.11.2.el8_0.x86_64) (vdsmd:152)...
> 2020-01-03 12:50:29,662-0500 ERROR (check/loop) [storage.Monitor] Error
> checking path /rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata
> (monitor:501)
> Traceback (most recent call last):
> File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line
> 499, in _pathChecked
> delay = result.delay()
> File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line 391,
> in delay
> raise exception.MiscFileReadException(self.path, self.rc, self.err)
> vdsm.storage.exception.MiscFileReadException: Internal file read failure:
> ('/rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata',
> 1, bytearray(b"/usr/bin/dd: failed to open
> \'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata\':
> Operation not permitted\n"))
> 2020-01-03 12:50:30,112-0500 DEBUG (jsonrpc/7) [jsonrpc.JsonRpcServer]
> Calling 'StoragePool.disconnect' in bridge with {'storagepoolID':
> 'c90b137f-6e1f-4b9a-9612-da58910a2439', 'hostID': 2, 'scsiKey':
> 'c90b137f-6e1f-4b9a-9612-da58910a2439'} (_init_:329)
> 2020-01-03 12:50:30,114-0500 INFO (jsonrpc/7) [vdsm.api] START
> disconnectStoragePool(spUUID='c90b137f-6e1f-4b9a-9612-da58910a2439',
> hostID=2, remove=False, options=None) from=::ffff:192.168.201.4,38786,
> flow_id=8d05a1, task_id=95573498-d1c7-41ad-ad33-28f2192b2b60 (api:48)
>
>
> Probably need to set NFS server export options as in
> https://bugzilla.redhat.com/show_bug.cgi?id=1776843#c7
>

Here is fix for NFS server on EL8: https://gerrit.ovirt.org/106120

Should this be changes also for NFS server on EL7?

_______________________________________________
> Devel mailing list – devel@ovirt.org
> To unsubscribe send an email to devel-leave@ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/WLLVWPZU6HFDWVWDIIJAS6OMBG4HRWF5/
>


Martin Perina
Manager, Software Engineering
Red Hat Czech s.r.o.

Amit Bawer
January 6, 2020, 10:15 AM

AF

On Mon, Jan 6, 2020 at 11:48 AM Martin Perina <mperina@redhat.com> wrote:

>
>
> On Sun, Jan 5, 2020 at 10:08 AM Amit Bawer <abawer@redhat.com> wrote:
>
>> Seems we have NFS permissions issue for el8 vdsm in some of the runs.
>>
>> Example from
>> https://jenkins.ovirt.org/view/Amit/job/ovirt-system-tests_manual/6302/artifact/exported-artifacts/test_logs/basic-suite-master/post-004_basic_sanity.py/lago-basic-suite-master-host-1/_var_log/vdsm/vdsm.log
>> :
>>
>>
>> 2020-01-03 12:07:34,169-0500 INFO (MainThread) [vds] (PID: 1264) I am
>> the actual vdsm 4.40.0.1458.git1fca84350 lago-basic-suite-master-host-1
>> (4.18.0-80.11.2.el8_0.x86_64) (vdsmd:152)...
>> 2020-01-03 12:50:29,662-0500 ERROR (check/loop) [storage.Monitor] Error
>> checking path /rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata
>> (monitor:501)
>> Traceback (most recent call last):
>> File "/usr/lib/python3.6/site-packages/vdsm/storage/monitor.py", line
>> 499, in _pathChecked
>> delay = result.delay()
>> File "/usr/lib/python3.6/site-packages/vdsm/storage/check.py", line
>> 391, in delay
>> raise exception.MiscFileReadException(self.path, self.rc, self.err)
>> vdsm.storage.exception.MiscFileReadException: Internal file read failure:
>> ('/rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata',
>> 1, bytearray(b"/usr/bin/dd: failed to open
>> \'/rhev/data-center/mnt/192.168.200.4:_exports_nfs_exported/b92b26cf-fac4-4ccf-ba31-f6fb4184e302/dom_md/metadata\':
>> Operation not permitted\n"))
>> 2020-01-03 12:50:30,112-0500 DEBUG (jsonrpc/7) [jsonrpc.JsonRpcServer]
>> Calling 'StoragePool.disconnect' in bridge with {'storagepoolID':
>> 'c90b137f-6e1f-4b9a-9612-da58910a2439', 'hostID': 2, 'scsiKey':
>> 'c90b137f-6e1f-4b9a-9612-da58910a2439'} (_init_:329)
>> 2020-01-03 12:50:30,114-0500 INFO (jsonrpc/7) [vdsm.api] START
>> disconnectStoragePool(spUUID='c90b137f-6e1f-4b9a-9612-da58910a2439',
>> hostID=2, remove=False, options=None) from=::ffff:192.168.201.4,38786,
>> flow_id=8d05a1, task_id=95573498-d1c7-41ad-ad33-28f2192b2b60 (api:48)
>>
>>
>> Probably need to set NFS server export options as in
>> https://bugzilla.redhat.com/show_bug.cgi?id=1776843#c7
>>
>
> Here is fix for NFS server on EL8: https://gerrit.ovirt.org/106120
>
> Should this be changes also for NFS server on EL7?
>

AFAICT the need to change the NFS options mostly arises from changes in
vdsm dependencies for el8, such as libvirt, requiring a different access
for NFS shares than before.
So if we are testing el8 hosts with el7 NFS server that might be relvent
for el7 NFS server as well.

> _______________________________________________
>> Devel mailing list – devel@ovirt.org
>> To unsubscribe send an email to devel-leave@ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>> https://lists.ovirt.org/archives/list/devel@ovirt.org/message/WLLVWPZU6HFDWVWDIIJAS6OMBG4HRWF5/
>>
>
>
> –
> Martin Perina
> Manager, Software Engineering
> Red Hat Czech s.r.o.
>

Evgheni Dereveanchin
March 9, 2020, 10:07 AM

This has been fixed for quite some time now and OST manual is working: https://jenkins.ovirt.org/job/ovirt-system-tests_manual/

Closing the ticket.

Fixed

Assignee

Galit Rosenthal

Reporter

Amit Bawer

Blocked By

None

Priority

Medium
Configure