Change-Id: Iebcf2a80949d8d5d14a2e6aeb6582884b11d343f
Signed-off-by: Oleg Drokin <green@whamcloud.com>

tfile seems to be a typo

Fixes: 43c3a804 ("LU-13805 tests: Add racing tests of BIO, DIO")

Test-Parameters: trivial
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I85b0afce577b708ef9e69747774bd248484bd9dd
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52772


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mikhail Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

RPCSEC init requests are kept in the rsi cache. While this is useful
during request processing involving upcall/downcall with userspace,
rsi entries are never used again once RPCSEC init requests have been
handled completely.
And keeping entries in the rsi cache has some impact on authentication
speed. When a new RPCSEC init request is received, the first step is
to check if there is a valid matching entry in the cache. It is never
the case, except if an authentication request is replayed, but GSS
rejects that anyway.
So we spend time browsing a cache from which we expect no match. Even
if the upcall cache mechanism takes this lookup opportunity to remove
invalid or expired entries, it is even better to remove cache entries
as soon as we know they are done.

Test-Parameters: kerberos=true testlist=sanity-krb5
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Ia9946578c3d3149e6235d832df28214ae8984f1e
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53488


Reviewed-by: Aurelien Degremont <adegremont@nvidia.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

Update SLES15 SP5 kernel to 5.14.21-150500.55.39.1 for Lustre client.

Test-Parameters: trivial mdtcount=4 mdscount=2 \
clientdistro=sles15sp5 testlist=sanity

Change-Id: Id9476e8726728b00d4079cdaf31b081f89190eb1
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53467


Reviewed-by: Colin Faber <cfaber@ddn.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

class_str2dev() was missing an obd_device_unlock(). I haven't
seen any bugs related to this missing unlock. I suspect
the mount state machine avoids this. Add the unlock just
to be safe.

Fixes: c5e5060d ("LU-8802 obd: remove MAX_OBD_DEVICES")
Test-Parameters: trivial
Signed-off-by: Timothy Day <timday@amazon.com>
Change-Id: I7a813f9d4931a7a9979223bfde5efea07f1e5228
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53466


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Update all LNetGetId() calls to use large NIDs if the connect
flags report large NID support. For the case of lmv_setup()
we update setting qos_rr_index, to avoid the thundering herd,
using nidhash().

Change-Id: I80fda9454f154e27fbc75abb1899c0ccca03097b
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53398


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Improve lljobstat command to read dumped stats file.
Usually the file is generated by command:
  lctl get_param *.*.job_stats > all_job_stats.txt

Multiple files can be specified with multiple --statsfile
options. For example:
  lljobstat --statsfile=1.txt --statsfile=2.txt

Stats data from multiple files will be added up and
sorted. Then the top jobs will be listed.

Try to use CLoader to accelerate the YAML parsing.

Handle SIGINT and exit silently if lljobstat is in the loop
of reading system job_stats files periodically.

Fix a bug when the job_id is a pure number.

Signed-off-by: Lei Feng <flei@whamcloud.com>
Test-Parameters: trivial
Change-Id: Iee1ce69d2befb9d021e34effd4fc65a47297c1fb
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53397


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Remove commit-msg and prepare-commit-msg links
in the build/ directory. Add proper headers and
SPDX to the remaining files in build/.

Test-Parameters: trivial
Fixes: 25c93758 ("LU-1199 build: Clean out the build directory")
Signed-off-by: Timothy Day <timday@amazon.com>
Change-Id: I649ed60f5541be2832555efa2e0cf64cd1a5c67c
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53396


Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

Add mdt.*.enable_cap_mask to allow specific capabilities to
be enabled and disabled individually.

Fixes: f05edf8e ("LU-13791 sec: enable FS capabilities")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I6fc0130a90693d673d8c2158e7e31c2de951553d
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53538


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Reviewed-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

In case a server uses an expired gss context to send a callback
request to a client, it might be that the associated context on
the client has already expired, and been purged from the cache.
This results in a GSS_S_NO_CONTEXT reply.
In this specific scenario, the server must mark its reverse context
as dead. This will lead to destruction of the expired context, and
creation of a new context suitable for further callback requests.

Test-Parameters: kerberos=true testlist=sanity-krb5
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I4af90cd70a3815851ec555ea85b49714c8da4202
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53375


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Aurelien Degremont <adegremont@nvidia.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

When using some lfs utilities (find, getstripe) with multiple paths,
if a subset of the paths has an error, for example due to a typo, the
error message produced can be misleading. For "getstripe" it refers to
the last file on the command line regardless of which file had the
error, and for "find" it prints out the right filename but uses
the error code from the last file on the command line.

This cleans up these error messages for "lfs find" and
"lfs getstripe".

This also adjusts "lfs setdirstripe" to continue for subsequent files
if it encounters an error for earlier files on the command line.

Signed-off-by: Thomas Bertschinger <bertschinger@lanl.gov>
Fixes: bc500536 ("LU-930 utils: fix 'lfs find' error message")
Fixes: 4affa48f ("LU-5170 utils: Continue on error when multiple files requested")
Fixes: a24f6153 ("LU-11213 dne: add new dir hash type "space"")
Change-Id: I9cdd007912ffb4f6ebc31e422851977e49186ae7
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52942


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Timothy Day <timday@amazon.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Add a client tunable and mount option to turn off project-enabled
statfs() if needed, for example to speed up statfs() execution by
avoiding project quota check.

This new llite tunable statfs_project is set to 1 by default (feature
enabled). To turn statfs_project off:

    lctl set_param llite.*.statfs_project=0

Additionally, statfs_project can be disabled at mount time with:

   mount -t lustre -o nostatfs_project ...

Signed-off-by: Stephane Thiell <sthiell@stanford.edu>
Change-Id: I1c3eb27e66b1d05a1c713732dfe0a4d8f7af769f
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52872


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alexander Boyko <alexander.boyko@hpe.com>
Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

This patch replaces the deprecated gethostbyname() and
gethostbyaddr() functions with getaddrinfo() and getnameinfo()
functions respectively.

The getaddrinfo() function combines the functionality provided by the
gethostbyname() and getservbyname() functions into a single interface,
but unlike the latter functions, getaddrinfo() is reentrant and allows
programs to eliminate IPv4-versus-IPv6 dependencies.

The getnameinfo() function is the inverse of getaddrinfo(): it
converts a socket address to a corresponding host and service, in a
protocol-independent manner. It combines the functionality of
gethostbyaddr() and getservbyport(), but unlike those functions,
getnameinfo() is reentrant and allows programs to eliminate
IPv4-versus-IPv6 dependencies.

Test-Parameters: kerberos=true testlist=sanity-krb5
Test-Parameters: testgroup=review-dne-selinux-ssk-part-2
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: Iacb5583826cd2f7329455bc6cbb4477f9087f15a
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52632


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Some applications do stat() calls under a directory within which
all children files have regularized file name:
- mdtest benchmark tool: mdtest.$rank.$i
- ML/AI with ingested data that have typically a format rule of
  the filename in the directory.

The most common format for regularized file name is that the
suffix part of the file name is number-indexing.
However, in the current statahead mechanism, the statahead is
populated by the order of the hash of the file name via readdir()
calls, not a kind of sorting order.

In this patch, we improve the statahead to prefetch attributes for
the files with regularized indexing file name via asynchronous
batching RPC.

This patch adds the support to do statahead for these kinds of
applications, which can be optimized, but without opendir()/
close() to start/stop statahead thread explicitly.

Instead, the statahead thread will stop and quit when found
that there was no acitivy for more than a certain time period
(i.e. 30 seconds).

Test-Parameters: mdtcount=4 mdscount=2 testlist=sanity env=ONLY=27p,ONLY_REPEAT=5
Test-Parameters: mdtcount=4 mdscount=2 testlist=sanity env=ONLY=27p,ONLY_REPEAT=5
Test-Parameters: mdtcount=4 mdscount=2 testlist=sanity env=ONLY=123f,ONLY_REPEAT=10
Test-Parameters: mdtcount=4 mdscount=2 testlist=sanity env=ONLY=123f,ONLY_REPEAT=10
Signed-off-by: Qian Yingjin <qian@ddn.com>
Change-Id: Ide11ec5a651ae74884ddbe1cecede4f5c961e38d
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/41308


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

lnetctl peer/route show command, without any other arguments,  should
not return an error if the peer/route tables are empty. If the user
specifies a particular peer/route to show, and that peer/route does
not exist then this is an error.

Modify the dumpit routines to check the netlink message length to
determine whether the user supplied any arguments to the show
commands, and use this information to return the proper status. Some
dead code was also removed from lnet_route_show_dump().

We also fix an issue with older kernels where non-zero return status
from old dumpit commands was not being returned correctly.

Test-Parameters: trivial
Signed-off-by: Chris Horn <chris.horn@hpe.com>
Change-Id: I9a188c573b0f373052208dbea52ea56181719769
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53366


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>

The original lnet ping Netlink code was setup to handle the pre-MR
API. Update this code to handle the newer MR version of ping. This
unifies all ping handling under one system and it supports larger
NID handling for IPv6. The big change is now that we support
updating the key table we can report failed pings in a different
format than the successful pings.

Instead of using the Netlink API version flag test if the passed
in LNet processid is LNET_PID_LUSTRE to display a NID or PID for
the ping results. Also cleanup the memory for failed ping NID
list.

Test-Parameters: trivial testlist=sanity-lnet
Change-Id: I77a0e313bf2b7035e501726068fd45bb3a118d06
Signed-off-by: James Simmons <jsimmons@infradead.org>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/49361


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Trying to start a transaction after journal destroy
during umount will lead to a crash.

This patch is adding the same checks from
04134040 LU-16982 ldiskfs: Fix crash after "umount -d -f /mnt/..."
for el9 series.

Change-Id: Ibb89e9f5104b0980a8d9543561ac643322e3724d
Fixes: e27a7b33 ("LU-16298 ldiskfs: Periodically write ldiskfs superblock")
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53499


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Vitaliy Kuznetsov <vkuznetsov@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Sanity test_24/26a started failing recently due to the landing of
new test_23d.  Disable test_23d for now to avoid tests failing, but
do not remove it so that it is possible to continue debugging it.
Add extra debugging to see why this is happening.

Test-Parameters: trivial testlist=sanity-lfsck mdscount=2 mdtcount=4
Test-Parameters: testlist=sanity-lfsck mdscount=2 mdtcount=4
Test-Parameters: testlist=sanity-lfsck mdscount=2 mdtcount=4
Test-Parameters: testlist=sanity-lfsck mdscount=2 mdtcount=4
Test-Parameters: testlist=sanity-lfsck mdscount=2 mdtcount=4
Test-Parameters: testlist=sanity-lfsck mdscount=2 mdtcount=4
Test-Parameters: testlist=sanity-lfsck mdscount=2 mdtcount=4
Test-Parameters: testlist=sanity-lfsck mdscount=2 mdtcount=4
Test-Parameters: testlist=sanity-lfsck mdscount=2 mdtcount=4
Test-Parameters: testlist=sanity-lfsck mdscount=2 mdtcount=4
Fixes: 07e02a60 ("LU-16826 tests: lfsck to repair a dangling remote entry")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Ib6edf1d014ceb6b5d965aadc11272a88e8c001d5
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53544


Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

Fix quoting job_id to account for leading '@' being reserved.

Test-Parameters: trivial
Signed-off-by: Nathaniel Clark <nclark@whamcloud.com>
Change-Id: Ifce3edc9b636db2f059ab9960488972a152d2e7a
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53424


Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Feng Lei <flei@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

Decrease cfs fail timeout in sanity-quota_81 from 30
to 10 seconds to avoid soft lockup.

Fixes: 862f0baa ("LU-15097 quota: stop pool_recalc before killing pool")
Test-Parameters: trivial testlist=sanity-quota
Signed-off-by: Sergey Cheremencev <scherementsev@ddn.com>
Change-Id: I8630db7b3948b335fef5d5349f960f79cb877fc3
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53384


Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

When building debian packages, the resulting packages are
moved into a 'debs/' subdir.

Don't miss the debug symbol packages 'dbgsym', which are
suffixed .ddeb.

Also add .buildinfo file.

Test-Parameters: trivial
Change-Id: I52d0bddfaafc67c4a2a2dbc786d7f320c0b979f8
Signed-off-by: Aurelien Degremont <adegremont@nvidia.com>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53378


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

RPCSEC init and RPCSEC context caches hold gss-related information
of security contexts established between network peers. These cache
entries are tightly coupled with contexts handled in the sptlrpc layer
so they must not be purged directly. They are inserted into the cache
when sptlrpc security contexts are established, and removed when the
corresponding security contexts are destroyed.

Test-Parameters: trivial
Test-Parameters: kerberos=true testlist=sanity-krb5
Test-Parameters: testgroup=review-dne-selinux-ssk-part-2
Fixes: 8d828762 ("LU-17015 gss: support large kerberos token for rpc sec init")
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I903f75a4b5229286fcaed3e9d96b5eee7f653f15
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53377


Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Aurelien Degremont <adegremont@nvidia.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

Update RHEL 8.9 kernel to 4.18.0-513.9.1.el8_9.

Test-Parameters: trivial fstype=ldiskfs mdtcount=4 mdscount=2 \
clientdistro=el8.9 serverdistro=el8.8 testlist=sanity

Test-Parameters: trivial fstype=zfs mdtcount=4 mdscount=2 \
clientdistro=el8.9 serverdistro=el8.8 testlist=sanity

Test-Parameters: trivial fstype=ldiskfs mdtcount=4 mdscount=2 \
clientdistro=el8.8 serverdistro=el8.9 testlist=sanity

Test-Parameters: trivial fstype=zfs mdtcount=4 mdscount=2 \
clientdistro=el8.8 serverdistro=el8.9 testlist=sanity

Test-Parameters: optional clientdistro=el8.9 serverdistro=el8.9 \
testgroup=full-part-1

Test-Parameters: optional clientdistro=el8.9 serverdistro=el8.9 \
testgroup=full-part-2

Test-Parameters: optional clientdistro=el8.9 serverdistro=el8.9 \
testgroup=full-part-3

Change-Id: Ied0d2873974a3c8cc6e346373457c8ebc09740d6
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53357


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Yang Sheng <ys@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

rsi_upcall_seq_write() uses sscanf to get the string passed from
userspace, but this needs to be copied to a kernel buffer first.

Test-Parameters: trivial
Test-Parameters: kerberos=true testlist=sanity-krb5
Test-Parameters: testgroup=review-dne-selinux-ssk-part-2
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I2ec875b7c6c158695857fe912ec1dd9f41ddc25d
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53342


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Aurelien Degremont <adegremont@nvidia.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

This patch adds return value checks to function
yaml_parser_initialize() and fopen() under lustre_cfg.c
And funciton cYAML_build_tree() under cyaml.c

Test-Parameters: trivial
CoverityID: 410239 ("Unchecked return value")
CoverityID: 410238 ("Unchecked return value")
Fixes: 65062463 (LU-14359 hsm: support a flatter HSM archive format)
Fixes: 8961f2d8 (LU-4939 utils: allow configuration through yaml files)
Change-Id: I67a34adee3e4d25f97244487684a613426637a70
Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53331


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

The "qbody" should be checked whether it is NULL or not.

CoverityID: 410242 ("Dereference after null check")

Fixes: 57ac32a2 ("LU-16097 quota: release preacquired quota when over limits")
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Change-Id: Idab61f3ebac24307c6d5db0d42429914858d21cb
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53330


Reviewed-by: Sergey Cheremencev <scherementsev@ddn.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

This patch fixes 'conn->ksnc_proto' which was
dereferenced under function ksocknal_recv_hello()
even though it could be NULL.

This patch also removes 'returns' in between
the function and replaces it with 'goto'.
Allowing exit from a single place.

CoverityID: 410244 ("Dereference after null check")
Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Fixes: cb5f92c0 (LU-10391 ksocklnd: use ksocknal_protocol v4 for IPv6)
Change-Id: I95196d481b537281ab8643f1ee6162db450bef20
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53305


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Some commit messages have control characters, or fancy quotation
marks, or mdash hyphens or similar, and this messes up the display
of "git log" and other tools depending on the current locale and
character set used in the terminal.

Add a check into commit-msg to reject commit messages that have
non-ASCII characters. This does not apply to characters used in
the Signed-off-by: or similar fields that list people's names.

Test-Parameters: trivial
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I99d0954a68f8a5391195553ebf4b69181b6991f2
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53304


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Timothy Day <timday@amazon.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

When a new OST is added to a filesystem without no_create,
then a new object created on the OST relatively quickly
after it is added to the filesystem, in particular because
the new OST would be preferred by QOS space balancing
due to lots of free space. However, it might take a few
seconds for the addition of the new OST to be propagated
across all of the clients, so there is a risk that the MDS
creates file object on OSTs that a client is not yet aware of,
which returns an error to the application immediately.

This patch fixes the issue by adding a loop in lsme_unpack()
that is waiting and retrying for some number of seconds for
the filesystem layout to be updated if either the
"loi->loi_ost_idx >= lov->desc.ld_tgt_count" or "!ltd"
condition is hit.

Change-Id: Idc29b8c66079afaea25428577daf51370fa2b084
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53335


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Handle extra setattr_prepare() argument added in Linux 5.12 kernels
when building on older kernels.

HPE-bug-id: LUS-12059
Signed-off-by: Andrew Perepechko <andrew.perepechko@hpe.com>
Change-Id: Ie7fd1c4d51b7a9b086cfca0db941321cbcce7057
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53503


Reviewed-by: Alexander Zarochentsev <alexander.zarochentsev@hpe.com>
Reviewed-by: Sebastien Buisson <sbuisson@ddn.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Tested-by: Sebastien Buisson <sbuisson@ddn.com>
Tested-by: James Simmons <jsimmons@infradead.org>

There were examples in the field with RoCE setups which demonstrate
that CM_EVENT_UNREACHABLE may be received when connection is already
in ESTABLISHED state. This causes an assert in kiblnd_cm_callback to
fail.

Handle this in a more gracious manner: report the event as unexpected
and allow the flow to continue. If there are indeed issues on
the connection, it is expected to report transaction errors later
and get cleaned up without crashing the whole system.

Test-Parameters: trivial testlist=sanity-lnet
Signed-off-by: Serguei Smirnov <ssmirnov@whamcloud.com>
Change-Id: If32166fe9fc59e025609c2035cb1c03d3bed22f2
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53298


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Chris Horn <chris.horn@hpe.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

If the OST object precreate count gets large (usually due to high
MDT file create workload, but sometimes also forced during testing)
then send an OST_CREATE RPC sooner when the number of precreated
objects gets low.

Currently the MDS will wait until 1/2 of the precreated OST objects
are consumed, but if create_count = 10000, then this can put bursty
create workloads on the OST.  Instead, send an OST_CREATE RPC when
the precreate pool is at most 1024 objects below target, so that the
MDS keeps its precreated pool more full and the OST doesn't have to
create so many objects at once (which also locks object directories
for a longer time).

Don't set opd_force_creation=true when osp.*.create_count is set
larger, and instead rely on the improved precreate check to force
OST object creation to start sooner, as opd_force_creation=true
can cause the OSP precreation to stop completely in some cases.

Test-Parameters: testlist=sanity env=ONLY=1-130,HONOR_EXCEPT=y
Test-Parameters: testlist=sanity env=ONLY=1-130,HONOR_EXCEPT=y
Test-Parameters: testlist=sanity env=ONLY=1-130,HONOR_EXCEPT=y
Test-Parameters: testlist=sanity env=ONLY=1-130,HONOR_EXCEPT=y
Test-Parameters: testlist=parallel-scale env=ONLY=rr_alloc,ONLY_REPEAT=10
Test-Parameters: testlist=parallel-scale env=ONLY=rr_alloc,ONLY_REPEAT=10
Test-Parameters: testlist=parallel-scale env=ONLY=rr_alloc,ONLY_REPEAT=10
Test-Parameters: testlist=parallel-scale env=ONLY=rr_alloc,ONLY_REPEAT=10
Fixes: df5b4c0a ("LU-17251 osp: force precreate if create_count grows")
Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: Id2d12636d535485919ca5eec3adb18b1e6ce7057
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53245


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Alex Deiter <alex.deiter@gmail.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Remove bash wrapper script lutf_start.sh. Set the environment
natively in Python. LUTF currently involves a number of nested
wrapper scripts. Hence, this patch aims to simplify LUTF.

It also makes it simplier to import this script into another
Python script, by providing a reusable function to set the
environment natively.

Test-Parameters: @lnet
Signed-off-by: Timothy Day <timday@amazon.com>
Change-Id: I56d80c12f9e50f3f8de1668ffa04c855a9829601
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53237


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Cyril Bordage <cbordage@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Python virutal environments make it easy to install the
correct Python packages isolated from the rest of the
system.

 https://docs.python.org/3/library/venv.html



.venv is added to .gitignore and a simple virtual environment
example has been added to the README.

This patch collects all of the requirements for various
scripts in the Lustre tree and consolidates them in a
top level requirements.txt. lu_object.py spacing was
fixed due to parsing errors.

Test-Parameters: trivial
Signed-off-by: Timothy Day <timday@amazon.com>
Change-Id: I69d074e9ba50022817bd243fb82d004366ab6adf
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53209


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LUTF did not have a proper configuration option. Since
no message was printed at configure time, this made it
hard to debug why LUTF was not being built.

Fix a few minor bugs in headers that prevented shared
libraries from being `import`ed by python.

Fix a small Clang error in liblutf_agent.c.

Test-Parameters: @lnet
Signed-off-by: Timothy Day <timday@amazon.com>
Change-Id: I6680b203bef08b7afa326a1cbe30c96b5c29e95c
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53200


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Reviewed-by: Frank Sehr <fsehr@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Fix memory leak reported by Coverity in
setup_indexes() in case of errors during
OST UUID initialization.

CoverityID: 397693 ("Resource leak")

Test-Parameters: trivial testlist=sanity,conf-sanity
Signed-off-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Fixes: 05334b90 ("LU-16331 utils: fix 'lfs find -O <uuid>' with gaps")
Change-Id: Ibfd10cebaf3198ae2e9bb35686be420e4cd0050b
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53182


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Timothy Day <timday@amazon.com>
Reviewed-by: Anjus George <georgea@ornl.gov>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Rick Mohr <mohrrf@ornl.gov>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Add aheadmany to .gitignore.

Fixes: 5317f8a5 ("LU-14361 statahead: Add test for statahead advise")
Test-Parameters: trivial
Signed-off-by: Timothy Day <timday@amazon.com>
Change-Id: I1003200b7ed34e90d2aa0f75cb4c4f071eaeea04
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53173


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: James Simmons <jsimmons@infradead.org>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

if some component of a mirrored file is broken, it makes sense
to try another (possible valid) replica rather than give up
immediately.

Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Change-Id: I32ea0efa90109f5159bf8b6c4e0efe1d543580c3
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52996


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Zhenyu Xu <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Change the 'lfs find -size' argument to 512-byte blocks by default if
no unit is given.  This better matches find(1) and avoids confusion
when converting "find" arguments to "lfs find".  Accept the 'c' suffix
like find(1) to specify a number of characters (bytes).

Most users/scripts will specify a unit, so it is expected not to cause
significant upset with this change.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Change-Id: I3124e667acc06928f41a3d3006e1d9b4a43ebbe5
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/52994


Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Reviewed-by: Anjus George <georgea@ornl.gov>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Do not assign newmod a value past the end of the allocated
space. This can confuse coverity.

Instead only assign valid addresses (or NULL).

CoverityID: 410235 ("Memory - illegal access")

Signed-off-by: Shaun Tancheff <shaun.tancheff@hpe.com>
Change-Id: I767ed1273ebfab68d634b3ff22b81a4621405dd2
Reviewed-on: https://review.whamcloud.com/c/fs/lustre-release/+/53315


Reviewed-by: Oleg Drokin <green@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Arshad Hussain <arshad.hussain@aeoncomputing.com>
Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>