GitLab

In lod_layout_convert() zero out lcm and lcme before constructing the
converted layout.

Lustre-commit: 6f2a1c91
Lustre-change: https://review.whamcloud.com/40153

Signed-off-by: John L. Hammond <jhammond@whamcloud.com>
Change-Id: I40f96d51cb63816a9bfc34217f02ff7c450de974
Reviewed-on: https://review.whamcloud.com/40306

Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Add function lod_striped_lookup() for
lod_striped_index_ops.dio_lookup to allow name lookup under striped
directory.

Currently this is used by subdir mount, which needs to lookup FID
of the subdir on server side.

Function lfsck_namespace_repair_dirent() should call dt_lookup() with
bottom object, because child may be shard.

Add sanity 247f.

Lustre-change: https://review.whamcloud.com/37903
Lustre-commit: 42b0304e



Change-Id: Iba844d1a34a318bcbd42b00186ed6fa9d165effc
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/38691

Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Cleanup xattr checking in mdd and lod layers for PFL.

Lustre-change: https://review.whamcloud.com/37010
Lustre-commit: f765c6ce

Reported-by: Clement Barthelemy <clement.barthelemy@nextino.eu>
Signed-off-by: Sebastien Buisson <sbuisson@ddn.com>
Change-Id: I2841b615ee304785fbf316b829d8280eefc3878a
Reviewed-by: Olaf Faaland-LLNL <faaland1@llnl.gov>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/37132

Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

Sometimes the layout of striped directories may become broken:
* creation/unlink is partially executed on some MDT.
* disk failure or stopped MDS cause some stripe inaccessible.
* software bugs.

In this situation, this directory should still be accessible,
and specially be able to migrate to other active MDTs.

This patch add this support on both server and client: don't
imply stripe FID is sane, and when stripe doesn't exist, skip
it.

Add OBD_FAIL_MDS_STRIPE_FID to simulate insane stripe FID, and
OBD_FAIL_MDS_STRIPE_CREATE to simulate stripe creation failure.

Add sanity 60h.

Lustre-change: https://review.whamcloud.com/34750
Lustre-commit: d2725563

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I8a05a0e0cef8b051a935b3fa3d3e26c0b6ef3b4a
Reviewed-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/36939

Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Files opened with O_APPEND are almost always log files,
which generally stay small and do not benefit from being
striped widely.  Additionally, PFL files accessed with
O_APPEND are fully instantiated, meaning that because the
files usually stay small, these objects usually wasted.

This patch adds special striping for files created with
O_APPEND.  This is controlled on the MDS by two new proc
variables:
mdd_append_stripe_count
mdd_append_pool

If the stripe count is set to 0 and the pool is not set,
this functionality is disabled and files created with
O_APPEND will be striped like any other file.

Lustre-change: https://review.whamcloud.com/35617
Lustre-commit: e2ac6e1e

Signed-off-by: Patrick Farrell <pfarrell@whamcloud.com>
Change-Id: I433d1b8c80488a851b8eb26c78cf5519a6cd75bf
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/37007

Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>

Though sub directories under ROOT don't inherit ROOT default
layout, they should hornor ROOT default layout in creation.

Add sub test in sanity.sh 65n to verify this.

Fixes: 0a988cae ("LU-11739 lod: don't inherit default
                    layout from root directory")

Lustre-change: https://review.whamcloud.com/35204
Lustre-commit: 693fb63a

Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I1edf91da7944a8871652df7bca2104d00f66163a
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/36370

Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

"lfs setstripe" allows setting stale flag on the last
non-stale mirror of a file, which makes the file have
no valid component to read and return IO error.

This patch fixes the above issue by disallowing that.
It also disallows "lfs mirror split" to destroy the
last non-stale mirror of a file.

This patch is back-ported from the following one:
Lustre-commit: 29be32a759f696006a539d3cff74ca55a281aa64
Lustre-change: https://review.whamcloud.com/36141



Change-Id: I6934cfe0190cd1ea83de1cf28ddf840b9f96193a
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/36195

Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Add "lfs mirror copy" command to copy a mirror's content to other
mirror(s) of a mirrored file.

Usage:

lfs mirror copy {--read-mirror|-i <id0>}
	{--write-mirror|-o <id1>[,<id2>,...]} <mirrored_file>

Options:

--read-mirror|-i <id0>
  This option indicates the content of which mirror specified by id0
  needs to be read. The id0 is the numerical unique identifier for a
  mirror.

--write-mirror|-o <id1>[,<id2>,...]
  This option indicates the content of which mirror(s) specified by
  mirror IDs needs to be written. The mirror IDs are separated with
  comma.  If the mirror id -1 is used here, it means that all mirrors
  other than the read mirror are to be written.

Note:

Be ware that the written mirror(s) will be marked as non-stale
mirror(s), be careful that after using this command, you could get a
file with non-stale mirrors while containing different contents.

This patch is back-ported from the following one:
Lustre-commit: c6e7c078
Lustre-change: https://review.whamcloud.com/33220

Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: Id138368cdb29ec14b7c03a5db3b2dd1e0db5ea37
Reviewed-on: https://review.whamcloud.com/36193

Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>

In fld_client_rpc(), if the FLD_QUERY request between MDTs fails
with -EWOUDBLOCK because the connection is lost, return -EAGAIN
to notify the caller to retry.

It also reverts the patch https://review.whamcloud.com/12586/, which
was landed on b2_6_90_0-5-g6db07f09 to avoid returning -EAGAIN from
lod_object_init() to confuse lu_object_find_at() (thinks the object
was dying when it encounters -EAGAIN). In current Lustre version,
lu_object_find_at() just returned found object and let's caller to
check whether it's dying.

Fixes: 6db07f09 ("LU-5871 lod: Do not return EAGAIN in lod_object_init")

Lustre-change: https://review.whamcloud.com/34962
Lustre-commit: e3f6111d



Change-Id: Ie83ebfdae2bd50c96a59a065f7f3c3dcfad04e42
Signed-off-by: Hongchao Zhang <hongchao@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/35661

Tested-by: jenkins <devops@whamcloud.com>
Tested-by: Maloo <maloo@whamcloud.com>
Reviewed-by: Stephan Thiell <sthiell@stanford.edu>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

lod_prep_md_striped_create() tries to allocat big
chunk of memory because
lum->lum_stripe_count == -1 and is converted to __u32.

ldo_dir_stripe_count was calculated already in lod_ah_init()

Lustre-change: https://review.whamcloud.com/33829
Lustre-commit: 622a94d5



Change-Id: Id99d9e024638dfb1b34262840d2e543c808a9cdc
Cray-bug-id: LUS-6694
Signed-off-by: Andriy Skulysh <c17819@cray.com>
Reviewed-by: Alexander Boyko <c17825@cray.com>
Tested-by: Elena Gryaznova <c17455@cray.com>
Reviewed-by: Vladimir Saveliev <c17830@cray.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Lai Siyao <lai.siyao@whamcloud.com>
Signed-off-by: Minh Diep <mdiep@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/34313


Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>

There is no need to inherit the default directory layout from
the root directory when subdirectories are created therein.
This consumes xattr space on the subdirectories, and makes it
more complex to change the filesystem default layout in the future.

This patch fixes the above issue in lod_ah_init() to check if
the parent directory is the root directory and not copy
the default layout xattr to the new subdirectory.

Lustre-change: https://review.whamcloud.com/33956
Lustre-commit: 0a988cae

Lustre-change: https://review.whamcloud.com/34175


Lustre-commit: ad1a74527f0ec59510bfa124b8280617a2b93840

Change-Id: Ie0d286785bdbcd73e2ae60b429e66d5d54b44eef
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <pfarrell@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/34135


Tested-by: Jenkins
Tested-by: Maloo <maloo@whamcloud.com>

Rename "lfs mirror dump" command to "lfs mirror read".

Add "lfs mirror write" command to write a mirror's content of a
mirrored file.

Usage:

lfs mirror write {--mirror-id|-N <mirror_id>}
	[--inputfile|-i <input_file>] <mirrored_file>

Options:

--mirror-id|-N <mirror_id>
  This  option  indicates  the  content of which mirror specified by
  mirror_id needs to be written. The mirror_id is the numerical unique
  identifier for a mirror.

--inputfile|-i <input_file>
  The path name of the input file, if not specified, the standard
  input stream will be used. The input stream or input_file cannot
  be the same mirrored file as the mirrored_file.

This command will issue a RESYNC lease write lock to notify MDS to
prepare destination mirror for the write (instantiate components of
the mirror), then client copy data from a file or STDIN to the
specified mirror of the mirrored file. After the data copy, a
RESYNC_DONE lease unlock is issued to MDS to update the layout
of the mirrored file.
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I02022349d4ce871319903a8714ffc4534186c0e4
Reviewed-on: https://review.whamcloud.com/33219


Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

If directory migration failed, it will be marked
LMV_HASH_FLAG_MIGRATION, lod_parse_dir_striping() should parse
layout for such directory, otherwise such directory can't be
accessed correctly.
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I5ee2c2e3e5aa3f9befc3ad81be4bcdc6f9267842
Reviewed-on: https://review.whamcloud.com/33456


Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

This patch will create MDT layout in the path of mdo_create() before
mdt_object_open_lock() is invoked. The previous implementation created
layout in mdt_create_data() that caused the problem that layout lock
couldn't be packed in the reply of the open request. Later on an extra
layout request has to be issued for layout lock, which kills all
performance gains by DoM for small files write.
Signed-off-by: Jinshan Xiong <jinshan.xiong@uber.com>
Change-Id: Id11ac79c89d12bbe0e925fbc89417fca3e72e479
Reviewed-on: https://review.whamcloud.com/33233


Tested-by: Jenkins
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

In directory migration, when it needs to update OST object PFID,
it should always do via source object, because target object may
be remote. And in this case, lod_obj_stripe_replace_parent_fid_cb()
doesn't compare parent FID with that in XATTR_NAME_FID, but set
with the passed in FID directly.
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: Icd7f6521ecac43cfeaee3e61e662d94115d63d68
Reviewed-on: https://review.whamcloud.com/33324


Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

New files created in directories with a default layout
should inherit the per-component layout flags.

This allows us to set the prefer or nosync flags in a
default layout and apply them to files created in that
directory.

Cray-bug-id: LUS-6574
Signed-off-by: Patrick Farrell <paf@cray.com>
Change-Id: I053ca0f3db3e0967799f469feeb4f1f12b144be7
Reviewed-on: https://review.whamcloud.com/33326


Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Remove the unused 'ignore quota' parameters from dt_write(),
dt_insert(), and their implementations.
Signed-off-by: John L. Hammond <john.hammond@intel.com>
Change-Id: Ib6e61c5dff8c2b1027ed3a7e0808fe208fd17d84
Reviewed-on: https://review.whamcloud.com/33138


Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

This patch allows 'nosync' flag to be set for FLR mirror components,
which makes lfs mirror resync skip on mirrors with this flag unless
mirror resync explicitly requested those mirrors to be resync.

This flag can be cleared by set '^nosync' on any component of the
mirror.
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I209ad91fd9a96d82f8c83a3eb047f665ef4ccf86
Reviewed-on: https://review.whamcloud.com/33205


Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

to save on env allocation.

Benchmarks made by Shuichi Ihara demonstrated 13% improvement
for small I/Os: 564k vs 639k IOPS. the details can be found
LU-11164.

Change-Id: I797e3d7e19ef408993004a2b872842d655240525
Signed-off-by: Alex Zhuravlev <bzzz@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/32832


Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Patrick Farrell <paf@cray.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Use setxattr(XATTR_NAME_LMV) to shrink dir layout after migration,
this may change dir fid because it may shrink dir to 1-stripe, if so
replace this dir with its stripe.
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I4fd3e83e188ac0c6c4845183c17701276fae94a8
Reviewed-on: https://review.whamcloud.com/31626

Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Migrate striped directory in below steps:
1. create target object if needed: if source is directory, a
   target object is always created, otherwise if source is
   already located on the target MDT, or source still has
   link on source MDT, then skip creating.
	a) if source is directory, detach source stripes and
	   attach them to target.
	b) migrate source xattrs to target.
	c) if source is regular file, update PFID to target
	   fid.
	d) update fid to target for all links of source
2. update namespace
	a) migrate dirent from source parent to target parent.
	b) update linkea parent fid to target parent.
	c) destroy source object.

This implementation improves following fields:
1. all involved objects are locked to avoid race.
2. directory migration doesn't migrate its dir entries, instead
   it's done in each sub file migration, this avoids timeout in
   migrating dir entries for large directory, and also avoids
   touching dir entries without lock.
3. file/dir is migrated in one transaction, so migrate recovery
   is the same as others.
4. migrating directory can be accessed (modifiable) like normal
   directory.
5. if migration of sub files under a directory fails, user can
   redo migrate to finish migration of this directory.

Disable migrate in racer.sh.
Signed-off-by: Lai Siyao <lai.siyao@whamcloud.com>
Change-Id: I7906e50a0bf55375eafdf2cf5500979dd2d94d48
Reviewed-on: https://review.whamcloud.com/31427


Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Ostlist might be allocated previously, and we should
reset them properly, otherwise, it will pollute new
default setting and cause unexpected behavior.

Test-Parameters: trivial testlist=sanity,sanity,sanity,sanity,sanity,sanity,sanity,sanity,sanity
Change-Id: I9b7acb5f05ec4b371da99f68b9647f0b75cd7021
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/33069

Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Replay of a layout change request on a PFL file leaves the object
in an unexpected state: Some components can have llc_stripe set
but ldo_comp_cached is not set in the object. The next layout
change request on the same object will LBUG when it tries to free
the comp entries.

The fix is to set ldo_comp_cached on replay so subsequent layout
change requests will use the in memory components rather than
fetching them from disk.
Signed-off-by: Ann Koehler <amk@cray.com>
Change-Id: I8eaee5614c7f2f6e6a3f2c51de93a65422a3122b
Reviewed-on: https://review.whamcloud.com/32904


Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

MDT can act as a proxy for statfs data. this should
make df faster (RTT vs RTT*(#MDTs+1)) and enable
idling connections so that clients don't connect to
each OST just to report statfs data. the protocol
has been changing slightly to let MDT differentiate
self and aggregated statfs.

also, obd_statfs has got a new field "granted" where
OST reports how much space has been granted to the
requesting MDT so that space can be added to available
space.

client's NID is used to distribute MDS_STATFS among
MDTS.

Change-Id: I59e03cb5abf809ae8820f874ec51dd2b74e1806c
Signed-off-by: Alex Zhuravlev <alexey.zhuravlev@intel.com>
Reviewed-on: https://review.whamcloud.com/29136

Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Mike Pershin <mpershin@whamcloud.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

When merging/expanding existing mirrors of a FLR file, we need keep
its existing mirror's mirror ID.
Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: If139076c37c33bb1a330e1a5e997f8f56015fd9a
Reviewed-on: https://review.whamcloud.com/32938


Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

LOV_USER_MAGIC_SPECIFIC function is broken and it
was not available for setting directory.

1)llite doesn't handle LOV_USER_MAGIC_SPECIFIC case
properly for dir {set,get}_stripe, and ioctl
LL_IOC_LOV_SETSTRIPE did not alloc enough buf,
copy ost lists from userspace.

2)lod_get_default_lov_striping() did not handle
LOV_USER_MAGIC_SPECIFIC type that newly created
files/dir won't inherit parent setting well.

3)there is not any case to cover lfs setstripe
'-o' interface which make it hard to figure out
when this function was broken.

Change-Id: Icc2ee60a474e5e565db12b35a9a38fde65b05bbd
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-on: https://review.whamcloud.com/32814


Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Jian Yu <yujian@whamcloud.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Use lod_layout_component::llc_ost_indices to serve the same purpose.
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Change-Id: I66c89fe6349b48b89593e34e9e985ec6ea5a1758
Reviewed-on: https://review.whamcloud.com/32813

Reviewed-by: Patrick Farrell <paf@cray.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@gmail.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

lod_object layout is loaded on demand, and it may be updated
by layout split/merge. To avoid race, add ldo_layouyt_mutex to
serialize layout load/free/reload.
Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Change-Id: I43c15a3b07254eadef95a14b288267904a1cd621
Reviewed-on: https://review.whamcloud.com/32589

Reviewed-by: Alex Zhuravlev <bzzz@whamcloud.com>
Tested-by: Jenkins
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

While choosing OST to create object, both lod_alloc_qos() and
lod_alloc_rr() functions use lod_statfs_and_check() function
to check whether the OST is available for new OST objects or not.
However, OST with max_create_count=0 is not checked in that
function and just returned as an available OST.

This patch fixes the above issue by detecting OST with
max_create_count=0 in lod_statfs_and_check() and skip it.

Change-Id: I04476a4b369e99133bd89c00155fd9f51bf0c930
Signed-off-by: Jian Yu <yujian@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/32823

Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Tested-by: Jenkins
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

* add lod_layout_component::llc_ost_indices to track the map
  of dt_object to its OST index.
* add lod_device::lod_avoid to collect information of objects on other
  mirrors which overlapped the target component
* lod_should_avoid_ost() use the avoid guidance information to avoid
  allocating objects on the same OST for different mirrors.

Change-Id: Ib7e155e4b02c2e25d3955aa9a4acff7569ab7d8f
Signed-off-by: Bobi Jam <bobijam@whamcloud.com>
Reviewed-on: https://review.whamcloud.com/32404

Reviewed-by: Jinshan Xiong <jinshan.xiong@gmail.com>
Reviewed-by: Patrick Farrell <paf@cray.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <green@whamcloud.com>

Striped directory has an implication that the first stripe is
local, and others are remote, but this is not true for migrating
directory because its stripes consists of both the original and
the newly created stripes.

This patch also put striped directory master object locking and
stripes locking into one function called mdt_reint_striped_lock(),
which simplifies locking code.
Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Change-Id: I4724447e5b10c301b6799e1827f6d13a40876945
Reviewed-on: https://review.whamcloud.com/31425

Reviewed-by: Fan Yong <fan.yong@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

When setstripe -d is run on a directory, we should
clear the OST pool along with all the other settings
Currently there is no way to clear an OST pool,
only change them.
Signed-off-by: Ben Evans <bevans@cray.com>
Cray-bug-id: LUS-5696
Change-Id: I50426ce79ab153a715d29cc5d54b0ce70726da41
Reviewed-on: https://review.whamcloud.com/31364

Reviewed-by: Jinshan Xiong <jinshan.xiong@gmail.com>
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

When some components are failed to resync due to various reasons,
those components will still have STALE bit set but the file statue may
become to RDONLY.

This patch makes resync RDONLY FLR file possible.
Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I2e3b518bb969aedd7f214e6b09b895079cab69ab
Reviewed-on: https://review.whamcloud.com/31010


Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

As a mirrored file being written for the first time, MDS will choose
a mirror to write the data, a primary choosing policy function is
defined in this patch (lod_primary_pick()) to avoid the mirror with
unavailable OSTs.
Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I5d6d0459e96583294c3040a7994c33114be1e439
Reviewed-on: https://review.whamcloud.com/30711

Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Lai Siyao <lai.siyao@intel.com>
Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

In PFL design, client issues [0, size) intent to MDS to instantiate
objects in this extent. While in FLR design, the intent serves two
purposes: 1) make objects across [size, EOF) on other mirrors stale,
and 2) instantiate objects in the chosen write mirror. And original
FLR chose to use [size, EOF) as the extent of truncate write intent
request.

This patch reverts the choice, and still uses [0, size) as the
truncate write intent extent.
Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: I12d90320ba704f01457670f864d78fe764233000
Reviewed-on: https://review.whamcloud.com/30786

Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

Having "lfs getstripe" print out "lcm_flags: not_flr" is not very
useful, as the composite files may not relate to FLR (e.g. PFL).
Rename "LCM_FL_NOT_FLR" to "LCM_FL_NONE" so it is more clear there
are no composite flags on the layout.

Print out "0" for flags if no flags are set, to match old behaviour.
Signed-off-by: Andreas Dilger <andreas.dilger@intel.com>
Change-Id: Ib3701da8368253969567b927300cd42bc33ebbe5
Reviewed-on: https://review.whamcloud.com/31047

Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Tested-by: Jenkins
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

Splits a mirror with mirror_id out of a mirrored file.
Signed-off-by: Bobi Jam <bobijam.xu@intel.com>
Change-Id: Ib9c2ca7deb329ba0f95880ebeee77563317d0fca
Reviewed-on: https://review.whamcloud.com/30388

Reviewed-by: Fan Yong <fan.yong@intel.com>
Reviewed-by: Jinshan Xiong <jinshan.xiong@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

lfs --component-add -E <end> -c -1 would apply the default
stripe count to a file when it should be striping across
all OSTs. This was due to incorrect handling of sentinel
values in various portions of lod_object.
Signed-off-by: Giuseppe Di Natale <dinatale2@llnl.gov>
Change-Id: I178f13be81ee546f44edcdadcf7086a1a57f7e9a
Reviewed-on: https://review.whamcloud.com/30790


Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

If a file is created (with lfs) in the directory associated
with pool without -p pool_name option then limit stripe count
to number of osts in the pool as that directory is associated
with the pool. This patch fixes this problem.

Also removed the wrong check from ost-pools.sh, test_20 where
we were creating file in a directory associated with pool and
checking it as not part of the pool.

Add test cases in ost_pool.sh test_20.
Signed-off-by: Rahul Deshmukh <rahul.deshmukh@seagate.com>
Signed-off-by: Lai Siyao <lai.siyao@intel.com>
Signed-off-by: Hongchao Zhang <hongchao.zhang@intel.com>
Seagate-bug-id: MRP-3615
Change-Id: Id6dd5126856db7fc773a1fe9c837a214db8d6d70
Reviewed-on: https://review.whamcloud.com/20849

Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>

This patch will allow flags to be set when creating mirrors.
The flags are set to individual components therefore it would be
flexible to flags based on the location of components. Also, 'stale'
and 'prefer' flags are allowed to set to individual components later
on.

This patch also revises component flags matching rules to allow
flags and inverted flags to be set at the same time in the command
lfs-find(1) and lfs-getstripe(1).
Signed-off-by: Jinshan Xiong <jinshan.xiong@intel.com>
Change-Id: Ia077ca5454d49eb411bd82bd451c9dfc426d780c
Reviewed-on: https://review.whamcloud.com/30360

Reviewed-by: Andreas Dilger <andreas.dilger@intel.com>
Reviewed-by: Bobi Jam <bobijam@hotmail.com>
Reviewed-by: Jian Yu <jian.yu@intel.com>
Tested-by: Jenkins
Tested-by: Maloo <hpdd-maloo@intel.com>
Reviewed-by: Oleg Drokin <oleg.drokin@intel.com>