diff options
| author | Linus Torvalds <torvalds@linux-foundation.org> | 2026-06-16 13:02:47 +0530 |
|---|---|---|
| committer | Linus Torvalds <torvalds@linux-foundation.org> | 2026-06-16 13:02:47 +0530 |
| commit | ba9c792c824fff732df85119011d399d9b6d9155 (patch) | |
| tree | a6de1eda726e4e9156366b6fd960b81d09297aa3 | |
| parent | 9b40ba14edcdf70240af8114092a76f75f070774 (diff) | |
| parent | c7c76f9232bd34835d821f14abdc5fafc17bc938 (diff) | |
Merge tag 'for-7.2/block-20260615' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux
Pull block updates from Jens Axboe:
- NVMe pull request via Keith:
- Per-controller admin and IO timeout sysfs attributes, and
letting the block layer set request timeouts (Maurizio,
Maximilian)
- Multipath passthrough iostats, and PCI P2PDMA enablement for
multipath devices (Keith, Kiran)
- A new diag sysfs attribute group exporting per-controller
counters (retries, multipath failover, error counters, requeue
and failure counts, reset and reconnect events) (Nilay)
- FDP configuration validation and bounds check fixes (liuxixin)
- Various nvmet fixes, including a pre-auth out-of-bounds read in
the Discovery Get Log Page handler, auth payload bounds
validation, and tcp error-path leak fixes (Bryam, Tianchu,
Geliang)
- nvme-tcp lockdep and workqueue fixes (Shin'ichiro, Kuniyuki,
Eric)
- Assorted other fixes and cleanups (John, Yao, Chao, Mateusz,
Achkinazi, Wentao)
- MD pull request via Yu Kuai:
- raid1/raid10 fixes for a deadlock in the read error recovery
path, error-path detection and bio accounting with cloned bios,
and an nr_pending leak in the REQ_ATOMIC bad-block error path
(Abd-Alrhman)
- PCI P2PDMA propagation from member devices to the RAID device
(Kiran)
- dm-raid bio requeue fix, and various smaller fixes and cleanups
(Benjamin, Chen, Li, Thorsten)
- Enable Clang lock context analysis for the block layer, with the
accompanying annotations across queue limits, the blk_holder_ops
callbacks, crypto, cgroup, iocost, kyber and mq-deadline (Bart)
- Block status code infrastructure work: a tagged status table, a
str_to_blk_op() helper, a bio_endio_status() helper, and on top of
that a new configurable block-layer error injection facility
(Christoph)
- DRBD netlink rework, replacing the genl_magic machinery with explicit
netlink serialization and moving the DRBD UAPI headers to
include/uapi/linux/ (Christoph Böhmwalder)
- bvec improvements: a bvec_folio() helper and making the bvec_iter
helpers proper inline functions (Willy, Christoph)
- ublk cleanups and a canceling-flag fix for the disk-not-allocated
case (Caleb, Ming)
- Partition handling fixes: bound the AIX pp_count scan, fix an of_node
refcount leak, and replace __get_free_page() with kmalloc() (Bryam,
Wentao, Mike)
- Convert numa_node to int in blk_mq_hw_ctx and ->init_request, and add
WQ_PERCPU to the block workqueue users (Mateusz, Marco)
- Block statistics and tracing: propagate in-flight to the whole disk
on partition IO, export passthrough stats, and a new
block_rq_tag_wait tracepoint (Tang, Keith, Aaron)
- A round of removals, unexports and cleanups across bio, direct-io and
the bvec helpers (Christoph)
- Various driver fixes (mtip32xx use-after-free, rbd snap_count
validation and strscpy conversion, nbd socket lockdep reclassify,
virtio-blk zone report clamp, floppy) and a batch of MAINTAINERS
email/list updates (Coly, Li, Yu, Christoph Böhmwalder)
- Other little fixes and cleanups all over
* tag 'for-7.2/block-20260615' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (117 commits)
MAINTAINERS: Update Coly Li's email address
block: check bio split for unaligned bvec
nbd: Reclassify sockets to avoid lockdep circular dependency
block: add configurable error injection
block: add a str_to_blk_op helper
block: add a "tag" for block status codes
block: add a macro to initialize the status table
floppy: Drop unused pnp driver data
block: propagate in_flight to whole disk on partition I/O
virtio-blk: clamp zone report to the report buffer capacity
block: optimize I/O merge hot path with unlikely() hints
drivers/block/rbd: Use strscpy() to copy strings into arrays
partitions: aix: bound the pp_count scan to the ppe array
block: Enable lock context analysis
block/mq-deadline: Make the lock context annotations compatible with Clang
block/Kyber: Make the lock context annotations compatible with Clang
block/blk-mq-debugfs: Improve lock context annotations
block/blk-iocost: Inline iocg_lock() and iocg_unlock()
block/blk-iocost: Split ioc_rqos_throttle()
block/crypto: Annotate the crypto functions
...
98 files changed, 5780 insertions, 2321 deletions
diff --git a/Documentation/block/error-injection.rst b/Documentation/block/error-injection.rst new file mode 100644 index 000000000000..81f31af82e65 --- /dev/null +++ b/Documentation/block/error-injection.rst @@ -0,0 +1,59 @@ +.. SPDX-License-Identifier: GPL-2.0 + +============================ +Configurable Error Injection +============================ + +Overview +-------- + +Configurable error injection allows injecting specific block layer status codes +for sector ranges of a block device. Errors can be injected unconditionally, or +with a given probability. + +To use configurable error injection, CONFIG_BLK_ERROR_INJECTION must be enabled. + +The only interface is the error_injection debugfs file, which is created for +each registered gendisk. Writes to this file are used to create or delete rules +and reads return a list of the current error injection sites. + +Options +------- + +The following options specify the operations: + +=================== ======================================================= +add add a new rule +removeall remove all existing rules +=================== ======================================================= + +The following options specify the details of the rule for the add operation: + +=================== ======================================================= +op=<string> block layer operation this rule applies to. This uses + the XYZ for each REQ_OP_XYZ operation, e.g. READ, WRITE + or DISCARD. Mandatory. +status=<string> Status to return. This uses XYZ for each BLK_STS_XYZ + code, e.g. IOERR or MEDIUM. Mandatory. +start=<number> First block layer sector the rule applies to. + Optional, defaults to 0. +nr_sectors=<number> Number of sectors this rule applies. + Optional, defaults to the remainder of the device. +chance=<number> Only return a failure with a likelihood of 1/chance. + Optional, defaults to 1 (always). +=================== ======================================================= + +Example +------- + +Return BLK_STS_IOERR for one in 10 reads of sector 0 of /dev/nvme0n1: + + $ echo 'add,op=READ,start=0,status=IOERR,chance=10' > /sys/kernel/debug/block/nvme0n1/error_injection + +Return BLK_STS_MEDIUM for every write to /dev/nvme0n1: + + $ echo 'add,op=WRITE,start=0,status=MEDIUM' > /sys/kernel/debug/block/nvme0n1/error_injection + +Remove all rules for /dev/nvme0n1: + + $ echo 'removeall' > /sys/kernel/debug/block/nvme0n1/error_injection diff --git a/Documentation/block/index.rst b/Documentation/block/index.rst index 9fea696f9daa..bfa1bbd31ddf 100644 --- a/Documentation/block/index.rst +++ b/Documentation/block/index.rst @@ -22,3 +22,4 @@ Block switching-sched writeback_cache_control ublk + error-injection diff --git a/Documentation/core-api/kernel-api.rst b/Documentation/core-api/kernel-api.rst index e8211c4ca662..4c4a57c1c094 100644 --- a/Documentation/core-api/kernel-api.rst +++ b/Documentation/core-api/kernel-api.rst @@ -307,6 +307,7 @@ Accounting Framework Block Devices ============= +.. kernel-doc:: include/linux/bvec.h .. kernel-doc:: include/linux/bio.h .. kernel-doc:: block/blk-core.c :export: diff --git a/MAINTAINERS b/MAINTAINERS index 025e566e813a..3d6db8cb608f 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -4462,7 +4462,7 @@ F: include/uapi/linux/batman_adv.h F: net/batman-adv/ BCACHE (BLOCK LAYER CACHE) -M: Coly Li <colyli@fnnas.com> +M: Coly Li <colyli@fygo.io> M: Kent Overstreet <kent.overstreet@linux.dev> L: linux-bcache@vger.kernel.org S: Maintained @@ -4500,7 +4500,7 @@ F: Documentation/filesystems/befs.rst F: fs/befs/ BFQ I/O SCHEDULER -M: Yu Kuai <yukuai@fnnas.com> +M: Yu Kuai <yukuai@fygo.io> L: linux-block@vger.kernel.org S: Odd Fixes F: Documentation/block/bfq-iosched.rst @@ -7804,7 +7804,7 @@ DRBD DRIVER M: Philipp Reisner <philipp.reisner@linbit.com> M: Lars Ellenberg <lars.ellenberg@linbit.com> M: Christoph Böhmwalder <christoph.boehmwalder@linbit.com> -L: drbd-dev@lists.linbit.com +L: drbd-dev@lists.linux.dev S: Supported W: http://www.drbd.org T: git git://git.linbit.com/linux-drbd.git @@ -24896,8 +24896,8 @@ F: include/linux/property.h SOFTWARE RAID (Multiple Disks) SUPPORT M: Song Liu <song@kernel.org> -M: Yu Kuai <yukuai@fnnas.com> -R: Li Nan <linan122@huawei.com> +M: Yu Kuai <yukuai@fygo.io> +R: Li Nan <magiclinan@didiglobal.com> R: Xiao Ni <xiao@kernel.org> L: linux-raid@vger.kernel.org S: Supported diff --git a/block/Kconfig b/block/Kconfig index 15027963472d..70e4a66d941f 100644 --- a/block/Kconfig +++ b/block/Kconfig @@ -221,6 +221,14 @@ config BLOCK_HOLDER_DEPRECATED config BLK_MQ_STACKING bool +config BLK_ERROR_INJECTION + bool "Enable block layer error injection" + select JUMP_LABEL if HAVE_ARCH_JUMP_LABEL + help + Enable inserting arbitrary block errors through a debugfs interface. + + See Documentation/block/error-injection.rst for details. + source "block/Kconfig.iosched" endif # BLOCK diff --git a/block/Makefile b/block/Makefile index 7dce2e44276c..e7bd320e3d69 100644 --- a/block/Makefile +++ b/block/Makefile @@ -3,6 +3,8 @@ # Makefile for the kernel block layer # +CONTEXT_ANALYSIS := y + obj-y := bdev.o fops.o bio.o elevator.o blk-core.o blk-sysfs.o \ blk-flush.o blk-settings.o blk-ioc.o blk-map.o \ blk-merge.o blk-timeout.o blk-lib.o blk-mq.o \ @@ -11,6 +13,7 @@ obj-y := bdev.o fops.o bio.o elevator.o blk-core.o blk-sysfs.o \ genhd.o ioprio.o badblocks.o partitions/ blk-rq-qos.o \ disk-events.o blk-ia-ranges.o early-lookup.o +obj-$(CONFIG_BLK_ERROR_INJECTION) += error-injection.o obj-$(CONFIG_BLK_DEV_BSG_COMMON) += bsg.o obj-$(CONFIG_BLK_DEV_BSGLIB) += bsg-lib.o obj-$(CONFIG_BLK_CGROUP) += blk-cgroup.o diff --git a/block/bdev.c b/block/bdev.c index bb0ffa3bb4df..85ce57bd2ae4 100644 --- a/block/bdev.c +++ b/block/bdev.c @@ -446,15 +446,10 @@ EXPORT_SYMBOL_GPL(blockdev_superblock); void __init bdev_cache_init(void) { - int err; - bdev_cachep = kmem_cache_create("bdev_cache", sizeof(struct bdev_inode), 0, (SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT| SLAB_ACCOUNT|SLAB_PANIC), init_once); - err = register_filesystem(&bd_type); - if (err) - panic("Cannot register bdev pseudo-fs"); blockdev_mnt = kern_mount(&bd_type); if (IS_ERR(blockdev_mnt)) panic("Cannot create bdev pseudo-fs"); @@ -1250,7 +1245,13 @@ void bdev_mark_dead(struct block_device *bdev, bool surprise) bdev->bd_holder_ops->mark_dead(bdev, surprise); else { mutex_unlock(&bdev->bd_holder_lock); - sync_blockdev(bdev); + /* + * On surprise removal the device is already gone; syncing is + * futile and can hang forever waiting on I/O that will never + * complete. Match fs_bdev_mark_dead(), which also skips it. + */ + if (!surprise) + sync_blockdev(bdev); } invalidate_bdev(bdev); diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c index ac83b0668764..0bd0332b3d78 100644 --- a/block/bfq-cgroup.c +++ b/block/bfq-cgroup.c @@ -300,6 +300,25 @@ static struct bfq_group *bfqg_parent(struct bfq_group *bfqg) return pblkg ? blkg_to_bfqg(pblkg) : NULL; } +static void bfqg_stats_exit(struct bfqg_stats *stats) +{ + blkg_rwstat_exit(&stats->bytes); + blkg_rwstat_exit(&stats->ios); +#ifdef CONFIG_BFQ_CGROUP_DEBUG + blkg_rwstat_exit(&stats->merged); + blkg_rwstat_exit(&stats->service_time); + blkg_rwstat_exit(&stats->wait_time); + blkg_rwstat_exit(&stats->queued); + bfq_stat_exit(&stats->time); + bfq_stat_exit(&stats->avg_queue_size_sum); + bfq_stat_exit(&stats->avg_queue_size_samples); + bfq_stat_exit(&stats->dequeue); + bfq_stat_exit(&stats->group_wait_time); + bfq_stat_exit(&stats->idle_time); + bfq_stat_exit(&stats->empty_time); +#endif +} + struct bfq_group *bfqq_group(struct bfq_queue *bfqq) { struct bfq_entity *group_entity = bfqq->entity.parent; @@ -321,8 +340,10 @@ static void bfqg_get(struct bfq_group *bfqg) static void bfqg_put(struct bfq_group *bfqg) { - if (refcount_dec_and_test(&bfqg->ref)) + if (refcount_dec_and_test(&bfqg->ref)) { + bfqg_stats_exit(&bfqg->stats); kfree(bfqg); + } } static void bfqg_and_blkg_get(struct bfq_group *bfqg) @@ -433,25 +454,6 @@ void bfq_init_entity(struct bfq_entity *entity, struct bfq_group *bfqg) entity->sched_data = &bfqg->sched_data; } -static void bfqg_stats_exit(struct bfqg_stats *stats) -{ - blkg_rwstat_exit(&stats->bytes); - blkg_rwstat_exit(&stats->ios); -#ifdef CONFIG_BFQ_CGROUP_DEBUG - blkg_rwstat_exit(&stats->merged); - blkg_rwstat_exit(&stats->service_time); - blkg_rwstat_exit(&stats->wait_time); - blkg_rwstat_exit(&stats->queued); - bfq_stat_exit(&stats->time); - bfq_stat_exit(&stats->avg_queue_size_sum); - bfq_stat_exit(&stats->avg_queue_size_samples); - bfq_stat_exit(&stats->dequeue); - bfq_stat_exit(&stats->group_wait_time); - bfq_stat_exit(&stats->idle_time); - bfq_stat_exit(&stats->empty_time); -#endif -} - static int bfqg_stats_init(struct bfqg_stats *stats, gfp_t gfp) { if (blkg_rwstat_init(&stats->bytes, gfp) || @@ -552,7 +554,6 @@ static void bfq_pd_free(struct blkg_policy_data *pd) { struct bfq_group *bfqg = pd_to_bfqg(pd); - bfqg_stats_exit(&bfqg->stats); bfqg_put(bfqg); } @@ -1051,9 +1052,13 @@ static ssize_t bfq_io_set_device_weight(struct kernfs_open_file *of, blkg_conf_init(&ctx, buf); + ret = blkg_conf_open_bdev(&ctx); + if (ret) + return ret; + ret = blkg_conf_prep(blkcg, &blkcg_policy_bfq, &ctx); if (ret) - goto out; + goto close_bdev; if (sscanf(ctx.body, "%llu", &v) == 1) { /* require "default" on dfl */ @@ -1074,8 +1079,11 @@ static ssize_t bfq_io_set_device_weight(struct kernfs_open_file *of, bfq_group_set_weight(bfqg, bfqg->entity.weight, v); ret = 0; } + out: - blkg_conf_exit(&ctx); + blkg_conf_unprep(&ctx); +close_bdev: + blkg_conf_close_bdev(&ctx); return ret ?: nbytes; } diff --git a/block/bio.c b/block/bio.c index 5f10900b3f42..811a96796202 100644 --- a/block/bio.c +++ b/block/bio.c @@ -635,15 +635,15 @@ struct bio *bio_kmalloc(unsigned short nr_vecs, gfp_t gfp_mask) } EXPORT_SYMBOL(bio_kmalloc); -void zero_fill_bio_iter(struct bio *bio, struct bvec_iter start) +void zero_fill_bio(struct bio *bio) { struct bio_vec bv; struct bvec_iter iter; - __bio_for_each_segment(bv, bio, iter, start) + bio_for_each_segment(bv, bio, iter) memzero_bvec(&bv); } -EXPORT_SYMBOL(zero_fill_bio_iter); +EXPORT_SYMBOL(zero_fill_bio); /** * bio_truncate - truncate the bio to small size of @new_size @@ -1300,7 +1300,7 @@ static void bio_free_folios(struct bio *bio) int i; bio_for_each_bvec_all(bv, bio, i) { - struct folio *folio = page_folio(bv->bv_page); + struct folio *folio = bvec_folio(bv); if (!is_zero_folio(folio)) folio_put(folio); @@ -1409,7 +1409,7 @@ int bio_iov_iter_bounce(struct bio *bio, struct iov_iter *iter, size_t maxlen, static void bvec_unpin(struct bio_vec *bv, bool mark_dirty) { - struct folio *folio = page_folio(bv->bv_page); + struct folio *folio = bvec_folio(bv); size_t nr_pages = (bv->bv_offset + bv->bv_len - 1) / PAGE_SIZE - bv->bv_offset / PAGE_SIZE + 1; @@ -1443,7 +1443,7 @@ static void bio_iov_iter_unbounce_read(struct bio *bio, bool is_error, bvec_unpin(&bio->bi_io_vec[1 + i], mark_dirty); } - folio_put(page_folio(bio->bi_io_vec[0].bv_page)); + folio_put(bvec_folio(&bio->bi_io_vec[0])); } /** @@ -1578,26 +1578,6 @@ void __bio_advance(struct bio *bio, unsigned bytes) } EXPORT_SYMBOL(__bio_advance); -void bio_copy_data_iter(struct bio *dst, struct bvec_iter *dst_iter, - struct bio *src, struct bvec_iter *src_iter) -{ - while (src_iter->bi_size && dst_iter->bi_size) { - struct bio_vec src_bv = bio_iter_iovec(src, *src_iter); - struct bio_vec dst_bv = bio_iter_iovec(dst, *dst_iter); - unsigned int bytes = min(src_bv.bv_len, dst_bv.bv_len); - void *src_buf = bvec_kmap_local(&src_bv); - void *dst_buf = bvec_kmap_local(&dst_bv); - - memcpy(dst_buf, src_buf, bytes); - - kunmap_local(dst_buf); - kunmap_local(src_buf); - - bio_advance_iter_single(src, src_iter, bytes); - bio_advance_iter_single(dst, dst_iter, bytes); - } -} -EXPORT_SYMBOL(bio_copy_data_iter); /** * bio_copy_data - copy contents of data buffers from one bio to another @@ -1612,7 +1592,21 @@ void bio_copy_data(struct bio *dst, struct bio *src) struct bvec_iter src_iter = src->bi_iter; struct bvec_iter dst_iter = dst->bi_iter; - bio_copy_data_iter(dst, &dst_iter, src, &src_iter); + while (src_iter.bi_size && dst_iter.bi_size) { + struct bio_vec src_bv = bio_iter_iovec(src, src_iter); + struct bio_vec dst_bv = bio_iter_iovec(dst, dst_iter); + unsigned int bytes = min(src_bv.bv_len, dst_bv.bv_len); + void *src_buf = bvec_kmap_local(&src_bv); + void *dst_buf = bvec_kmap_local(&dst_bv); + + memcpy(dst_buf, src_buf, bytes); + + kunmap_local(dst_buf); + kunmap_local(src_buf); + + bio_advance_iter_single(src, &src_iter, bytes); + bio_advance_iter_single(dst, &dst_iter, bytes); + } } EXPORT_SYMBOL(bio_copy_data); @@ -1659,7 +1653,6 @@ void bio_set_pages_dirty(struct bio *bio) folio_unlock(fi.folio); } } -EXPORT_SYMBOL_GPL(bio_set_pages_dirty); /* * bio_check_pages_dirty() will check that all the BIO's pages are still dirty. @@ -1718,7 +1711,6 @@ defer: spin_unlock_irqrestore(&bio_dirty_lock, flags); schedule_work(&bio_dirty_work); } -EXPORT_SYMBOL_GPL(bio_check_pages_dirty); static inline bool bio_remaining_done(struct bio *bio) { @@ -1884,7 +1876,7 @@ EXPORT_SYMBOL_GPL(bio_trim); * create memory pools for biovec's in a bio_set. * use the global biovec slabs created for general use. */ -int biovec_init_pool(mempool_t *pool, int pool_entries) +static int biovec_init_pool(mempool_t *pool, int pool_entries) { struct biovec_slab *bp = bvec_slabs + ARRAY_SIZE(bvec_slabs) - 1; diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index bc63bd220865..3093c1c03902 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -755,7 +755,7 @@ EXPORT_SYMBOL_GPL(__blkg_prfill_u64); * * Initialize @ctx which can be used to parse blkg config input string @input. * Once initialized, @ctx can be used with blkg_conf_open_bdev() and - * blkg_conf_prep(), and must be cleaned up with blkg_conf_exit(). + * blkg_conf_prep(). */ void blkg_conf_init(struct blkg_conf_ctx *ctx, char *input) { @@ -771,10 +771,7 @@ EXPORT_SYMBOL_GPL(blkg_conf_init); * @ctx->input and get and store the matching bdev in @ctx->bdev. @ctx->body is * set to point past the device node prefix. * - * This function may be called multiple times on @ctx and the extra calls become - * NOOPs. blkg_conf_prep() implicitly calls this function. Use this function - * explicitly if bdev access is needed without resolving the blkcg / policy part - * of @ctx->input. Returns -errno on error. + * Returns: -errno on error. */ int blkg_conf_open_bdev(struct blkg_conf_ctx *ctx) { @@ -783,8 +780,8 @@ int blkg_conf_open_bdev(struct blkg_conf_ctx *ctx) struct block_device *bdev; int key_len; - if (ctx->bdev) - return 0; + if (WARN_ON_ONCE(ctx->bdev)) + return -EINVAL; if (sscanf(input, "%u:%u%n", &major, &minor, &key_len) != 2) return -EINVAL; @@ -813,38 +810,7 @@ int blkg_conf_open_bdev(struct blkg_conf_ctx *ctx) ctx->bdev = bdev; return 0; } -/* - * Similar to blkg_conf_open_bdev, but additionally freezes the queue, - * ensures the correct locking order between freeze queue and q->rq_qos_mutex. - * - * This function returns negative error on failure. On success it returns - * memflags which must be saved and later passed to blkg_conf_exit_frozen - * for restoring the memalloc scope. - */ -unsigned long __must_check blkg_conf_open_bdev_frozen(struct blkg_conf_ctx *ctx) -{ - int ret; - unsigned long memflags; - - if (ctx->bdev) - return -EINVAL; - - ret = blkg_conf_open_bdev(ctx); - if (ret < 0) - return ret; - /* - * At this point, we haven’t started protecting anything related to QoS, - * so we release q->rq_qos_mutex here, which was first acquired in blkg_ - * conf_open_bdev. Later, we re-acquire q->rq_qos_mutex after freezing - * the queue to maintain the correct locking order. - */ - mutex_unlock(&ctx->bdev->bd_queue->rq_qos_mutex); - - memflags = blk_mq_freeze_queue(ctx->bdev->bd_queue); - mutex_lock(&ctx->bdev->bd_queue->rq_qos_mutex); - - return memflags; -} +EXPORT_SYMBOL_GPL(blkg_conf_open_bdev); /** * blkg_conf_prep - parse and prepare for per-blkg config update @@ -857,22 +823,20 @@ unsigned long __must_check blkg_conf_open_bdev_frozen(struct blkg_conf_ctx *ctx) * following MAJ:MIN, @ctx->bdev points to the target block device and * @ctx->blkg to the blkg being configured. * - * blkg_conf_open_bdev() may be called on @ctx beforehand. On success, this + * blkg_conf_open_bdev() must be called on @ctx beforehand. On success, this * function returns with queue lock held and must be followed by - * blkg_conf_exit(). + * blkg_conf_close_bdev(). */ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol, struct blkg_conf_ctx *ctx) - __acquires(&bdev->bd_queue->queue_lock) { struct gendisk *disk; struct request_queue *q; struct blkcg_gq *blkg; int ret; - ret = blkg_conf_open_bdev(ctx); - if (ret) - return ret; + if (WARN_ON_ONCE(!ctx->bdev)) + return -EINVAL; disk = ctx->bdev->bd_disk; q = disk->queue; @@ -970,43 +934,29 @@ fail_exit: EXPORT_SYMBOL_GPL(blkg_conf_prep); /** - * blkg_conf_exit - clean up per-blkg config update + * blkg_conf_unprep - counterpart of blkg_conf_prep() * @ctx: blkg_conf_ctx initialized with blkg_conf_init() - * - * Clean up after per-blkg config update. This function must be called on all - * blkg_conf_ctx's initialized with blkg_conf_init(). */ -void blkg_conf_exit(struct blkg_conf_ctx *ctx) - __releases(&ctx->bdev->bd_queue->queue_lock) - __releases(&ctx->bdev->bd_queue->rq_qos_mutex) +void blkg_conf_unprep(struct blkg_conf_ctx *ctx) { - if (ctx->blkg) { - spin_unlock_irq(&bdev_get_queue(ctx->bdev)->queue_lock); - ctx->blkg = NULL; - } - - if (ctx->bdev) { - mutex_unlock(&ctx->bdev->bd_queue->rq_qos_mutex); - blkdev_put_no_open(ctx->bdev); - ctx->body = NULL; - ctx->bdev = NULL; - } + WARN_ON_ONCE(!ctx->blkg); + spin_unlock_irq(&ctx->bdev->bd_disk->queue->queue_lock); + ctx->blkg = NULL; } -EXPORT_SYMBOL_GPL(blkg_conf_exit); +EXPORT_SYMBOL_GPL(blkg_conf_unprep); -/* - * Similar to blkg_conf_exit, but also unfreezes the queue. Should be used - * when blkg_conf_open_bdev_frozen is used to open the bdev. +/** + * blkg_conf_close_bdev - counterpart of blkg_conf_open_bdev() + * @ctx: blkg_conf_ctx initialized with blkg_conf_init() */ -void blkg_conf_exit_frozen(struct blkg_conf_ctx *ctx, unsigned long memflags) +void blkg_conf_close_bdev(struct blkg_conf_ctx *ctx) { - if (ctx->bdev) { - struct request_queue *q = ctx->bdev->bd_queue; - - blkg_conf_exit(ctx); - blk_mq_unfreeze_queue(q, memflags); - } + mutex_unlock(&ctx->bdev->bd_queue->rq_qos_mutex); + blkdev_put_no_open(ctx->bdev); + ctx->body = NULL; + ctx->bdev = NULL; } +EXPORT_SYMBOL_GPL(blkg_conf_close_bdev); static void blkg_iostat_add(struct blkg_iostat *dst, struct blkg_iostat *src) { diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h index 1cce3294634d..f25fecb87c43 100644 --- a/block/blk-cgroup.h +++ b/block/blk-cgroup.h @@ -218,12 +218,15 @@ struct blkg_conf_ctx { }; void blkg_conf_init(struct blkg_conf_ctx *ctx, char *input); -int blkg_conf_open_bdev(struct blkg_conf_ctx *ctx); -unsigned long blkg_conf_open_bdev_frozen(struct blkg_conf_ctx *ctx); +int blkg_conf_open_bdev(struct blkg_conf_ctx *ctx) + __cond_acquires(0, &ctx->bdev->bd_queue->rq_qos_mutex); int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol, - struct blkg_conf_ctx *ctx); -void blkg_conf_exit(struct blkg_conf_ctx *ctx); -void blkg_conf_exit_frozen(struct blkg_conf_ctx *ctx, unsigned long memflags); + struct blkg_conf_ctx *ctx) + __cond_acquires(0, &ctx->bdev->bd_disk->queue->queue_lock); +void blkg_conf_unprep(struct blkg_conf_ctx *ctx) + __releases(ctx->bdev->bd_disk->queue->queue_lock); +void blkg_conf_close_bdev(struct blkg_conf_ctx *ctx) + __releases(&ctx->bdev->bd_queue->rq_qos_mutex); /** * bio_issue_as_root_blkg - see if this bio needs to be issued as root blkg diff --git a/block/blk-core.c b/block/blk-core.c index 17450058ea6d..73a41df98c9a 100644 --- a/block/blk-core.c +++ b/block/blk-core.c @@ -50,6 +50,7 @@ #include "blk-cgroup.h" #include "blk-throttle.h" #include "blk-ioprio.h" +#include "error-injection.h" struct dentry *blk_debugfs_root; @@ -132,39 +133,56 @@ inline const char *blk_op_str(enum req_op op) } EXPORT_SYMBOL_GPL(blk_op_str); +enum req_op str_to_blk_op(const char *op) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(blk_op_name); i++) + if (blk_op_name[i] && !strcmp(blk_op_name[i], op)) + return (enum req_op)i; + return REQ_OP_LAST; +} + +#define ENT(_tag, _errno, _desc) \ +[BLK_STS_##_tag] = { \ + .errno = _errno, \ + .tag = __stringify(_tag), \ + .name = _desc, \ +} static const struct { int errno; + const char *tag; const char *name; } blk_errors[] = { - [BLK_STS_OK] = { 0, "" }, - [BLK_STS_NOTSUPP] = { -EOPNOTSUPP, "operation not supported" }, - [BLK_STS_TIMEOUT] = { -ETIMEDOUT, "timeout" }, - [BLK_STS_NOSPC] = { -ENOSPC, "critical space allocation" }, - [BLK_STS_TRANSPORT] = { -ENOLINK, "recoverable transport" }, - [BLK_STS_TARGET] = { -EREMOTEIO, "critical target" }, - [BLK_STS_RESV_CONFLICT] = { -EBADE, "reservation conflict" }, - [BLK_STS_MEDIUM] = { -ENODATA, "critical medium" }, - [BLK_STS_PROTECTION] = { -EILSEQ, "protection" }, - [BLK_STS_RESOURCE] = { -ENOMEM, "kernel resource" }, - [BLK_STS_DEV_RESOURCE] = { -EBUSY, "device resource" }, - [BLK_STS_AGAIN] = { -EAGAIN, "nonblocking retry" }, - [BLK_STS_OFFLINE] = { -ENODEV, "device offline" }, + ENT(OK, 0, ""), + ENT(NOTSUPP, -EOPNOTSUPP, "operation not supported"), + ENT(TIMEOUT, -ETIMEDOUT, "timeout"), + ENT(NOSPC, -ENOSPC, "critical space allocation"), + ENT(TRANSPORT, -ENOLINK, "recoverable transport"), + ENT(TARGET, -EREMOTEIO, "critical target"), + ENT(RESV_CONFLICT, -EBADE, "reservation conflict"), + ENT(MEDIUM, -ENODATA, "critical medium"), + ENT(PROTECTION, -EILSEQ, "protection"), + ENT(RESOURCE, -ENOMEM, "kernel resource"), + ENT(DEV_RESOURCE, -EBUSY, "device resource"), + ENT(AGAIN, -EAGAIN, "nonblocking retry"), + ENT(OFFLINE, -ENODEV, "device offline"), /* device mapper special case, should not leak out: */ - [BLK_STS_DM_REQUEUE] = { -EREMCHG, "dm internal retry" }, + ENT(DM_REQUEUE, -EREMCHG, "dm internal retry"), /* zone device specific errors */ - [BLK_STS_ZONE_OPEN_RESOURCE] = { -ETOOMANYREFS, "open zones exceeded" }, - [BLK_STS_ZONE_ACTIVE_RESOURCE] = { -EOVERFLOW, "active zones exceeded" }, + ENT(ZONE_OPEN_RESOURCE, -ETOOMANYREFS, "open zones exceeded"), + ENT(ZONE_ACTIVE_RESOURCE, -EOVERFLOW, "active zones exceeded"), /* Command duration limit device-side timeout */ - [BLK_STS_DURATION_LIMIT] = { -ETIME, "duration limit exceeded" }, - - [BLK_STS_INVAL] = { -EINVAL, "invalid" }, + ENT(DURATION_LIMIT, -ETIME, "duration limit exceeded"), + ENT(INVAL, -EINVAL, "invalid"), /* everything else not covered above: */ - [BLK_STS_IOERR] = { -EIO, "I/O" }, + ENT(IOERR, -EIO, "I/O"), }; +#undef ENT blk_status_t errno_to_blk_status(int errno) { @@ -197,7 +215,32 @@ const char *blk_status_to_str(blk_status_t status) return "<null>"; return blk_errors[idx].name; } -EXPORT_SYMBOL_GPL(blk_status_to_str); + +const char *blk_status_to_tag(blk_status_t status) +{ + int idx = (__force int)status; + + if (WARN_ON_ONCE(idx >= ARRAY_SIZE(blk_errors) || !blk_errors[idx].tag)) + return "<null>"; + return blk_errors[idx].tag; +} + +blk_status_t tag_to_blk_status(const char *tag) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(blk_errors); i++) { + if (blk_errors[i].tag && + !strcmp(blk_errors[i].tag, tag)) + return (__force blk_status_t)i; + } + + /* + * Return BLK_STS_OK for mismatches as this function is intended to + * parse error status values. + */ + return BLK_STS_OK; +} /** * blk_sync_queue - cancel any pending callbacks on a queue @@ -637,12 +680,10 @@ static void __submit_bio(struct bio *bio) struct gendisk *disk = bio->bi_bdev->bd_disk; if ((bio->bi_opf & REQ_POLLED) && - !(disk->queue->limits.features & BLK_FEAT_POLL)) { - bio->bi_status = BLK_STS_NOTSUPP; - bio_endio(bio); - } else { + !(disk->queue->limits.features & BLK_FEAT_POLL)) + bio_endio_status(bio, BLK_STS_NOTSUPP); + else disk->fops->submit_bio(bio); - } blk_queue_exit(disk->queue); } @@ -727,6 +768,9 @@ static void __submit_bio_noacct_mq(struct bio *bio) void submit_bio_noacct_nocheck(struct bio *bio, bool split) { + if (unlikely(blk_error_inject(bio))) + return; + blk_cgroup_bio_start(bio); if (!bio_flagged(bio, BIO_TRACE_COMPLETION)) { @@ -887,8 +931,7 @@ void submit_bio_noacct(struct bio *bio) not_supported: status = BLK_STS_NOTSUPP; end_io: - bio->bi_status = status; - bio_endio(bio); + bio_endio_status(bio, status); } EXPORT_SYMBOL(submit_bio_noacct); @@ -1042,7 +1085,7 @@ unsigned long bdev_start_io_acct(struct block_device *bdev, enum req_op op, { part_stat_lock(); update_io_ticks(bdev, start_time, false); - part_stat_local_inc(bdev, in_flight[op_is_write(op)]); + bdev_inc_in_flight(bdev, op); part_stat_unlock(); return start_time; @@ -1073,7 +1116,7 @@ void bdev_end_io_acct(struct block_device *bdev, enum req_op op, part_stat_inc(bdev, ios[sgrp]); part_stat_add(bdev, sectors[sgrp], sectors); part_stat_add(bdev, nsecs[sgrp], jiffies_to_nsecs(duration)); - part_stat_local_dec(bdev, in_flight[op_is_write(op)]); + bdev_dec_in_flight(bdev, op); part_stat_unlock(); } EXPORT_SYMBOL(bdev_end_io_acct); @@ -1270,7 +1313,6 @@ void blk_io_schedule(void) else io_schedule(); } -EXPORT_SYMBOL_GPL(blk_io_schedule); int __init blk_dev_init(void) { diff --git a/block/blk-crypto-fallback.c b/block/blk-crypto-fallback.c index ab6924fba280..2a5c52ab74b4 100644 --- a/block/blk-crypto-fallback.c +++ b/block/blk-crypto-fallback.c @@ -362,8 +362,7 @@ static void blk_crypto_fallback_encrypt_bio(struct bio *src_bio) status = blk_crypto_get_keyslot(blk_crypto_fallback_profile, bc->bc_key, &slot); if (status != BLK_STS_OK) { - src_bio->bi_status = status; - bio_endio(src_bio); + bio_endio_status(src_bio, status); return; } __blk_crypto_fallback_encrypt_bio(src_bio, @@ -438,8 +437,7 @@ static void blk_crypto_fallback_decrypt_bio(struct work_struct *work) } mempool_free(f_ctx, bio_fallback_crypt_ctx_pool); - bio->bi_status = status; - bio_endio(bio); + bio_endio_status(bio, status); } /** @@ -500,8 +498,7 @@ bool blk_crypto_fallback_bio_prep(struct bio *bio) if (!__blk_crypto_cfg_supported(blk_crypto_fallback_profile, &bc->bc_key->crypto_cfg)) { - bio->bi_status = BLK_STS_NOTSUPP; - bio_endio(bio); + bio_endio_status(bio, BLK_STS_NOTSUPP); return false; } diff --git a/block/blk-crypto-profile.c b/block/blk-crypto-profile.c index 4ac74443687a..cf447ba4a66e 100644 --- a/block/blk-crypto-profile.c +++ b/block/blk-crypto-profile.c @@ -43,6 +43,7 @@ struct blk_crypto_keyslot { }; static inline void blk_crypto_hw_enter(struct blk_crypto_profile *profile) + __acquires(&profile->lock) { /* * Calling into the driver requires profile->lock held and the device @@ -55,6 +56,7 @@ static inline void blk_crypto_hw_enter(struct blk_crypto_profile *profile) } static inline void blk_crypto_hw_exit(struct blk_crypto_profile *profile) + __releases(&profile->lock) { up_write(&profile->lock); if (profile->dev) diff --git a/block/blk-crypto.c b/block/blk-crypto.c index 856d3c5b1fa0..165c9d2cce07 100644 --- a/block/blk-crypto.c +++ b/block/blk-crypto.c @@ -267,8 +267,7 @@ bool __blk_crypto_submit_bio(struct bio *bio) if (!IS_ENABLED(CONFIG_BLK_INLINE_ENCRYPTION_FALLBACK)) { pr_warn_once("%pg: crypto API fallback disabled; failing request.\n", bdev); - bio->bi_status = BLK_STS_NOTSUPP; - bio_endio(bio); + bio_endio_status(bio, BLK_STS_NOTSUPP); return false; } return blk_crypto_fallback_bio_prep(bio); diff --git a/block/blk-iocost.c b/block/blk-iocost.c index 0cca88a366dc..563cc7dcf348 100644 --- a/block/blk-iocost.c +++ b/block/blk-iocost.c @@ -727,26 +727,6 @@ static void iocg_commit_bio(struct ioc_gq *iocg, struct bio *bio, put_cpu_ptr(gcs); } -static void iocg_lock(struct ioc_gq *iocg, bool lock_ioc, unsigned long *flags) -{ - if (lock_ioc) { - spin_lock_irqsave(&iocg->ioc->lock, *flags); - spin_lock(&iocg->waitq.lock); - } else { - spin_lock_irqsave(&iocg->waitq.lock, *flags); - } -} - -static void iocg_unlock(struct ioc_gq *iocg, bool unlock_ioc, unsigned long *flags) -{ - if (unlock_ioc) { - spin_unlock(&iocg->waitq.lock); - spin_unlock_irqrestore(&iocg->ioc->lock, *flags); - } else { - spin_unlock_irqrestore(&iocg->waitq.lock, *flags); - } -} - #define CREATE_TRACE_POINTS #include <trace/events/iocost.h> @@ -1589,9 +1569,17 @@ static enum hrtimer_restart iocg_waitq_timer_fn(struct hrtimer *timer) ioc_now(iocg->ioc, &now); - iocg_lock(iocg, pay_debt, &flags); - iocg_kick_waitq(iocg, pay_debt, &now); - iocg_unlock(iocg, pay_debt, &flags); + if (pay_debt) { + spin_lock_irqsave(&iocg->ioc->lock, flags); + spin_lock(&iocg->waitq.lock); + iocg_kick_waitq(iocg, pay_debt, &now); + spin_unlock(&iocg->waitq.lock); + spin_unlock_irqrestore(&iocg->ioc->lock, flags); + } else { + spin_lock_irqsave(&iocg->waitq.lock, flags); + iocg_kick_waitq(iocg, pay_debt, &now); + spin_unlock_irqrestore(&iocg->waitq.lock, flags); + } return HRTIMER_NORESTART; } @@ -2614,6 +2602,88 @@ static u64 calc_size_vtime_cost(struct request *rq, struct ioc *ioc) return cost; } +enum over_budget_action { + action_retry, + action_commit, + action_wait, + action_return, +}; + +static enum over_budget_action +iocg_handle_over_budget(struct rq_qos *rqos, struct ioc_gq *iocg, + struct bio *bio, struct ioc_now *now, + struct iocg_wait *wait, bool use_debt, bool ioc_locked, + u64 abs_cost, u64 cost) +{ + lockdep_assert_held(&iocg->waitq.lock); + + /* + * @iocg must stay activated for debt and waitq handling. Deactivation + * is synchronized against both ioc->lock and waitq.lock and we won't + * get deactivated as long as we're waiting or have debt, so we're good + * if we're activated here. In the unlikely cases that we aren't, just + * issue the IO. + */ + if (unlikely(list_empty(&iocg->active_list))) + return action_commit; + + /* + * We're over budget. If @bio has to be issued regardless, remember + * the abs_cost instead of advancing vtime. iocg_kick_waitq() will pay + * off the debt before waking more IOs. + * + * This way, the debt is continuously paid off each period with the + * actual budget available to the cgroup. If we just wound vtime, we + * would incorrectly use the current hw_inuse for the entire amount + * which, for example, can lead to the cgroup staying blocked for a + * long time even with substantially raised hw_inuse. + * + * An iocg with vdebt should stay online so that the timer can keep + * deducting its vdebt and [de]activate use_delay mechanism + * accordingly. We don't want to race against the timer trying to + * clear them and leave @iocg inactive w/ dangling use_delay heavily + * penalizing the cgroup and its descendants. + */ + if (use_debt) { + iocg_incur_debt(iocg, abs_cost, now); + if (iocg_kick_delay(iocg, now)) + blkcg_schedule_throttle(rqos->disk, + (bio->bi_opf & REQ_SWAP) == + REQ_SWAP); + return action_return; + } + + /* guarantee that iocgs w/ waiters have maximum inuse */ + if (!iocg->abs_vdebt && iocg->inuse != iocg->active) { + if (!ioc_locked) + return action_retry; + lockdep_assert_held(&iocg->ioc->lock); + propagate_weights(iocg, iocg->active, iocg->active, true, now); + } + + /* + * Append self to the waitq and schedule the wakeup timer if we're + * the first waiter. The timer duration is calculated based on the + * current vrate. vtime and hweight changes can make it too short + * or too long. Each wait entry records the absolute cost it's + * waiting for to allow re-evaluation using a custom wait entry. + * + * If too short, the timer simply reschedules itself. If too long, + * the period timer will notice and trigger wakeups. + * + * All waiters are on iocg->waitq and the wait states are + * synchronized using waitq.lock. + */ + init_wait_func(&wait->wait, iocg_wake_fn); + wait->bio = bio; + wait->abs_cost = abs_cost; + wait->committed = false; /* will be set true by waker */ + + __add_wait_queue_entry_tail(&iocg->waitq, &wait->wait); + iocg_kick_waitq(iocg, ioc_locked, now); + return action_wait; +} + static void ioc_rqos_throttle(struct rq_qos *rqos, struct bio *bio) { struct blkcg_gq *blkg = bio->bi_blkg; @@ -2623,6 +2693,7 @@ static void ioc_rqos_throttle(struct rq_qos *rqos, struct bio *bio) struct iocg_wait wait; u64 abs_cost, cost, vtime; bool use_debt, ioc_locked; + enum over_budget_action action; unsigned long flags; /* bypass IOs if disabled, still initializing, or for root cgroup */ @@ -2662,81 +2733,34 @@ static void ioc_rqos_throttle(struct rq_qos *rqos, struct bio *bio) use_debt = bio_issue_as_root_blkg(bio) || fatal_signal_pending(current); ioc_locked = use_debt || READ_ONCE(iocg->abs_vdebt); retry_lock: - iocg_lock(iocg, ioc_locked, &flags); - - /* - * @iocg must stay activated for debt and waitq handling. Deactivation - * is synchronized against both ioc->lock and waitq.lock and we won't - * get deactivated as long as we're waiting or has debt, so we're good - * if we're activated here. In the unlikely cases that we aren't, just - * issue the IO. - */ - if (unlikely(list_empty(&iocg->active_list))) { - iocg_unlock(iocg, ioc_locked, &flags); + if (ioc_locked) { + spin_lock_irqsave(&iocg->ioc->lock, flags); + spin_lock(&iocg->waitq.lock); + action = iocg_handle_over_budget(rqos, iocg, bio, &now, &wait, + use_debt, ioc_locked, abs_cost, + cost); + spin_unlock(&iocg->waitq.lock); + spin_unlock_irqrestore(&iocg->ioc->lock, flags); + } else { + spin_lock_irqsave(&iocg->waitq.lock, flags); + action = iocg_handle_over_budget(rqos, iocg, bio, &now, &wait, + use_debt, ioc_locked, abs_cost, + cost); + spin_unlock_irqrestore(&iocg->waitq.lock, flags); + } + switch (action) { + case action_retry: + ioc_locked = true; + goto retry_lock; + case action_commit: iocg_commit_bio(iocg, bio, abs_cost, cost); return; - } - - /* - * We're over budget. If @bio has to be issued regardless, remember - * the abs_cost instead of advancing vtime. iocg_kick_waitq() will pay - * off the debt before waking more IOs. - * - * This way, the debt is continuously paid off each period with the - * actual budget available to the cgroup. If we just wound vtime, we - * would incorrectly use the current hw_inuse for the entire amount - * which, for example, can lead to the cgroup staying blocked for a - * long time even with substantially raised hw_inuse. - * - * An iocg with vdebt should stay online so that the timer can keep - * deducting its vdebt and [de]activate use_delay mechanism - * accordingly. We don't want to race against the timer trying to - * clear them and leave @iocg inactive w/ dangling use_delay heavily - * penalizing the cgroup and its descendants. - */ - if (use_debt) { - iocg_incur_debt(iocg, abs_cost, &now); - if (iocg_kick_delay(iocg, &now)) - blkcg_schedule_throttle(rqos->disk, - (bio->bi_opf & REQ_SWAP) == REQ_SWAP); - iocg_unlock(iocg, ioc_locked, &flags); + case action_return: return; + case action_wait: + break; } - /* guarantee that iocgs w/ waiters have maximum inuse */ - if (!iocg->abs_vdebt && iocg->inuse != iocg->active) { - if (!ioc_locked) { - iocg_unlock(iocg, false, &flags); - ioc_locked = true; - goto retry_lock; - } - propagate_weights(iocg, iocg->active, iocg->active, true, - &now); - } - - /* - * Append self to the waitq and schedule the wakeup timer if we're - * the first waiter. The timer duration is calculated based on the - * current vrate. vtime and hweight changes can make it too short - * or too long. Each wait entry records the absolute cost it's - * waiting for to allow re-evaluation using a custom wait entry. - * - * If too short, the timer simply reschedules itself. If too long, - * the period timer will notice and trigger wakeups. - * - * All waiters are on iocg->waitq and the wait states are - * synchronized using waitq.lock. - */ - init_wait_func(&wait.wait, iocg_wake_fn); - wait.bio = bio; - wait.abs_cost = abs_cost; - wait.committed = false; /* will be set true by waker */ - - __add_wait_queue_entry_tail(&iocg->waitq, &wait.wait); - iocg_kick_waitq(iocg, ioc_locked, &now); - - iocg_unlock(iocg, ioc_locked, &flags); - while (true) { set_current_state(TASK_UNINTERRUPTIBLE); if (wait.committed) @@ -3140,19 +3164,25 @@ static ssize_t ioc_weight_write(struct kernfs_open_file *of, char *buf, blkg_conf_init(&ctx, buf); + ret = blkg_conf_open_bdev(&ctx); + if (ret) + return ret; + ret = blkg_conf_prep(blkcg, &blkcg_policy_iocost, &ctx); if (ret) - goto err; + goto close_bdev; iocg = blkg_to_iocg(ctx.blkg); + ret = -EINVAL; + if (!strncmp(ctx.body, "default", 7)) { v = 0; } else { if (!sscanf(ctx.body, "%u", &v)) - goto einval; + goto unprep; if (v < CGROUP_WEIGHT_MIN || v > CGROUP_WEIGHT_MAX) - goto einval; + goto unprep; } spin_lock(&iocg->ioc->lock); @@ -3161,14 +3191,15 @@ static ssize_t ioc_weight_write(struct kernfs_open_file *of, char *buf, weight_updated(iocg, &now); spin_unlock(&iocg->ioc->lock); - blkg_conf_exit(&ctx); - return nbytes; + ret = 0; -einval: - ret = -EINVAL; -err: - blkg_conf_exit(&ctx); - return ret; +unprep: + blkg_conf_unprep(&ctx); + +close_bdev: + blkg_conf_close_bdev(&ctx); + + return ret ?: nbytes; } static u64 ioc_qos_prfill(struct seq_file *sf, struct blkg_policy_data *pd, @@ -3226,34 +3257,43 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input, size_t nbytes, loff_t off) { struct blkg_conf_ctx ctx; + struct request_queue *q; struct gendisk *disk; struct ioc *ioc; u32 qos[NR_QOS_PARAMS]; bool enable, user; char *body, *p; - unsigned long memflags; + unsigned int memflags; int ret; blkg_conf_init(&ctx, input); - memflags = blkg_conf_open_bdev_frozen(&ctx); - if (IS_ERR_VALUE(memflags)) { - ret = memflags; - goto err; - } + ret = blkg_conf_open_bdev(&ctx); + if (ret) + return ret; + /* + * At this point, we haven’t started protecting anything related to QoS, + * so we release q->rq_qos_mutex here, which was first acquired in blkg_ + * conf_open_bdev. Later, we re-acquire q->rq_qos_mutex after freezing + * the queue to maintain the correct locking order. + */ + mutex_unlock(&ctx.bdev->bd_queue->rq_qos_mutex); + + memflags = blk_mq_freeze_queue(ctx.bdev->bd_queue); + mutex_lock(&ctx.bdev->bd_queue->rq_qos_mutex); body = ctx.body; disk = ctx.bdev->bd_disk; if (!queue_is_mq(disk->queue)) { ret = -EOPNOTSUPP; - goto err; + goto close_bdev; } ioc = q_to_ioc(disk->queue); if (!ioc) { ret = blk_iocost_init(disk); if (ret) - goto err; + goto close_bdev; ioc = q_to_ioc(disk->queue); } @@ -3357,15 +3397,17 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input, blk_mq_unquiesce_queue(disk->queue); - blkg_conf_exit_frozen(&ctx, memflags); - return nbytes; +close_bdev: + q = ctx.bdev->bd_queue; + blkg_conf_close_bdev(&ctx); + blk_mq_unfreeze_queue(q, memflags); + return ret ?: nbytes; + einval: spin_unlock_irq(&ioc->lock); blk_mq_unquiesce_queue(disk->queue); ret = -EINVAL; -err: - blkg_conf_exit_frozen(&ctx, memflags); - return ret; + goto close_bdev; } static u64 ioc_cost_model_prfill(struct seq_file *sf, @@ -3430,20 +3472,20 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input, ret = blkg_conf_open_bdev(&ctx); if (ret) - goto err; + return ret; body = ctx.body; q = bdev_get_queue(ctx.bdev); if (!queue_is_mq(q)) { ret = -EOPNOTSUPP; - goto err; + goto close_bdev; } ioc = q_to_ioc(q); if (!ioc) { ret = blk_iocost_init(ctx.bdev->bd_disk); if (ret) - goto err; + goto close_bdev; ioc = q_to_ioc(q); } @@ -3454,6 +3496,8 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input, memcpy(u, ioc->params.i_lcoefs, sizeof(u)); user = ioc->user_cost_model; + ret = -EINVAL; + while ((p = strsep(&body, " \t\n"))) { substring_t args[MAX_OPT_ARGS]; char buf[32]; @@ -3471,20 +3515,20 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input, else if (!strcmp(buf, "user")) user = true; else - goto einval; + goto unlock; continue; case COST_MODEL: match_strlcpy(buf, &args[0], sizeof(buf)); if (strcmp(buf, "linear")) - goto einval; + goto unlock; continue; } tok = match_token(p, i_lcoef_tokens, args); if (tok == NR_I_LCOEFS) - goto einval; + goto unlock; if (match_u64(&args[0], &v)) - goto einval; + goto unlock; u[tok] = v; user = true; } @@ -3496,24 +3540,18 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input, ioc->user_cost_model = false; } ioc_refresh_params(ioc, true); - spin_unlock_irq(&ioc->lock); - - blk_mq_unquiesce_queue(q); - blk_mq_unfreeze_queue(q, memflags); - blkg_conf_exit(&ctx); - return nbytes; + ret = 0; -einval: +unlock: spin_unlock_irq(&ioc->lock); blk_mq_unquiesce_queue(q); blk_mq_unfreeze_queue(q, memflags); - ret = -EINVAL; -err: - blkg_conf_exit(&ctx); - return ret; +close_bdev: + blkg_conf_close_bdev(&ctx); + return ret ?: nbytes; } static struct cftype ioc_files[] = { diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c index 53e8dd2dfa8a..1aaee6fb0f59 100644 --- a/block/blk-iolatency.c +++ b/block/blk-iolatency.c @@ -840,7 +840,7 @@ static ssize_t iolatency_set_limit(struct kernfs_open_file *of, char *buf, ret = blkg_conf_open_bdev(&ctx); if (ret) - goto out; + return ret; /* * blk_iolatency_init() may fail after rq_qos_add() succeeds which can @@ -850,11 +850,11 @@ static ssize_t iolatency_set_limit(struct kernfs_open_file *of, char *buf, if (!iolat_rq_qos(ctx.bdev->bd_queue)) ret = blk_iolatency_init(ctx.bdev->bd_disk); if (ret) - goto out; + goto close_bdev; ret = blkg_conf_prep(blkcg, &blkcg_policy_iolatency, &ctx); if (ret) - goto out; + goto close_bdev; iolat = blkg_to_lat(ctx.blkg); p = ctx.body; @@ -865,7 +865,7 @@ static ssize_t iolatency_set_limit(struct kernfs_open_file *of, char *buf, char val[21]; /* 18446744073709551616 */ if (sscanf(tok, "%15[^=]=%20s", key, val) != 2) - goto out; + goto unprep; if (!strcmp(key, "target")) { u64 v; @@ -875,9 +875,9 @@ static ssize_t iolatency_set_limit(struct kernfs_open_file *of, char *buf, else if (sscanf(val, "%llu", &v) == 1) lat_val = v * NSEC_PER_USEC; else - goto out; + goto unprep; } else { - goto out; + goto unprep; } } @@ -889,8 +889,11 @@ static ssize_t iolatency_set_limit(struct kernfs_open_file *of, char *buf, if (oldval != iolat->min_lat_nsec) iolatency_clear_scaling(blkg); ret = 0; -out: - blkg_conf_exit(&ctx); + +unprep: + blkg_conf_unprep(&ctx); +close_bdev: + blkg_conf_close_bdev(&ctx); return ret ?: nbytes; } diff --git a/block/blk-merge.c b/block/blk-merge.c index fcf09325b22e..ab1161ca69f1 100644 --- a/block/blk-merge.c +++ b/block/blk-merge.c @@ -122,8 +122,7 @@ struct bio *bio_submit_split_bioset(struct bio *bio, unsigned int split_sectors, struct bio *split = bio_split(bio, split_sectors, GFP_NOIO, bs); if (IS_ERR(split)) { - bio->bi_status = errno_to_blk_status(PTR_ERR(split)); - bio_endio(bio); + bio_endio_status(bio, errno_to_blk_status(PTR_ERR(split))); return NULL; } @@ -143,8 +142,7 @@ EXPORT_SYMBOL_GPL(bio_submit_split_bioset); static struct bio *bio_submit_split(struct bio *bio, int split_sectors) { if (unlikely(split_sectors < 0)) { - bio->bi_status = errno_to_blk_status(split_sectors); - bio_endio(bio); + bio_endio_status(bio, errno_to_blk_status(split_sectors)); return NULL; } @@ -547,7 +545,7 @@ static inline int ll_new_hw_segment(struct request *req, struct bio *bio, if (!blk_cgroup_mergeable(req, bio)) goto no_merge; - if (blk_integrity_merge_bio(req->q, req, bio) == false) + if (unlikely(!blk_integrity_merge_bio(req->q, req, bio))) goto no_merge; /* discard request merge won't add new segment */ @@ -649,7 +647,7 @@ static int ll_merge_requests_fn(struct request_queue *q, struct request *req, if (!blk_cgroup_mergeable(req, next->bio)) return 0; - if (blk_integrity_merge_rq(q, req, next) == false) + if (unlikely(!blk_integrity_merge_rq(q, req, next))) return 0; if (!bio_crypt_ctx_merge_rq(req, next)) @@ -723,8 +721,7 @@ static void blk_account_io_merge_request(struct request *req) if (req->rq_flags & RQF_IO_STAT) { part_stat_lock(); part_stat_inc(req->part, merges[op_stat_group(req_op(req))]); - part_stat_local_dec(req->part, - in_flight[op_is_write(req_op(req))]); + bdev_dec_in_flight(req->part, req_op(req)); part_stat_unlock(); } } @@ -905,7 +902,7 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio) if (!blk_cgroup_mergeable(rq, bio)) return false; - if (blk_integrity_merge_bio(rq->q, rq, bio) == false) + if (unlikely(!blk_integrity_merge_bio(rq->q, rq, bio))) return false; if (!bio_crypt_rq_ctx_compatible(rq, bio)) return false; @@ -915,7 +912,7 @@ bool blk_rq_merge_ok(struct request *rq, struct bio *bio) return false; if (rq->bio->bi_ioprio != bio->bi_ioprio) return false; - if (blk_atomic_write_mergeable_rq_bio(rq, bio) == false) + if (unlikely(!blk_atomic_write_mergeable_rq_bio(rq, bio))) return false; return true; diff --git a/block/blk-mq-debugfs.c b/block/blk-mq-debugfs.c index 047ec887456b..6754d8f9449c 100644 --- a/block/blk-mq-debugfs.c +++ b/block/blk-mq-debugfs.c @@ -19,8 +19,10 @@ static int queue_poll_stat_show(void *data, struct seq_file *m) return 0; } +#define TO_REQUEST_QUEUE(m) ((struct request_queue *)m->private) + static void *queue_requeue_list_start(struct seq_file *m, loff_t *pos) - __acquires(&q->requeue_lock) + __acquires(&TO_REQUEST_QUEUE(m)->requeue_lock) { struct request_queue *q = m->private; @@ -36,13 +38,15 @@ static void *queue_requeue_list_next(struct seq_file *m, void *v, loff_t *pos) } static void queue_requeue_list_stop(struct seq_file *m, void *v) - __releases(&q->requeue_lock) + __releases(&TO_REQUEST_QUEUE(m)->requeue_lock) { struct request_queue *q = m->private; spin_unlock_irq(&q->requeue_lock); } +#undef TO_REQUEST_QUEUE + static const struct seq_operations queue_requeue_list_seq_ops = { .start = queue_requeue_list_start, .next = queue_requeue_list_next, @@ -297,8 +301,10 @@ int blk_mq_debugfs_rq_show(struct seq_file *m, void *v) } EXPORT_SYMBOL_GPL(blk_mq_debugfs_rq_show); +#define TO_HCTX(m) ((struct blk_mq_hw_ctx *)m->private) + static void *hctx_dispatch_start(struct seq_file *m, loff_t *pos) - __acquires(&hctx->lock) + __acquires(&TO_HCTX(m)->lock) { struct blk_mq_hw_ctx *hctx = m->private; @@ -314,13 +320,15 @@ static void *hctx_dispatch_next(struct seq_file *m, void *v, loff_t *pos) } static void hctx_dispatch_stop(struct seq_file *m, void *v) - __releases(&hctx->lock) + __releases(&TO_HCTX(m)->lock) { struct blk_mq_hw_ctx *hctx = m->private; spin_unlock(&hctx->lock); } +#undef TO_HCTX + static const struct seq_operations hctx_dispatch_seq_ops = { .start = hctx_dispatch_start, .next = hctx_dispatch_next, @@ -484,9 +492,11 @@ static int hctx_dispatch_busy_show(void *data, struct seq_file *m) return 0; } +#define TO_CTX(m) ((struct blk_mq_ctx *)m->private) + #define CTX_RQ_SEQ_OPS(name, type) \ static void *ctx_##name##_rq_list_start(struct seq_file *m, loff_t *pos) \ - __acquires(&ctx->lock) \ + __acquires(&TO_CTX(m)->lock) \ { \ struct blk_mq_ctx *ctx = m->private; \ \ @@ -503,7 +513,7 @@ static void *ctx_##name##_rq_list_next(struct seq_file *m, void *v, \ } \ \ static void ctx_##name##_rq_list_stop(struct seq_file *m, void *v) \ - __releases(&ctx->lock) \ + __releases(&TO_CTX(m)->lock) \ { \ struct blk_mq_ctx *ctx = m->private; \ \ @@ -521,6 +531,8 @@ CTX_RQ_SEQ_OPS(default, HCTX_TYPE_DEFAULT); CTX_RQ_SEQ_OPS(read, HCTX_TYPE_READ); CTX_RQ_SEQ_OPS(poll, HCTX_TYPE_POLL); +#undef TO_CTX + static int blk_mq_debugfs_show(struct seq_file *m, void *v) { const struct blk_mq_debugfs_attr *attr = m->private; diff --git a/block/blk-mq-tag.c b/block/blk-mq-tag.c index 33946cdb5716..35deee5bbc73 100644 --- a/block/blk-mq-tag.c +++ b/block/blk-mq-tag.c @@ -13,6 +13,7 @@ #include <linux/kmemleak.h> #include <linux/delay.h> +#include <trace/events/block.h> #include "blk.h" #include "blk-mq.h" #include "blk-mq-sched.h" @@ -181,6 +182,11 @@ unsigned int blk_mq_get_tag(struct blk_mq_alloc_data *data) if (tag != BLK_MQ_NO_TAG) break; + /* Log the starvation event before altering task state */ + trace_block_rq_tag_wait(data->q, data->hctx, + data->rq_flags & RQF_SCHED_TAGS, + data->flags); + sbitmap_prepare_to_wait(bt, ws, &wait, TASK_UNINTERRUPTIBLE); tag = __blk_mq_get_tag(data, bt); diff --git a/block/blk-mq.c b/block/blk-mq.c index a24175441380..88cb5acc4f39 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -1082,49 +1082,18 @@ static inline void blk_account_io_done(struct request *req, u64 now) update_io_ticks(req->part, jiffies, true); part_stat_inc(req->part, ios[sgrp]); part_stat_add(req->part, nsecs[sgrp], now - req->start_time_ns); - part_stat_local_dec(req->part, - in_flight[op_is_write(req_op(req))]); + bdev_dec_in_flight(req->part, req_op(req)); part_stat_unlock(); } } -static inline bool blk_rq_passthrough_stats(struct request *req) -{ - struct bio *bio = req->bio; - - if (!blk_queue_passthrough_stat(req->q)) - return false; - - /* Requests without a bio do not transfer data. */ - if (!bio) - return false; - - /* - * Stats are accumulated in the bdev, so must have one attached to a - * bio to track stats. Most drivers do not set the bdev for passthrough - * requests, but nvme is one that will set it. - */ - if (!bio->bi_bdev) - return false; - - /* - * We don't know what a passthrough command does, but we know the - * payload size and data direction. Ensuring the size is aligned to the - * block size filters out most commands with payloads that don't - * represent sector access. - */ - if (blk_rq_bytes(req) & (bdev_logical_block_size(bio->bi_bdev) - 1)) - return false; - return true; -} - static inline void blk_account_io_start(struct request *req) { trace_block_io_start(req); if (!blk_queue_io_stat(req->q)) return; - if (blk_rq_is_passthrough(req) && !blk_rq_passthrough_stats(req)) + if (blk_rq_is_passthrough(req) && !blk_rq_passthrough_stats(req, req->q)) return; req->rq_flags |= RQF_IO_STAT; @@ -1143,7 +1112,7 @@ static inline void blk_account_io_start(struct request *req) part_stat_lock(); update_io_ticks(req->part, jiffies, false); - part_stat_local_inc(req->part, in_flight[op_is_write(req_op(req))]); + bdev_inc_in_flight(req->part, req_op(req)); part_stat_unlock(); } @@ -3170,8 +3139,7 @@ void blk_mq_submit_bio(struct bio *bio) } if ((bio->bi_opf & REQ_POLLED) && !blk_mq_can_poll(q)) { - bio->bi_status = BLK_STS_NOTSUPP; - bio_endio(bio); + bio_endio_status(bio, BLK_STS_NOTSUPP); goto queue_exit; } @@ -3215,8 +3183,7 @@ new_request: ret = blk_crypto_rq_get_keyslot(rq); if (ret != BLK_STS_OK) { - bio->bi_status = ret; - bio_endio(bio); + bio_endio_status(bio, ret); blk_mq_free_request(rq); return; } diff --git a/block/blk-settings.c b/block/blk-settings.c index 78c83817b9d3..8274631290db 100644 --- a/block/blk-settings.c +++ b/block/blk-settings.c @@ -795,6 +795,8 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b, t->features &= ~BLK_FEAT_NOWAIT; if (!(b->features & BLK_FEAT_POLL)) t->features &= ~BLK_FEAT_POLL; + if (!(b->features & BLK_FEAT_PCI_P2PDMA)) + t->features &= ~BLK_FEAT_PCI_P2PDMA; t->flags |= (b->flags & BLK_FLAG_MISALIGNED); diff --git a/block/blk-sysfs.c b/block/blk-sysfs.c index f22c1f253eb3..520972676ab4 100644 --- a/block/blk-sysfs.c +++ b/block/blk-sysfs.c @@ -19,6 +19,7 @@ #include "blk-wbt.h" #include "blk-cgroup.h" #include "blk-throttle.h" +#include "error-injection.h" struct queue_sysfs_entry { struct attribute attr; @@ -933,6 +934,8 @@ static void blk_debugfs_remove(struct gendisk *disk) blk_debugfs_lock_nomemsave(q); blk_trace_shutdown(q); + if (IS_ENABLED(CONFIG_BLK_ERROR_INJECTION)) + blk_error_injection_exit(disk); debugfs_remove_recursive(q->debugfs_dir); q->debugfs_dir = NULL; q->sched_debugfs_dir = NULL; @@ -963,6 +966,8 @@ int blk_register_queue(struct gendisk *disk) memflags = blk_debugfs_lock(q); q->debugfs_dir = debugfs_create_dir(disk->disk_name, blk_debugfs_root); + if (IS_ENABLED(CONFIG_BLK_ERROR_INJECTION)) + blk_error_injection_init(disk); if (queue_is_mq(q)) blk_mq_debugfs_register(q); blk_debugfs_unlock(q, memflags); diff --git a/block/blk-throttle.c b/block/blk-throttle.c index cabf91f0d0dc..47052ba21d1b 100644 --- a/block/blk-throttle.c +++ b/block/blk-throttle.c @@ -1353,21 +1353,21 @@ static ssize_t tg_set_conf(struct kernfs_open_file *of, ret = blkg_conf_open_bdev(&ctx); if (ret) - goto out_finish; + return ret; if (!blk_throtl_activated(ctx.bdev->bd_queue)) { ret = blk_throtl_init(ctx.bdev->bd_disk); if (ret) - goto out_finish; + goto close_bdev; } ret = blkg_conf_prep(blkcg, &blkcg_policy_throtl, &ctx); if (ret) - goto out_finish; + goto close_bdev; ret = -EINVAL; if (sscanf(ctx.body, "%llu", &v) != 1) - goto out_finish; + goto unprep; if (!v) v = U64_MAX; @@ -1381,8 +1381,12 @@ static ssize_t tg_set_conf(struct kernfs_open_file *of, tg_conf_updated(tg, false); ret = 0; -out_finish: - blkg_conf_exit(&ctx); + +unprep: + blkg_conf_unprep(&ctx); + +close_bdev: + blkg_conf_close_bdev(&ctx); return ret ?: nbytes; } @@ -1537,17 +1541,17 @@ static ssize_t tg_set_limit(struct kernfs_open_file *of, ret = blkg_conf_open_bdev(&ctx); if (ret) - goto out_finish; + return ret; if (!blk_throtl_activated(ctx.bdev->bd_queue)) { ret = blk_throtl_init(ctx.bdev->bd_disk); if (ret) - goto out_finish; + goto close_bdev; } ret = blkg_conf_prep(blkcg, &blkcg_policy_throtl, &ctx); if (ret) - goto out_finish; + goto close_bdev; tg = blkg_to_tg(ctx.blkg); tg_update_carryover(tg); @@ -1573,11 +1577,11 @@ static ssize_t tg_set_limit(struct kernfs_open_file *of, p = tok; strsep(&p, "="); if (!p || (sscanf(p, "%llu", &val) != 1 && strcmp(p, "max"))) - goto out_finish; + goto unprep; ret = -ERANGE; if (!val) - goto out_finish; + goto unprep; ret = -EINVAL; if (!strcmp(tok, "rbps")) @@ -1589,7 +1593,7 @@ static ssize_t tg_set_limit(struct kernfs_open_file *of, else if (!strcmp(tok, "wiops")) v[3] = min_t(u64, val, UINT_MAX); else - goto out_finish; + goto unprep; } tg->bps[READ] = v[0]; @@ -1599,8 +1603,10 @@ static ssize_t tg_set_limit(struct kernfs_open_file *of, tg_conf_updated(tg, false); ret = 0; -out_finish: - blkg_conf_exit(&ctx); +unprep: + blkg_conf_unprep(&ctx); +close_bdev: + blkg_conf_close_bdev(&ctx); return ret ?: nbytes; } @@ -1649,7 +1655,7 @@ static void tg_flush_bios(struct throtl_grp *tg) */ tg_update_disptime(tg); - throtl_schedule_pending_timer(sq, jiffies + 1); + throtl_schedule_next_dispatch(sq->parent_sq, true); } static void throtl_pd_offline(struct blkg_policy_data *pd) @@ -1668,11 +1674,52 @@ struct blkcg_policy blkcg_policy_throtl = { .pd_free_fn = throtl_pd_free, }; +static void tg_cancel_writeback_bios(struct throtl_grp *tg, + struct bio_list *cancel_bios) +{ + struct throtl_service_queue *sq = &tg->service_queue; + struct throtl_data *td = sq_to_td(sq); + int rw; + + if (tg->flags & THROTL_TG_CANCELING) + return; + tg->flags |= THROTL_TG_CANCELING; + + for (rw = READ; rw <= WRITE; rw++) { + struct throtl_qnode *qn, *tmp; + unsigned int nr_bios = 0; + + list_for_each_entry_safe(qn, tmp, &sq->queued[rw], node) { + struct bio *bio; + + while ((bio = bio_list_pop(&qn->bios_iops))) { + sq->nr_queued_iops[rw]--; + bio_list_add(&cancel_bios[rw], bio); + nr_bios++; + } + while ((bio = bio_list_pop(&qn->bios_bps))) { + sq->nr_queued_bps[rw]--; + bio_list_add(&cancel_bios[rw], bio); + nr_bios++; + } + + list_del_init(&qn->node); + blkg_put(tg_to_blkg(qn->tg)); + } + + td->nr_queued[rw] -= nr_bios; + } + + throtl_dequeue_tg(tg); +} + void blk_throtl_cancel_bios(struct gendisk *disk) { struct request_queue *q = disk->queue; struct cgroup_subsys_state *pos_css; struct blkcg_gq *blkg; + struct bio_list cancel_bios[2] = { }; + int rw; if (!blk_throtl_activated(q)) return; @@ -1693,10 +1740,16 @@ void blk_throtl_cancel_bios(struct gendisk *disk) * Cancel bios here to ensure no bios are inflight after * del_gendisk. */ - tg_flush_bios(blkg_to_tg(blkg)); + tg_cancel_writeback_bios(blkg_to_tg(blkg), cancel_bios); } rcu_read_unlock(); spin_unlock_irq(&q->queue_lock); + + for (rw = READ; rw <= WRITE; rw++) { + struct bio *bio; + while ((bio = bio_list_pop(&cancel_bios[rw]))) + bio_io_error(bio); + } } static bool tg_within_limit(struct throtl_grp *tg, struct bio *bio, bool rw) diff --git a/block/blk-zoned.c b/block/blk-zoned.c index 6a221c180889..bea817f3de56 100644 --- a/block/blk-zoned.c +++ b/block/blk-zoned.c @@ -1924,7 +1924,7 @@ static int disk_alloc_zone_resources(struct gendisk *disk, goto free_hash; disk->zone_wplugs_wq = - alloc_workqueue("%s_zwplugs", WQ_MEM_RECLAIM | WQ_HIGHPRI, + alloc_workqueue("%s_zwplugs", WQ_MEM_RECLAIM | WQ_HIGHPRI | WQ_PERCPU, pool_size, disk->disk_name); if (!disk->zone_wplugs_wq) goto destroy_pool; diff --git a/block/blk.h b/block/blk.h index b998a7761faf..25af8ac5ef0f 100644 --- a/block/blk.h +++ b/block/blk.h @@ -4,6 +4,7 @@ #include <linux/bio-integrity.h> #include <linux/blk-crypto.h> +#include <linux/part_stat.h> #include <linux/lockdep.h> #include <linux/memblock.h> /* for max_pfn/max_low_pfn */ #include <linux/sched/sysctl.h> @@ -49,6 +50,11 @@ struct blk_flush_queue *blk_alloc_flush_queue(int node, int cmd_size, gfp_t flags); void blk_free_flush_queue(struct blk_flush_queue *q); +const char *blk_status_to_str(blk_status_t status); +const char *blk_status_to_tag(blk_status_t status); +blk_status_t tag_to_blk_status(const char *tag); +enum req_op str_to_blk_op(const char *op); + bool __blk_mq_unfreeze_queue(struct request_queue *q, bool force_atomic); bool blk_queue_start_drain(struct request_queue *q); bool __blk_freeze_queue_start(struct request_queue *q, @@ -402,6 +408,8 @@ static inline bool bio_may_need_split(struct bio *bio, bv = __bvec_iter_bvec(bio->bi_io_vec, bio->bi_iter); if (bio->bi_iter.bi_size > bv->bv_len - bio->bi_iter.bi_bvec_done) return true; + if ((bv->bv_offset | bv->bv_len) & lim->dma_alignment) + return true; return bv->bv_len + bv->bv_offset > lim->max_fast_segment_size; } @@ -485,6 +493,26 @@ static inline void req_set_nomerge(struct request_queue *q, struct request *req) q->last_merge = NULL; } +static inline void bdev_inc_in_flight(struct block_device *bdev, + enum req_op op) +{ + bool rw = op_is_write(op); + + part_stat_local_inc(bdev, in_flight[rw]); + if (bdev_is_partition(bdev)) + part_stat_local_inc(bdev_whole(bdev), in_flight[rw]); +} + +static inline void bdev_dec_in_flight(struct block_device *bdev, + enum req_op op) +{ + bool rw = op_is_write(op); + + part_stat_local_dec(bdev, in_flight[rw]); + if (bdev_is_partition(bdev)) + part_stat_local_dec(bdev_whole(bdev), in_flight[rw]); +} + /* * Internal io_context interface */ @@ -754,16 +782,19 @@ static inline void blk_unfreeze_release_lock(struct request_queue *q) * reclaim from triggering block I/O. */ static inline void blk_debugfs_lock_nomemsave(struct request_queue *q) + __acquires(&q->debugfs_mutex) { mutex_lock(&q->debugfs_mutex); } static inline void blk_debugfs_unlock_nomemrestore(struct request_queue *q) + __releases(&q->debugfs_mutex) { mutex_unlock(&q->debugfs_mutex); } static inline unsigned int __must_check blk_debugfs_lock(struct request_queue *q) + __acquires(&q->debugfs_mutex) { unsigned int memflags = memalloc_noio_save(); @@ -773,6 +804,7 @@ static inline unsigned int __must_check blk_debugfs_lock(struct request_queue *q static inline void blk_debugfs_unlock(struct request_queue *q, unsigned int memflags) + __releases(&q->debugfs_mutex) { blk_debugfs_unlock_nomemrestore(q); memalloc_noio_restore(memflags); diff --git a/block/bsg-lib.c b/block/bsg-lib.c index fdb4b290ca68..895db30a7033 100644 --- a/block/bsg-lib.c +++ b/block/bsg-lib.c @@ -299,7 +299,7 @@ out: /* called right after the request is allocated for the request_queue */ static int bsg_init_rq(struct blk_mq_tag_set *set, struct request *req, - unsigned int hctx_idx, unsigned int numa_node) + unsigned int hctx_idx, int numa_node) { struct bsg_job *job = blk_mq_rq_to_pdu(req); diff --git a/block/error-injection.c b/block/error-injection.c new file mode 100644 index 000000000000..d24c90e9a25f --- /dev/null +++ b/block/error-injection.c @@ -0,0 +1,315 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Copyright (c) 2026 Christoph Hellwig. + */ +#include <linux/debugfs.h> +#include <linux/blkdev.h> +#include <linux/parser.h> +#include <linux/seq_file.h> +#include "blk.h" +#include "error-injection.h" + +struct blk_error_inject { + struct list_head entry; + sector_t start; + sector_t end; + enum req_op op; + blk_status_t status; + + /* only inject every 1 / chance times */ + unsigned int chance; +}; + +DEFINE_STATIC_KEY_FALSE(blk_error_injection_enabled); + +bool __blk_error_inject(struct bio *bio) +{ + struct gendisk *disk = bio->bi_bdev->bd_disk; + struct blk_error_inject *inj; + + rcu_read_lock(); + list_for_each_entry_rcu(inj, &disk->error_injection_list, entry) { + if (bio_op(bio) != inj->op) + continue; + /* + * This never matches 0-sized bios like empty WRITEs with + * REQ_PREFLUSH or ZONE_RESET_ALL. While adding a special case + * for them would be trivial, that means any WRITE rule would + * trigger for flushes. So before we can make this work + * properly, we'll need to start using REQ_OP_FLUSH for pure + * flushes at the bio level like we already do in blk-mq. + */ + if (bio->bi_iter.bi_sector > inj->end || + bio_end_sector(bio) <= inj->start) + continue; + if (inj->chance > 1 && (get_random_u32() % inj->chance) != 0) + continue; + + pr_info_ratelimited("%pg: injecting %s error for %s at sector %llu:%u\n", + disk->part0, blk_status_to_str(inj->status), + blk_op_str(inj->op), bio->bi_iter.bi_sector, + bio_sectors(bio)); + bio->bi_status = inj->status; + rcu_read_unlock(); + bio_endio(bio); + return true; + } + rcu_read_unlock(); + return false; +} + +static int error_inject_add(struct gendisk *disk, enum req_op op, + sector_t start, u64 nr_sectors, blk_status_t status, + unsigned int chance) +{ + struct blk_error_inject *inj; + int error = -EINVAL; + + if (op == REQ_OP_LAST) + return -EINVAL; + if (status == BLK_STS_OK) + return -EINVAL; + + inj = kzalloc_obj(*inj); + if (!inj) + return -ENOMEM; + + if (nr_sectors) { + if (U64_MAX - nr_sectors < start) + goto out_free_inj; + inj->end = start + nr_sectors - 1; + } else { + inj->end = U64_MAX; + } + + inj->op = op; + inj->start = start; + inj->status = status; + inj->chance = chance; + + pr_debug_ratelimited("%pg: adding %s injection for %s at sector %llu:%llu\n", + disk->part0, blk_status_to_str(status), + blk_op_str(op), + start, nr_sectors); + + /* + * Add to the front of the list so that newer entries can partially + * override other entries. This also intentionally allows duplicate + * entries as there is no real reason to reject them. + */ + mutex_lock(&disk->error_injection_lock); + if (!disk_live(disk)) { + mutex_unlock(&disk->error_injection_lock); + error = -ENODEV; + goto out_free_inj; + } + if (list_empty(&disk->error_injection_list)) + static_branch_inc(&blk_error_injection_enabled); + list_add_rcu(&inj->entry, &disk->error_injection_list); + set_bit(GD_ERROR_INJECT, &disk->state); + mutex_unlock(&disk->error_injection_lock); + return 0; + +out_free_inj: + kfree(inj); + return error; +} + +static void error_inject_removeall(struct gendisk *disk) +{ + struct blk_error_inject *inj; + + mutex_lock(&disk->error_injection_lock); + clear_bit(GD_ERROR_INJECT, &disk->state); + while ((inj = list_first_entry_or_null(&disk->error_injection_list, + struct blk_error_inject, entry))) { + list_del_rcu(&inj->entry); + kfree_rcu_mightsleep(inj); + } + static_branch_dec(&blk_error_injection_enabled); + mutex_unlock(&disk->error_injection_lock); +} + +enum options { + Opt_add = (1u << 0), + Opt_removeall = (1u << 1), + + Opt_op = (1u << 16), + Opt_start = (1u << 17), + Opt_nr_sectors = (1u << 18), + Opt_status = (1u << 19), + Opt_chance = (1u << 20), + + Opt_invalid, +}; + +static const match_table_t opt_tokens = { + { Opt_add, "add", }, + { Opt_removeall, "removeall", }, + { Opt_op, "op=%s", }, + { Opt_start, "start=%u" }, + { Opt_nr_sectors, "nr_sectors=%u" }, + { Opt_status, "status=%s" }, + { Opt_chance, "chance=%u" }, + { Opt_invalid, NULL, }, +}; + +static int match_op(substring_t *args, enum req_op *op) +{ + const char *tag; + + tag = match_strdup(args); + if (!tag) + return -ENOMEM; + *op = str_to_blk_op(tag); + if (*op == REQ_OP_LAST) + pr_warn("invalid op '%s'\n", tag); + kfree(tag); + return 0; +} + +static int match_status(substring_t *args, blk_status_t *status) +{ + const char *tag; + + tag = match_strdup(args); + if (!tag) + return -ENOMEM; + *status = tag_to_blk_status(tag); + if (!*status) + pr_warn("invalid status '%s'\n", tag); + kfree(tag); + return 0; +} + +static ssize_t blk_error_injection_parse_options(struct gendisk *disk, + char *options) +{ + enum { Unset, Add, Removeall } action = Unset; + unsigned int option_mask = 0, chance = 1; + enum req_op op = REQ_OP_LAST; + u64 start = 0, nr_sectors = 0; + blk_status_t status = BLK_STS_OK; + substring_t args[MAX_OPT_ARGS]; + char *p; + + while ((p = strsep(&options, ",\n")) != NULL) { + int error = 0; + ssize_t token; + + if (!*p) + continue; + token = match_token(p, opt_tokens, args); + option_mask |= token; + switch (token) { + case Opt_add: + if (action != Unset) + return -EINVAL; + action = Add; + break; + case Opt_removeall: + if (action != Unset) + return -EINVAL; + action = Removeall; + break; + case Opt_op: + error = match_op(args, &op); + break; + case Opt_start: + error = match_u64(args, &start); + break; + case Opt_nr_sectors: + error = match_u64(args, &nr_sectors); + break; + case Opt_status: + error = match_status(args, &status); + break; + case Opt_chance: + error = match_uint(args, &chance); + if (!error && chance == 0) + error = -EINVAL; + break; + default: + pr_warn("unknown parameter or missing value '%s'\n", p); + error = -EINVAL; + } + if (error) + return error; + } + + switch (action) { + case Add: + return error_inject_add(disk, op, start, nr_sectors, status, + chance); + case Removeall: + if (option_mask & ~Opt_removeall) + return -EINVAL; + error_inject_removeall(disk); + return 0; + default: + return -EINVAL; + } +} + +static ssize_t blk_error_injection_write(struct file *file, + const char __user *ubuf, size_t count, loff_t *pos) +{ + struct gendisk *disk = file_inode(file)->i_private; + char *options; + int error; + + options = memdup_user_nul(ubuf, count); + if (IS_ERR(options)) + return PTR_ERR(options); + error = blk_error_injection_parse_options(disk, options); + kfree(options); + + if (error) + return error; + return count; +} + +static int blk_error_injection_show(struct seq_file *s, void *private) +{ + struct gendisk *disk = s->private; + struct blk_error_inject *inj; + + rcu_read_lock(); + list_for_each_entry_rcu(inj, &disk->error_injection_list, entry) { + seq_printf(s, "%llu:%llu status=%s,chance=%u", + inj->start, inj->end, + blk_status_to_tag(inj->status), inj->chance); + seq_putc(s, '\n'); + } + rcu_read_unlock(); + return 0; +} + +static int blk_error_injection_open(struct inode *inode, struct file *file) +{ + return single_open(file, blk_error_injection_show, inode->i_private); +} + +static int blk_error_injection_release(struct inode *inode, struct file *file) +{ + return single_release(inode, file); +} + +static const struct file_operations blk_error_injection_fops = { + .owner = THIS_MODULE, + .write = blk_error_injection_write, + .read = seq_read, + .open = blk_error_injection_open, + .release = blk_error_injection_release, +}; + +void blk_error_injection_init(struct gendisk *disk) +{ + debugfs_create_file("error_injection", 0600, disk->queue->debugfs_dir, + disk, &blk_error_injection_fops); +} + +void blk_error_injection_exit(struct gendisk *disk) +{ + error_inject_removeall(disk); +} diff --git a/block/error-injection.h b/block/error-injection.h new file mode 100644 index 000000000000..9821d773abab --- /dev/null +++ b/block/error-injection.h @@ -0,0 +1,21 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _BLK_ERROR_INJECTION_H +#define _BLK_ERROR_INJECTION_H 1 + +#include <linux/jump_label.h> + +DECLARE_STATIC_KEY_FALSE(blk_error_injection_enabled); + +void blk_error_injection_init(struct gendisk *disk); +void blk_error_injection_exit(struct gendisk *disk); +bool __blk_error_inject(struct bio *bio); +static inline bool blk_error_inject(struct bio *bio) +{ + if (IS_ENABLED(CONFIG_BLK_ERROR_INJECTION) && + static_branch_unlikely(&blk_error_injection_enabled) && + test_bit(GD_ERROR_INJECT, &bio->bi_bdev->bd_disk->state)) + return __blk_error_inject(bio); + return false; +} + +#endif /* _BLK_ERROR_INJECTION_H */ diff --git a/block/fops.c b/block/fops.c index bb6642b45937..15783a6180de 100644 --- a/block/fops.c +++ b/block/fops.c @@ -218,8 +218,7 @@ static ssize_t __blkdev_direct_IO(struct kiocb *iocb, struct iov_iter *iter, ret = blkdev_iov_iter_get_pages(bio, iter, bdev); if (unlikely(ret)) { - bio->bi_status = BLK_STS_IOERR; - bio_endio(bio); + bio_endio_status(bio, BLK_STS_IOERR); break; } if (iocb->ki_flags & IOCB_NOWAIT) { @@ -499,36 +498,12 @@ static void blkdev_readahead(struct readahead_control *rac) mpage_readahead(rac, blkdev_get_block); } -static int blkdev_write_begin(const struct kiocb *iocb, - struct address_space *mapping, loff_t pos, - unsigned len, struct folio **foliop, - void **fsdata) -{ - return block_write_begin(mapping, pos, len, foliop, blkdev_get_block); -} - -static int blkdev_write_end(const struct kiocb *iocb, - struct address_space *mapping, - loff_t pos, unsigned len, unsigned copied, - struct folio *folio, void *fsdata) -{ - int ret; - ret = block_write_end(pos, len, copied, folio); - - folio_unlock(folio); - folio_put(folio); - - return ret; -} - const struct address_space_operations def_blk_aops = { .dirty_folio = block_dirty_folio, .invalidate_folio = block_invalidate_folio, .read_folio = blkdev_read_folio, .readahead = blkdev_readahead, .writepages = blkdev_writepages, - .write_begin = blkdev_write_begin, - .write_end = blkdev_write_end, .migrate_folio = buffer_migrate_folio_norefs, .is_dirty_writeback = buffer_check_dirty_writeback, }; diff --git a/block/genhd.c b/block/genhd.c index 7d6854fd28e9..f84b6a355b57 100644 --- a/block/genhd.c +++ b/block/genhd.c @@ -1486,6 +1486,10 @@ struct gendisk *__alloc_disk_node(struct request_queue *q, int node_id, #ifdef CONFIG_BLOCK_HOLDER_DEPRECATED INIT_LIST_HEAD(&disk->slave_bdevs); #endif +#ifdef CONFIG_BLK_ERROR_INJECTION + mutex_init(&disk->error_injection_lock); + INIT_LIST_HEAD(&disk->error_injection_list); +#endif mutex_init(&disk->rqos_state_mutex); kobject_init(&disk->queue_kobj, &blk_queue_ktype); return disk; diff --git a/block/kyber-iosched.c b/block/kyber-iosched.c index b84163d1f851..971818bcdc9d 100644 --- a/block/kyber-iosched.c +++ b/block/kyber-iosched.c @@ -882,6 +882,9 @@ static const struct elv_fs_entry kyber_sched_attrs[] = { }; #undef KYBER_LAT_ATTR +#define HCTX_FROM_SEQ_FILE(m) ((struct blk_mq_hw_ctx *)(m)->private) +#define KYBER_HCTX_DATA(hctx) ((struct kyber_hctx_data *)(hctx)->sched_data) + #ifdef CONFIG_BLK_DEBUG_FS #define KYBER_DEBUGFS_DOMAIN_ATTRS(domain, name) \ static int kyber_##name##_tokens_show(void *data, struct seq_file *m) \ @@ -894,7 +897,7 @@ static int kyber_##name##_tokens_show(void *data, struct seq_file *m) \ } \ \ static void *kyber_##name##_rqs_start(struct seq_file *m, loff_t *pos) \ - __acquires(&khd->lock) \ + __acquires(&KYBER_HCTX_DATA(HCTX_FROM_SEQ_FILE(m))->lock) \ { \ struct blk_mq_hw_ctx *hctx = m->private; \ struct kyber_hctx_data *khd = hctx->sched_data; \ @@ -913,7 +916,7 @@ static void *kyber_##name##_rqs_next(struct seq_file *m, void *v, \ } \ \ static void kyber_##name##_rqs_stop(struct seq_file *m, void *v) \ - __releases(&khd->lock) \ + __releases(&KYBER_HCTX_DATA(HCTX_FROM_SEQ_FILE(m))->lock) \ { \ struct blk_mq_hw_ctx *hctx = m->private; \ struct kyber_hctx_data *khd = hctx->sched_data; \ diff --git a/block/mq-deadline.c b/block/mq-deadline.c index 95917a88976f..824bfc17b2c6 100644 --- a/block/mq-deadline.c +++ b/block/mq-deadline.c @@ -794,11 +794,15 @@ static const struct elv_fs_entry deadline_attrs[] = { __ATTR_NULL }; +#define RQ_FROM_SEQ_FILE(m) ((struct request_queue *)(m)->private) +#define DD_DATA_FROM_RQ(rq) \ + ((struct deadline_data *)(rq)->elevator->elevator_data) + #ifdef CONFIG_BLK_DEBUG_FS #define DEADLINE_DEBUGFS_DDIR_ATTRS(prio, data_dir, name) \ static void *deadline_##name##_fifo_start(struct seq_file *m, \ loff_t *pos) \ - __acquires(&dd->lock) \ + __acquires(&DD_DATA_FROM_RQ(RQ_FROM_SEQ_FILE(m))->lock) \ { \ struct request_queue *q = m->private; \ struct deadline_data *dd = q->elevator->elevator_data; \ @@ -819,7 +823,7 @@ static void *deadline_##name##_fifo_next(struct seq_file *m, void *v, \ } \ \ static void deadline_##name##_fifo_stop(struct seq_file *m, void *v) \ - __releases(&dd->lock) \ + __releases(&DD_DATA_FROM_RQ(RQ_FROM_SEQ_FILE(m))->lock) \ { \ struct request_queue *q = m->private; \ struct deadline_data *dd = q->elevator->elevator_data; \ @@ -921,7 +925,7 @@ static int dd_owned_by_driver_show(void *data, struct seq_file *m) } static void *deadline_dispatch_start(struct seq_file *m, loff_t *pos) - __acquires(&dd->lock) + __acquires(&DD_DATA_FROM_RQ(RQ_FROM_SEQ_FILE(m))->lock) { struct request_queue *q = m->private; struct deadline_data *dd = q->elevator->elevator_data; @@ -939,7 +943,7 @@ static void *deadline_dispatch_next(struct seq_file *m, void *v, loff_t *pos) } static void deadline_dispatch_stop(struct seq_file *m, void *v) - __releases(&dd->lock) + __releases(&DD_DATA_FROM_RQ(RQ_FROM_SEQ_FILE(m))->lock) { struct request_queue *q = m->private; struct deadline_data *dd = q->elevator->elevator_data; diff --git a/block/partitions/acorn.c b/block/partitions/acorn.c index 9f7389f174d0..067d6a27a3bd 100644 --- a/block/partitions/acorn.c +++ b/block/partitions/acorn.c @@ -9,6 +9,7 @@ */ #include <linux/buffer_head.h> #include <linux/adfs_fs.h> +#include <linux/minmax.h> #include "check.h" @@ -80,7 +81,7 @@ static int riscix_partition(struct parsed_partitions *state, if (rr->magic == RISCIX_MAGIC) { - unsigned long size = nr_sects > 2 ? 2 : nr_sects; + unsigned long size = min(nr_sects, 2); int part; seq_buf_puts(&state->pp_buf, " <"); @@ -124,7 +125,7 @@ static int linux_partition(struct parsed_partitions *state, { Sector sect; struct linux_part *linuxp; - unsigned long size = nr_sects > 2 ? 2 : nr_sects; + unsigned long size = min(nr_sects, 2); seq_buf_puts(&state->pp_buf, " [Linux]"); diff --git a/block/partitions/aix.c b/block/partitions/aix.c index 29b8f4cebb63..f3c4174e003e 100644 --- a/block/partitions/aix.c +++ b/block/partitions/aix.c @@ -226,6 +226,15 @@ int aix_partition(struct parsed_partitions *state) int next_lp_ix = 1; int lp_ix; + /* + * pvd was read into a fixed-size struct pvd whose ppe[] array + * holds ARRAY_SIZE(pvd->ppe) entries. pp_count is an + * unvalidated on-disk __be16, so clamp the scan to the array + * size to avoid walking past the allocation. + */ + if (numpps > ARRAY_SIZE(pvd->ppe)) + numpps = ARRAY_SIZE(pvd->ppe); + for (i = 0; i < numpps; i += 1) { struct ppe *p = pvd->ppe + i; unsigned int lv_ix; diff --git a/block/partitions/core.c b/block/partitions/core.c index 5d5332ce586b..b5c59b79ca7c 100644 --- a/block/partitions/core.c +++ b/block/partitions/core.c @@ -124,7 +124,7 @@ static struct parsed_partitions *check_partition(struct gendisk *hd) state = allocate_partitions(hd); if (!state) return NULL; - state->pp_buf.buffer = (char *)__get_free_page(GFP_KERNEL); + state->pp_buf.buffer = kmalloc(PAGE_SIZE, GFP_KERNEL); if (!state->pp_buf.buffer) { free_partitions(state); return NULL; @@ -154,7 +154,7 @@ static struct parsed_partitions *check_partition(struct gendisk *hd) if (res > 0) { printk(KERN_INFO "%s", seq_buf_str(&state->pp_buf)); - free_page((unsigned long)state->pp_buf.buffer); + kfree(state->pp_buf.buffer); return state; } if (state->access_beyond_eod) @@ -170,7 +170,7 @@ static struct parsed_partitions *check_partition(struct gendisk *hd) printk(KERN_INFO "%s", seq_buf_str(&state->pp_buf)); } - free_page((unsigned long)state->pp_buf.buffer); + kfree(state->pp_buf.buffer); free_partitions(state); return ERR_PTR(res); } diff --git a/block/partitions/of.c b/block/partitions/of.c index c22b60661098..53664ea06b65 100644 --- a/block/partitions/of.c +++ b/block/partitions/of.c @@ -74,8 +74,10 @@ int of_partition(struct parsed_partitions *state) struct device_node *partitions_np = of_node_get(ddev->of_node); if (!partitions_np || - !of_device_is_compatible(partitions_np, "fixed-partitions")) + !of_device_is_compatible(partitions_np, "fixed-partitions")) { + of_node_put(partitions_np); return 0; + } slot = 1; /* Validate parition offset and size */ @@ -104,5 +106,6 @@ int of_partition(struct parsed_partitions *state) seq_buf_puts(&state->pp_buf, "\n"); + of_node_put(partitions_np); return 1; } diff --git a/drivers/block/drbd/Makefile b/drivers/block/drbd/Makefile index 187eaf81f0f8..5faaa8a8e7f0 100644 --- a/drivers/block/drbd/Makefile +++ b/drivers/block/drbd/Makefile @@ -3,6 +3,7 @@ drbd-y := drbd_buildtag.o drbd_bitmap.o drbd_proc.o drbd-y += drbd_worker.o drbd_receiver.o drbd_req.o drbd_actlog.o drbd-y += drbd_main.o drbd_strings.o drbd_nl.o drbd-y += drbd_interval.o drbd_state.o +drbd-y += drbd_nl_gen.o drbd-$(CONFIG_DEBUG_FS) += drbd_debugfs.o obj-$(CONFIG_BLK_DEV_DRBD) += drbd.o diff --git a/drivers/block/drbd/drbd_buildtag.c b/drivers/block/drbd/drbd_buildtag.c index cb1aa66d7d5d..cd0389488f63 100644 --- a/drivers/block/drbd/drbd_buildtag.c +++ b/drivers/block/drbd/drbd_buildtag.c @@ -1,5 +1,5 @@ // SPDX-License-Identifier: GPL-2.0-only -#include <linux/drbd_config.h> +#include "drbd_config.h" #include <linux/module.h> const char *drbd_buildtag(void) diff --git a/include/linux/drbd_config.h b/drivers/block/drbd/drbd_config.h index d215365c6bb1..d215365c6bb1 100644 --- a/include/linux/drbd_config.h +++ b/drivers/block/drbd/drbd_config.h diff --git a/drivers/block/drbd/drbd_debugfs.c b/drivers/block/drbd/drbd_debugfs.c index 12460b584bcb..371abcd7e880 100644 --- a/drivers/block/drbd/drbd_debugfs.c +++ b/drivers/block/drbd/drbd_debugfs.c @@ -844,7 +844,7 @@ static int drbd_version_show(struct seq_file *m, void *ignored) { seq_printf(m, "# %s\n", drbd_buildtag()); seq_printf(m, "VERSION=%s\n", REL_VERSION); - seq_printf(m, "API_VERSION=%u\n", GENL_MAGIC_VERSION); + seq_printf(m, "API_VERSION=%u\n", DRBD_FAMILY_VERSION); seq_printf(m, "PRO_VERSION_MIN=%u\n", PRO_VERSION_MIN); seq_printf(m, "PRO_VERSION_MAX=%u\n", PRO_VERSION_MAX); return 0; diff --git a/drivers/block/drbd/drbd_int.h b/drivers/block/drbd/drbd_int.h index f6d6276974ee..48b45c3142f7 100644 --- a/drivers/block/drbd/drbd_int.h +++ b/drivers/block/drbd/drbd_int.h @@ -32,14 +32,16 @@ #include <net/tcp.h> #include <linux/lru_cache.h> #include <linux/prefetch.h> -#include <linux/drbd_genl_api.h> #include <linux/drbd.h> -#include <linux/drbd_config.h> +#include "drbd_config.h" +#include "drbd_nl_gen.h" #include "drbd_strings.h" #include "drbd_state.h" #include "drbd_protocol.h" #include "drbd_polymorph_printk.h" +extern struct genl_family drbd_nl_family; + /* shared module parameters, defined in drbd_main.c */ #ifdef CONFIG_DRBD_FAULT_INJECTION extern int drbd_enable_faults; diff --git a/drivers/block/drbd/drbd_main.c b/drivers/block/drbd/drbd_main.c index b1a721dd0496..a2a841c89201 100644 --- a/drivers/block/drbd/drbd_main.c +++ b/drivers/block/drbd/drbd_main.c @@ -2324,7 +2324,7 @@ static void drbd_cleanup(void) if (retry.wq) destroy_workqueue(retry.wq); - drbd_genl_unregister(); + genl_unregister_family(&drbd_nl_family); idr_for_each_entry(&drbd_devices, device, i) drbd_delete_device(device); @@ -2846,7 +2846,7 @@ static int __init drbd_init(void) mutex_init(&resources_mutex); INIT_LIST_HEAD(&drbd_resources); - err = drbd_genl_register(); + err = genl_register_family(&drbd_nl_family); if (err) { pr_err("unable to register generic netlink family\n"); goto fail; @@ -2876,7 +2876,7 @@ static int __init drbd_init(void) pr_info("initialized. " "Version: " REL_VERSION " (api:%d/proto:%d-%d)\n", - GENL_MAGIC_VERSION, PRO_VERSION_MIN, PRO_VERSION_MAX); + DRBD_FAMILY_VERSION, PRO_VERSION_MIN, PRO_VERSION_MAX); pr_info("%s\n", drbd_buildtag()); pr_info("registered as block device major %d\n", DRBD_MAJOR); return 0; /* Success! */ diff --git a/drivers/block/drbd/drbd_nl.c b/drivers/block/drbd/drbd_nl.c index c2ac555473e7..f9ffcd67607b 100644 --- a/drivers/block/drbd/drbd_nl.c +++ b/drivers/block/drbd/drbd_nl.c @@ -31,59 +31,13 @@ #include <net/genetlink.h> -/* .doit */ -// int drbd_adm_create_resource(struct sk_buff *skb, struct genl_info *info); -// int drbd_adm_delete_resource(struct sk_buff *skb, struct genl_info *info); +#include "drbd_nl_gen.h" -int drbd_adm_new_minor(struct sk_buff *skb, struct genl_info *info); -int drbd_adm_del_minor(struct sk_buff *skb, struct genl_info *info); - -int drbd_adm_new_resource(struct sk_buff *skb, struct genl_info *info); -int drbd_adm_del_resource(struct sk_buff *skb, struct genl_info *info); -int drbd_adm_down(struct sk_buff *skb, struct genl_info *info); - -int drbd_adm_set_role(struct sk_buff *skb, struct genl_info *info); -int drbd_adm_attach(struct sk_buff *skb, struct genl_info *info); -int drbd_adm_disk_opts(struct sk_buff *skb, struct genl_info *info); -int drbd_adm_detach(struct sk_buff *skb, struct genl_info *info); -int drbd_adm_connect(struct sk_buff *skb, struct genl_info *info); -int drbd_adm_net_opts(struct sk_buff *skb, struct genl_info *info); -int drbd_adm_resize(struct sk_buff *skb, struct genl_info *info); -int drbd_adm_start_ov(struct sk_buff *skb, struct genl_info *info); -int drbd_adm_new_c_uuid(struct sk_buff *skb, struct genl_info *info); -int drbd_adm_disconnect(struct sk_buff *skb, struct genl_info *info); -int drbd_adm_invalidate(struct sk_buff *skb, struct genl_info *info); -int drbd_adm_invalidate_peer(struct sk_buff *skb, struct genl_info *info); -int drbd_adm_pause_sync(struct sk_buff *skb, struct genl_info *info); -int drbd_adm_resume_sync(struct sk_buff *skb, struct genl_info *info); -int drbd_adm_suspend_io(struct sk_buff *skb, struct genl_info *info); -int drbd_adm_resume_io(struct sk_buff *skb, struct genl_info *info); -int drbd_adm_outdate(struct sk_buff *skb, struct genl_info *info); -int drbd_adm_resource_opts(struct sk_buff *skb, struct genl_info *info); -int drbd_adm_get_status(struct sk_buff *skb, struct genl_info *info); -int drbd_adm_get_timeout_type(struct sk_buff *skb, struct genl_info *info); -/* .dumpit */ -int drbd_adm_get_status_all(struct sk_buff *skb, struct netlink_callback *cb); -int drbd_adm_dump_resources(struct sk_buff *skb, struct netlink_callback *cb); -int drbd_adm_dump_devices(struct sk_buff *skb, struct netlink_callback *cb); -int drbd_adm_dump_devices_done(struct netlink_callback *cb); -int drbd_adm_dump_connections(struct sk_buff *skb, struct netlink_callback *cb); -int drbd_adm_dump_connections_done(struct netlink_callback *cb); -int drbd_adm_dump_peer_devices(struct sk_buff *skb, struct netlink_callback *cb); -int drbd_adm_dump_peer_devices_done(struct netlink_callback *cb); -int drbd_adm_get_initial_state(struct sk_buff *skb, struct netlink_callback *cb); - -#include <linux/drbd_genl_api.h> - -static int drbd_pre_doit(const struct genl_split_ops *ops, - struct sk_buff *skb, struct genl_info *info); -static void drbd_post_doit(const struct genl_split_ops *ops, - struct sk_buff *skb, struct genl_info *info); - -#define GENL_MAGIC_FAMILY_PRE_DOIT drbd_pre_doit -#define GENL_MAGIC_FAMILY_POST_DOIT drbd_post_doit - -#include <linux/genl_magic_func.h> +static int drbd_genl_multicast_events(struct sk_buff *skb, gfp_t flags) +{ + return genlmsg_multicast(&drbd_nl_family, skb, 0, + DRBD_NLGRP_EVENTS, flags); +} static atomic_t drbd_genl_seq = ATOMIC_INIT(2); /* two. */ static atomic_t notify_genl_seq = ATOMIC_INIT(2); /* two. */ @@ -114,7 +68,7 @@ static int drbd_msg_put_info(struct sk_buff *skb, const char *info) if (!nla) return err; - err = nla_put_string(skb, T_info_text, info); + err = nla_put_string(skb, DRBD_A_DRBD_CFG_REPLY_INFO_TEXT, info); if (err) { nla_nest_cancel(skb, nla); return err; @@ -135,7 +89,7 @@ static int drbd_msg_sprintf_info(struct sk_buff *skb, const char *fmt, ...) if (!nla) return err; - txt = nla_reserve(skb, T_info_text, 256); + txt = nla_reserve(skb, DRBD_A_DRBD_CFG_REPLY_INFO_TEXT, 256); if (!txt) { nla_nest_cancel(skb, nla); return err; @@ -187,6 +141,15 @@ static const unsigned int drbd_genl_cmd_flags[] = { [DRBD_ADM_DOWN] = DRBD_ADM_NEED_RESOURCE, }; +/* Detect attempts to change invariant attributes in a _change_ handler. */ +#define has_invariant(ntb, attr) \ +({ \ + bool __found = !!(ntb)[attr]; \ + if (__found) \ + pr_info("must not change invariant attr: %s\n", #attr); \ + __found; \ +}) + /* * At this point, we still rely on the global genl_lock(). * If we want to avoid that, and allow "genl_family.parallel_ops", we may need @@ -210,7 +173,7 @@ static int drbd_adm_prepare(struct drbd_config_context *adm_ctx, } adm_ctx->reply_dh = genlmsg_put_reply(adm_ctx->reply_skb, - info, &drbd_genl_family, 0, cmd); + info, &drbd_nl_family, 0, cmd); /* put of a few bytes into a fresh skb of >= 4k will always succeed. * but anyways */ if (!adm_ctx->reply_dh) { @@ -223,9 +186,11 @@ static int drbd_adm_prepare(struct drbd_config_context *adm_ctx, adm_ctx->volume = VOLUME_UNSPECIFIED; if (info->attrs[DRBD_NLA_CFG_CONTEXT]) { + struct nlattr **ntb; struct nlattr *nla; - /* parse and validate only */ - err = drbd_cfg_context_from_attrs(NULL, info); + + /* parse and validate, get nested attribute table */ + err = drbd_cfg_context_ntb_from_attrs(&ntb, info); if (err) goto fail; @@ -234,18 +199,21 @@ static int drbd_adm_prepare(struct drbd_config_context *adm_ctx, err = nla_put_nohdr(adm_ctx->reply_skb, info->attrs[DRBD_NLA_CFG_CONTEXT]->nla_len, info->attrs[DRBD_NLA_CFG_CONTEXT]); - if (err) + if (err) { + kfree(ntb); goto fail; + } /* and assign stuff to the adm_ctx */ - nla = nested_attr_tb[T_ctx_volume]; + nla = ntb[DRBD_A_DRBD_CFG_CONTEXT_CTX_VOLUME]; if (nla) adm_ctx->volume = nla_get_u32(nla); - nla = nested_attr_tb[T_ctx_resource_name]; + nla = ntb[DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME]; if (nla) adm_ctx->resource_name = nla_data(nla); - adm_ctx->my_addr = nested_attr_tb[T_ctx_my_addr]; - adm_ctx->peer_addr = nested_attr_tb[T_ctx_peer_addr]; + adm_ctx->my_addr = ntb[DRBD_A_DRBD_CFG_CONTEXT_CTX_MY_ADDR]; + adm_ctx->peer_addr = ntb[DRBD_A_DRBD_CFG_CONTEXT_CTX_PEER_ADDR]; + kfree(ntb); if ((adm_ctx->my_addr && nla_len(adm_ctx->my_addr) > sizeof(adm_ctx->connection->my_addr)) || (adm_ctx->peer_addr && @@ -259,7 +227,7 @@ static int drbd_adm_prepare(struct drbd_config_context *adm_ctx, adm_ctx->device = minor_to_device(d_in->minor); /* We are protected by the global genl_lock(). - * But we may explicitly drop it/retake it in drbd_adm_set_role(), + * But we may explicitly drop it/retake it in drbd_nl_set_role(), * so make sure this object stays around. */ if (adm_ctx->device) kref_get(&adm_ctx->device->kref); @@ -334,8 +302,8 @@ fail: return err; } -static int drbd_pre_doit(const struct genl_split_ops *ops, - struct sk_buff *skb, struct genl_info *info) +int drbd_pre_doit(const struct genl_split_ops *ops, + struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx; u8 cmd = info->genlhdr->cmd; @@ -362,8 +330,8 @@ static int drbd_pre_doit(const struct genl_split_ops *ops, return 0; } -static void drbd_post_doit(const struct genl_split_ops *ops, - struct sk_buff *skb, struct genl_info *info) +void drbd_post_doit(const struct genl_split_ops *ops, + struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx = info->user_ptr[0]; @@ -828,7 +796,7 @@ static const char *from_attrs_err_to_txt(int err) "invalid attribute value"; } -int drbd_adm_set_role(struct sk_buff *skb, struct genl_info *info) +static int drbd_nl_set_role(struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx = info->user_ptr[0]; struct set_role_parms parms; @@ -868,6 +836,16 @@ out: return 0; } +int drbd_nl_primary_doit(struct sk_buff *skb, struct genl_info *info) +{ + return drbd_nl_set_role(skb, info); +} + +int drbd_nl_secondary_doit(struct sk_buff *skb, struct genl_info *info) +{ + return drbd_nl_set_role(skb, info); +} + /* Initializes the md.*_offset members, so we are able to find * the on disk meta data. * @@ -962,7 +940,7 @@ char *ppsize(char *buf, unsigned long long size) * peer may not initiate a resize. */ /* Note these are not to be confused with - * drbd_adm_suspend_io/drbd_adm_resume_io, + * drbd_nl_suspend_io_doit/drbd_nl_resume_io_doit, * which are (sub) state changes triggered by admin (drbdsetup), * and can be long lived. * This changes an device->flag, is triggered by drbd internals, @@ -1574,13 +1552,14 @@ out: return err; } -int drbd_adm_disk_opts(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_chg_disk_opts_doit(struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx = info->user_ptr[0]; enum drbd_ret_code retcode; struct drbd_device *device; struct disk_conf *new_disk_conf, *old_disk_conf; struct fifo_buffer *old_plan = NULL, *new_plan = NULL; + struct nlattr **ntb; int err; unsigned int fifo_size; @@ -1612,13 +1591,29 @@ int drbd_adm_disk_opts(struct sk_buff *skb, struct genl_info *info) if (should_set_defaults(info)) set_disk_conf_defaults(new_disk_conf); - err = disk_conf_from_attrs_for_change(new_disk_conf, info); + err = disk_conf_from_attrs(new_disk_conf, info); if (err && err != -ENOMSG) { retcode = ERR_MANDATORY_TAG; drbd_msg_put_info(adm_ctx->reply_skb, from_attrs_err_to_txt(err)); goto fail_unlock; } + err = disk_conf_ntb_from_attrs(&ntb, info); + if (!err) { + if (has_invariant(ntb, DRBD_A_DISK_CONF_BACKING_DEV) || + has_invariant(ntb, DRBD_A_DISK_CONF_META_DEV) || + has_invariant(ntb, DRBD_A_DISK_CONF_META_DEV_IDX) || + has_invariant(ntb, DRBD_A_DISK_CONF_DISK_SIZE) || + has_invariant(ntb, DRBD_A_DISK_CONF_MAX_BIO_BVECS)) { + retcode = ERR_MANDATORY_TAG; + drbd_msg_put_info(adm_ctx->reply_skb, + "cannot change invariant setting"); + kfree(ntb); + goto fail_unlock; + } + kfree(ntb); + } + if (!expect(device, new_disk_conf->resync_rate >= 1)) new_disk_conf->resync_rate = 1; @@ -1796,7 +1791,7 @@ void drbd_backing_dev_free(struct drbd_device *device, struct drbd_backing_dev * kfree(ldev); } -int drbd_adm_attach(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_attach_doit(struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx = info->user_ptr[0]; struct drbd_device *device; @@ -2236,7 +2231,7 @@ static int adm_detach(struct drbd_device *device, int force) * Then we transition to D_DISKLESS, and wait for put_ldev() to return all * internal references as well. * Only then we have finally detached. */ -int drbd_adm_detach(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_detach_doit(struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx = info->user_ptr[0]; enum drbd_ret_code retcode; @@ -2434,12 +2429,13 @@ static void free_crypto(struct crypto *crypto) crypto_free_shash(crypto->verify_tfm); } -int drbd_adm_net_opts(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_chg_net_opts_doit(struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx = info->user_ptr[0]; enum drbd_ret_code retcode; struct drbd_connection *connection; struct net_conf *old_net_conf, *new_net_conf = NULL; + struct nlattr **ntb; int err; int ovr; /* online verify running */ int rsr; /* re-sync running */ @@ -2476,13 +2472,26 @@ int drbd_adm_net_opts(struct sk_buff *skb, struct genl_info *info) if (should_set_defaults(info)) set_net_conf_defaults(new_net_conf); - err = net_conf_from_attrs_for_change(new_net_conf, info); + err = net_conf_from_attrs(new_net_conf, info); if (err && err != -ENOMSG) { retcode = ERR_MANDATORY_TAG; drbd_msg_put_info(adm_ctx->reply_skb, from_attrs_err_to_txt(err)); goto fail; } + err = net_conf_ntb_from_attrs(&ntb, info); + if (!err) { + if (has_invariant(ntb, DRBD_A_NET_CONF_DISCARD_MY_DATA) || + has_invariant(ntb, DRBD_A_NET_CONF_TENTATIVE)) { + retcode = ERR_MANDATORY_TAG; + drbd_msg_put_info(adm_ctx->reply_skb, + "cannot change invariant setting"); + kfree(ntb); + goto fail; + } + kfree(ntb); + } + retcode = check_net_options(connection, new_net_conf); if (retcode != NO_ERROR) goto fail; @@ -2575,7 +2584,7 @@ static void peer_device_to_info(struct peer_device_info *info, info->peer_resync_susp_dependency = device->state.aftr_isp; } -int drbd_adm_connect(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_connect_doit(struct sk_buff *skb, struct genl_info *info) { struct connection_info connection_info; enum drbd_notification_type flags; @@ -2790,7 +2799,7 @@ repeat: return rv; } -int drbd_adm_disconnect(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_disconnect_doit(struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx = info->user_ptr[0]; struct disconnect_parms parms; @@ -2845,7 +2854,7 @@ void resync_after_online_grow(struct drbd_device *device) _drbd_request_state(device, NS(conn, C_WF_SYNC_UUID), CS_VERBOSE + CS_SERIALIZE); } -int drbd_adm_resize(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_resize_doit(struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx = info->user_ptr[0]; struct disk_conf *old_disk_conf, *new_disk_conf = NULL; @@ -2981,7 +2990,7 @@ int drbd_adm_resize(struct sk_buff *skb, struct genl_info *info) goto fail; } -int drbd_adm_resource_opts(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_resource_opts_doit(struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx = info->user_ptr[0]; enum drbd_ret_code retcode; @@ -3019,7 +3028,7 @@ fail: return 0; } -int drbd_adm_invalidate(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_invalidate_doit(struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx = info->user_ptr[0]; struct drbd_device *device; @@ -3097,7 +3106,7 @@ static int drbd_bmio_set_susp_al(struct drbd_device *device, return rv; } -int drbd_adm_invalidate_peer(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_inval_peer_doit(struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx = info->user_ptr[0]; int retcode; /* drbd_ret_code, drbd_state_rv */ @@ -3148,7 +3157,7 @@ out: return 0; } -int drbd_adm_pause_sync(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_pause_sync_doit(struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx = info->user_ptr[0]; enum drbd_ret_code retcode; @@ -3168,7 +3177,7 @@ out: return 0; } -int drbd_adm_resume_sync(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_resume_sync_doit(struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx = info->user_ptr[0]; union drbd_dev_state s; @@ -3196,12 +3205,12 @@ out: return 0; } -int drbd_adm_suspend_io(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_suspend_io_doit(struct sk_buff *skb, struct genl_info *info) { return drbd_adm_simple_request_state(skb, info, NS(susp, 1)); } -int drbd_adm_resume_io(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_resume_io_doit(struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx = info->user_ptr[0]; struct drbd_device *device; @@ -3257,7 +3266,7 @@ out: return 0; } -int drbd_adm_outdate(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_outdate_doit(struct sk_buff *skb, struct genl_info *info) { return drbd_adm_simple_request_state(skb, info, NS(disk, D_OUTDATED)); } @@ -3272,16 +3281,20 @@ static int nla_put_drbd_cfg_context(struct sk_buff *skb, if (!nla) goto nla_put_failure; if (device && - nla_put_u32(skb, T_ctx_volume, device->vnr)) + nla_put_u32(skb, DRBD_A_DRBD_CFG_CONTEXT_CTX_VOLUME, device->vnr)) goto nla_put_failure; - if (nla_put_string(skb, T_ctx_resource_name, resource->name)) + if (nla_put_string(skb, DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME, resource->name)) goto nla_put_failure; if (connection) { if (connection->my_addr_len && - nla_put(skb, T_ctx_my_addr, connection->my_addr_len, &connection->my_addr)) + nla_put(skb, DRBD_A_DRBD_CFG_CONTEXT_CTX_MY_ADDR, + connection->my_addr_len, + &connection->my_addr)) goto nla_put_failure; if (connection->peer_addr_len && - nla_put(skb, T_ctx_peer_addr, connection->peer_addr_len, &connection->peer_addr)) + nla_put(skb, DRBD_A_DRBD_CFG_CONTEXT_CTX_PEER_ADDR, + connection->peer_addr_len, + &connection->peer_addr)) goto nla_put_failure; } nla_nest_end(skb, nla); @@ -3300,7 +3313,7 @@ nla_put_failure: */ static struct nlattr *find_cfg_context_attr(const struct nlmsghdr *nlh, int attr) { - const unsigned hdrlen = GENL_HDRLEN + GENL_MAGIC_FAMILY_HDRSZ; + const unsigned int hdrlen = GENL_HDRLEN + sizeof(struct drbd_genlmsghdr); struct nlattr *nla; nla = nla_find(nlmsg_attrdata(nlh, hdrlen), nlmsg_attrlen(nlh, hdrlen), @@ -3312,7 +3325,7 @@ static struct nlattr *find_cfg_context_attr(const struct nlmsghdr *nlh, int attr static void resource_to_info(struct resource_info *, struct drbd_resource *); -int drbd_adm_dump_resources(struct sk_buff *skb, struct netlink_callback *cb) +int drbd_nl_get_resources_dumpit(struct sk_buff *skb, struct netlink_callback *cb) { struct drbd_genlmsghdr *dh; struct drbd_resource *resource; @@ -3340,7 +3353,7 @@ found_resource: put_result: dh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid, - cb->nlh->nlmsg_seq, &drbd_genl_family, + cb->nlh->nlmsg_seq, &drbd_nl_family, NLM_F_MULTI, DRBD_ADM_GET_RESOURCES); err = -ENOMEM; if (!dh) @@ -3350,15 +3363,15 @@ put_result: err = nla_put_drbd_cfg_context(skb, resource, NULL, NULL); if (err) goto out; - err = res_opts_to_skb(skb, &resource->res_opts, !capable(CAP_SYS_ADMIN)); + err = res_opts_to_skb(skb, &resource->res_opts); if (err) goto out; resource_to_info(&resource_info, resource); - err = resource_info_to_skb(skb, &resource_info, !capable(CAP_SYS_ADMIN)); + err = resource_info_to_skb(skb, &resource_info); if (err) goto out; resource_statistics.res_stat_write_ordering = resource->write_ordering; - err = resource_statistics_to_skb(skb, &resource_statistics, !capable(CAP_SYS_ADMIN)); + err = resource_statistics_to_skb(skb, &resource_statistics); if (err) goto out; cb->args[0] = (long)resource; @@ -3423,7 +3436,7 @@ int drbd_adm_dump_devices_done(struct netlink_callback *cb) { static void device_to_info(struct device_info *, struct drbd_device *); -int drbd_adm_dump_devices(struct sk_buff *skb, struct netlink_callback *cb) +int drbd_nl_get_devices_dumpit(struct sk_buff *skb, struct netlink_callback *cb) { struct nlattr *resource_filter; struct drbd_resource *resource; @@ -3436,7 +3449,8 @@ int drbd_adm_dump_devices(struct sk_buff *skb, struct netlink_callback *cb) resource = (struct drbd_resource *)cb->args[0]; if (!cb->args[0] && !cb->args[1]) { - resource_filter = find_cfg_context_attr(cb->nlh, T_ctx_resource_name); + resource_filter = find_cfg_context_attr(cb->nlh, + DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME); if (resource_filter) { retcode = ERR_RES_NOT_KNOWN; resource = drbd_find_resource(nla_data(resource_filter)); @@ -3465,7 +3479,7 @@ int drbd_adm_dump_devices(struct sk_buff *skb, struct netlink_callback *cb) put_result: dh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid, - cb->nlh->nlmsg_seq, &drbd_genl_family, + cb->nlh->nlmsg_seq, &drbd_nl_family, NLM_F_MULTI, DRBD_ADM_GET_DEVICES); err = -ENOMEM; if (!dh) @@ -3481,18 +3495,18 @@ put_result: struct disk_conf *disk_conf = rcu_dereference(device->ldev->disk_conf); - err = disk_conf_to_skb(skb, disk_conf, !capable(CAP_SYS_ADMIN)); + err = disk_conf_to_skb(skb, disk_conf); put_ldev(device); if (err) goto out; } device_to_info(&device_info, device); - err = device_info_to_skb(skb, &device_info, !capable(CAP_SYS_ADMIN)); + err = device_info_to_skb(skb, &device_info); if (err) goto out; device_to_statistics(&device_statistics, device); - err = device_statistics_to_skb(skb, &device_statistics, !capable(CAP_SYS_ADMIN)); + err = device_statistics_to_skb(skb, &device_statistics); if (err) goto out; cb->args[1] = minor + 1; @@ -3514,7 +3528,7 @@ int drbd_adm_dump_connections_done(struct netlink_callback *cb) enum { SINGLE_RESOURCE, ITERATE_RESOURCES }; -int drbd_adm_dump_connections(struct sk_buff *skb, struct netlink_callback *cb) +int drbd_nl_get_connections_dumpit(struct sk_buff *skb, struct netlink_callback *cb) { struct nlattr *resource_filter; struct drbd_resource *resource = NULL, *next_resource; @@ -3527,7 +3541,8 @@ int drbd_adm_dump_connections(struct sk_buff *skb, struct netlink_callback *cb) rcu_read_lock(); resource = (struct drbd_resource *)cb->args[0]; if (!cb->args[0]) { - resource_filter = find_cfg_context_attr(cb->nlh, T_ctx_resource_name); + resource_filter = find_cfg_context_attr(cb->nlh, + DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME); if (resource_filter) { retcode = ERR_RES_NOT_KNOWN; resource = drbd_find_resource(nla_data(resource_filter)); @@ -3591,7 +3606,7 @@ found_resource: put_result: dh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid, - cb->nlh->nlmsg_seq, &drbd_genl_family, + cb->nlh->nlmsg_seq, &drbd_nl_family, NLM_F_MULTI, DRBD_ADM_GET_CONNECTIONS); err = -ENOMEM; if (!dh) @@ -3606,16 +3621,16 @@ put_result: goto out; net_conf = rcu_dereference(connection->net_conf); if (net_conf) { - err = net_conf_to_skb(skb, net_conf, !capable(CAP_SYS_ADMIN)); + err = net_conf_to_skb(skb, net_conf); if (err) goto out; } connection_to_info(&connection_info, connection); - err = connection_info_to_skb(skb, &connection_info, !capable(CAP_SYS_ADMIN)); + err = connection_info_to_skb(skb, &connection_info); if (err) goto out; connection_statistics.conn_congested = test_bit(NET_CONGESTED, &connection->flags); - err = connection_statistics_to_skb(skb, &connection_statistics, !capable(CAP_SYS_ADMIN)); + err = connection_statistics_to_skb(skb, &connection_statistics); if (err) goto out; cb->args[2] = (long)connection; @@ -3676,7 +3691,7 @@ int drbd_adm_dump_peer_devices_done(struct netlink_callback *cb) return put_resource_in_arg0(cb, 9); } -int drbd_adm_dump_peer_devices(struct sk_buff *skb, struct netlink_callback *cb) +int drbd_nl_get_peer_devices_dumpit(struct sk_buff *skb, struct netlink_callback *cb) { struct nlattr *resource_filter; struct drbd_resource *resource; @@ -3688,7 +3703,8 @@ int drbd_adm_dump_peer_devices(struct sk_buff *skb, struct netlink_callback *cb) resource = (struct drbd_resource *)cb->args[0]; if (!cb->args[0] && !cb->args[1]) { - resource_filter = find_cfg_context_attr(cb->nlh, T_ctx_resource_name); + resource_filter = find_cfg_context_attr(cb->nlh, + DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME); if (resource_filter) { retcode = ERR_RES_NOT_KNOWN; resource = drbd_find_resource(nla_data(resource_filter)); @@ -3735,7 +3751,7 @@ found_peer_device: put_result: dh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid, - cb->nlh->nlmsg_seq, &drbd_genl_family, + cb->nlh->nlmsg_seq, &drbd_nl_family, NLM_F_MULTI, DRBD_ADM_GET_PEER_DEVICES); err = -ENOMEM; if (!dh) @@ -3751,11 +3767,11 @@ put_result: if (err) goto out; peer_device_to_info(&peer_device_info, peer_device); - err = peer_device_info_to_skb(skb, &peer_device_info, !capable(CAP_SYS_ADMIN)); + err = peer_device_info_to_skb(skb, &peer_device_info); if (err) goto out; peer_device_to_statistics(&peer_device_statistics, peer_device); - err = peer_device_statistics_to_skb(skb, &peer_device_statistics, !capable(CAP_SYS_ADMIN)); + err = peer_device_statistics_to_skb(skb, &peer_device_statistics); if (err) goto out; cb->args[1] = minor; @@ -3795,11 +3811,11 @@ static int nla_put_status_info(struct sk_buff *skb, struct drbd_device *device, /* If sib != NULL, this is drbd_bcast_event, which anyone can listen * to. So we better exclude_sensitive information. * - * If sib == NULL, this is drbd_adm_get_status, executed synchronously + * If sib == NULL, this is drbd_nl_get_status_doit, executed synchronously * in the context of the requesting user process. Exclude sensitive * information, unless current has superuser. * - * NOTE: for drbd_adm_get_status_all(), this is a netlink dump, and + * NOTE: for drbd_nl_get_status_dumpit(), this is a netlink dump, and * relies on the current implementation of netlink_dump(), which * executes the dump callback successively from netlink_recvmsg(), * always in the context of the receiving process */ @@ -3812,7 +3828,7 @@ static int nla_put_status_info(struct sk_buff *skb, struct drbd_device *device, if (nla_put_drbd_cfg_context(skb, resource, the_only_connection(resource), device)) goto nla_put_failure; - if (res_opts_to_skb(skb, &device->resource->res_opts, exclude_sensitive)) + if (res_opts_to_skb(skb, &device->resource->res_opts)) goto nla_put_failure; rcu_read_lock(); @@ -3820,14 +3836,24 @@ static int nla_put_status_info(struct sk_buff *skb, struct drbd_device *device, struct disk_conf *disk_conf; disk_conf = rcu_dereference(device->ldev->disk_conf); - err = disk_conf_to_skb(skb, disk_conf, exclude_sensitive); + err = disk_conf_to_skb(skb, disk_conf); } if (!err) { struct net_conf *nc; nc = rcu_dereference(first_peer_device(device)->connection->net_conf); - if (nc) - err = net_conf_to_skb(skb, nc, exclude_sensitive); + if (nc) { + if (exclude_sensitive) { + struct net_conf nc_clean = *nc; + + memset(nc_clean.shared_secret, 0, + sizeof(nc_clean.shared_secret)); + nc_clean.shared_secret_len = 0; + err = net_conf_to_skb(skb, &nc_clean); + } else { + err = net_conf_to_skb(skb, nc); + } + } } rcu_read_unlock(); if (err) @@ -3836,42 +3862,57 @@ static int nla_put_status_info(struct sk_buff *skb, struct drbd_device *device, nla = nla_nest_start_noflag(skb, DRBD_NLA_STATE_INFO); if (!nla) goto nla_put_failure; - if (nla_put_u32(skb, T_sib_reason, sib ? sib->sib_reason : SIB_GET_STATUS_REPLY) || - nla_put_u32(skb, T_current_state, device->state.i) || - nla_put_u64_0pad(skb, T_ed_uuid, device->ed_uuid) || - nla_put_u64_0pad(skb, T_capacity, get_capacity(device->vdisk)) || - nla_put_u64_0pad(skb, T_send_cnt, device->send_cnt) || - nla_put_u64_0pad(skb, T_recv_cnt, device->recv_cnt) || - nla_put_u64_0pad(skb, T_read_cnt, device->read_cnt) || - nla_put_u64_0pad(skb, T_writ_cnt, device->writ_cnt) || - nla_put_u64_0pad(skb, T_al_writ_cnt, device->al_writ_cnt) || - nla_put_u64_0pad(skb, T_bm_writ_cnt, device->bm_writ_cnt) || - nla_put_u32(skb, T_ap_bio_cnt, atomic_read(&device->ap_bio_cnt)) || - nla_put_u32(skb, T_ap_pending_cnt, atomic_read(&device->ap_pending_cnt)) || - nla_put_u32(skb, T_rs_pending_cnt, atomic_read(&device->rs_pending_cnt))) + if (nla_put_u32(skb, DRBD_A_STATE_INFO_SIB_REASON, + sib ? sib->sib_reason : SIB_GET_STATUS_REPLY) || + nla_put_u32(skb, DRBD_A_STATE_INFO_CURRENT_STATE, + device->state.i) || + nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_ED_UUID, + device->ed_uuid, 0) || + nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_CAPACITY, + get_capacity(device->vdisk), 0) || + nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_SEND_CNT, + device->send_cnt, 0) || + nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_RECV_CNT, + device->recv_cnt, 0) || + nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_READ_CNT, + device->read_cnt, 0) || + nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_WRIT_CNT, + device->writ_cnt, 0) || + nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_AL_WRIT_CNT, + device->al_writ_cnt, 0) || + nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BM_WRIT_CNT, + device->bm_writ_cnt, 0) || + nla_put_u32(skb, DRBD_A_STATE_INFO_AP_BIO_CNT, + atomic_read(&device->ap_bio_cnt)) || + nla_put_u32(skb, DRBD_A_STATE_INFO_AP_PENDING_CNT, + atomic_read(&device->ap_pending_cnt)) || + nla_put_u32(skb, DRBD_A_STATE_INFO_RS_PENDING_CNT, + atomic_read(&device->rs_pending_cnt))) goto nla_put_failure; if (got_ldev) { int err; spin_lock_irq(&device->ldev->md.uuid_lock); - err = nla_put(skb, T_uuids, sizeof(si->uuids), device->ldev->md.uuid); + err = nla_put(skb, DRBD_A_STATE_INFO_UUIDS, + sizeof(si->uuids), + device->ldev->md.uuid); spin_unlock_irq(&device->ldev->md.uuid_lock); if (err) goto nla_put_failure; - if (nla_put_u32(skb, T_disk_flags, device->ldev->md.flags) || - nla_put_u64_0pad(skb, T_bits_total, drbd_bm_bits(device)) || - nla_put_u64_0pad(skb, T_bits_oos, - drbd_bm_total_weight(device))) + if (nla_put_u32(skb, DRBD_A_STATE_INFO_DISK_FLAGS, device->ldev->md.flags) || + nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BITS_TOTAL, drbd_bm_bits(device), 0) || + nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BITS_OOS, + drbd_bm_total_weight(device), 0)) goto nla_put_failure; if (C_SYNC_SOURCE <= device->state.conn && C_PAUSED_SYNC_T >= device->state.conn) { - if (nla_put_u64_0pad(skb, T_bits_rs_total, - device->rs_total) || - nla_put_u64_0pad(skb, T_bits_rs_failed, - device->rs_failed)) + if (nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BITS_RS_TOTAL, + device->rs_total, 0) || + nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BITS_RS_FAILED, + device->rs_failed, 0)) goto nla_put_failure; } } @@ -3882,17 +3923,17 @@ static int nla_put_status_info(struct sk_buff *skb, struct drbd_device *device, case SIB_GET_STATUS_REPLY: break; case SIB_STATE_CHANGE: - if (nla_put_u32(skb, T_prev_state, sib->os.i) || - nla_put_u32(skb, T_new_state, sib->ns.i)) + if (nla_put_u32(skb, DRBD_A_STATE_INFO_PREV_STATE, sib->os.i) || + nla_put_u32(skb, DRBD_A_STATE_INFO_NEW_STATE, sib->ns.i)) goto nla_put_failure; break; case SIB_HELPER_POST: - if (nla_put_u32(skb, T_helper_exit_code, + if (nla_put_u32(skb, DRBD_A_STATE_INFO_HELPER_EXIT_CODE, sib->helper_exit_code)) goto nla_put_failure; fallthrough; case SIB_HELPER_PRE: - if (nla_put_string(skb, T_helper, sib->helper_name)) + if (nla_put_string(skb, DRBD_A_STATE_INFO_HELPER, sib->helper_name)) goto nla_put_failure; break; } @@ -3907,7 +3948,7 @@ nla_put_failure: return err; } -int drbd_adm_get_status(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_get_status_doit(struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx = info->user_ptr[0]; enum drbd_ret_code retcode; @@ -3997,7 +4038,7 @@ next_resource: } dh = genlmsg_put(skb, NETLINK_CB(cb->skb).portid, - cb->nlh->nlmsg_seq, &drbd_genl_family, + cb->nlh->nlmsg_seq, &drbd_nl_family, NLM_F_MULTI, DRBD_ADM_GET_STATUS); if (!dh) goto out; @@ -4017,7 +4058,7 @@ next_resource: struct net_conf *nc; nc = rcu_dereference(connection->net_conf); - if (nc && net_conf_to_skb(skb, nc, 1) != 0) + if (nc && net_conf_to_skb(skb, nc) != 0) goto cancel; } goto done; @@ -4059,9 +4100,9 @@ out: * * Once things are setup properly, we call into get_one_status(). */ -int drbd_adm_get_status_all(struct sk_buff *skb, struct netlink_callback *cb) +int drbd_nl_get_status_dumpit(struct sk_buff *skb, struct netlink_callback *cb) { - const unsigned hdrlen = GENL_HDRLEN + GENL_MAGIC_FAMILY_HDRSZ; + const unsigned int hdrlen = GENL_HDRLEN + sizeof(struct drbd_genlmsghdr); struct nlattr *nla; const char *resource_name; struct drbd_resource *resource; @@ -4084,7 +4125,7 @@ int drbd_adm_get_status_all(struct sk_buff *skb, struct netlink_callback *cb) /* No explicit context given. Dump all. */ if (!nla) goto dump; - nla = nla_find_nested(nla, T_ctx_resource_name); + nla = nla_find_nested(nla, DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME); /* context given, but no name present? */ if (!nla) return -EINVAL; @@ -4107,7 +4148,7 @@ dump: return get_one_status(skb, cb); } -int drbd_adm_get_timeout_type(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_get_timeout_type_doit(struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx = info->user_ptr[0]; enum drbd_ret_code retcode; @@ -4125,7 +4166,7 @@ int drbd_adm_get_timeout_type(struct sk_buff *skb, struct genl_info *info) test_bit(USE_DEGR_WFC_T, &adm_ctx->device->flags) ? UT_DEGRADED : UT_DEFAULT; - err = timeout_parms_to_priv_skb(adm_ctx->reply_skb, &tp); + err = timeout_parms_to_skb(adm_ctx->reply_skb, &tp); if (err) { nlmsg_free(adm_ctx->reply_skb); adm_ctx->reply_skb = NULL; @@ -4136,7 +4177,7 @@ out: return 0; } -int drbd_adm_start_ov(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_start_ov_doit(struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx = info->user_ptr[0]; struct drbd_device *device; @@ -4182,7 +4223,7 @@ out: } -int drbd_adm_new_c_uuid(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_new_c_uuid_doit(struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx = info->user_ptr[0]; struct drbd_device *device; @@ -4285,7 +4326,7 @@ static void resource_to_info(struct resource_info *info, info->res_susp_fen = resource->susp_fen; } -int drbd_adm_new_resource(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_new_resource_doit(struct sk_buff *skb, struct genl_info *info) { struct drbd_connection *connection; struct drbd_config_context *adm_ctx = info->user_ptr[0]; @@ -4348,7 +4389,7 @@ static void device_to_info(struct device_info *info, } -int drbd_adm_new_minor(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_new_minor_doit(struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx = info->user_ptr[0]; struct drbd_genlmsghdr *dh = genl_info_userhdr(info); @@ -4455,7 +4496,7 @@ static enum drbd_ret_code adm_del_minor(struct drbd_device *device) return ERR_MINOR_CONFIGURED; } -int drbd_adm_del_minor(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_del_minor_doit(struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx = info->user_ptr[0]; enum drbd_ret_code retcode; @@ -4504,7 +4545,7 @@ static int adm_del_resource(struct drbd_resource *resource) return NO_ERROR; } -int drbd_adm_down(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_down_doit(struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx = info->user_ptr[0]; struct drbd_resource *resource; @@ -4567,7 +4608,7 @@ finish: return 0; } -int drbd_adm_del_resource(struct sk_buff *skb, struct genl_info *info) +int drbd_nl_del_resource_doit(struct sk_buff *skb, struct genl_info *info) { struct drbd_config_context *adm_ctx = info->user_ptr[0]; struct drbd_resource *resource; @@ -4601,7 +4642,7 @@ void drbd_bcast_event(struct drbd_device *device, const struct sib_info *sib) goto failed; err = -EMSGSIZE; - d_out = genlmsg_put(msg, 0, seq, &drbd_genl_family, 0, DRBD_EVENT); + d_out = genlmsg_put(msg, 0, seq, &drbd_nl_family, 0, DRBD_ADM_EVENT); if (!d_out) /* cannot happen, but anyways. */ goto nla_put_failure; d_out->minor = device_to_minor(device); @@ -4632,7 +4673,7 @@ static int nla_put_notification_header(struct sk_buff *msg, .nh_type = type, }; - return drbd_notification_header_to_skb(msg, &nh, true); + return drbd_notification_header_to_skb(msg, &nh); } int notify_resource_state(struct sk_buff *skb, @@ -4656,7 +4697,7 @@ int notify_resource_state(struct sk_buff *skb, } err = -EMSGSIZE; - dh = genlmsg_put(skb, 0, seq, &drbd_genl_family, 0, DRBD_RESOURCE_STATE); + dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_RESOURCE_STATE); if (!dh) goto nla_put_failure; dh->minor = -1U; @@ -4664,10 +4705,10 @@ int notify_resource_state(struct sk_buff *skb, if (nla_put_drbd_cfg_context(skb, resource, NULL, NULL) || nla_put_notification_header(skb, type) || ((type & ~NOTIFY_FLAGS) != NOTIFY_DESTROY && - resource_info_to_skb(skb, resource_info, true))) + resource_info_to_skb(skb, resource_info))) goto nla_put_failure; resource_statistics.res_stat_write_ordering = resource->write_ordering; - err = resource_statistics_to_skb(skb, &resource_statistics, !capable(CAP_SYS_ADMIN)); + err = resource_statistics_to_skb(skb, &resource_statistics); if (err) goto nla_put_failure; genlmsg_end(skb, dh); @@ -4708,7 +4749,7 @@ int notify_device_state(struct sk_buff *skb, } err = -EMSGSIZE; - dh = genlmsg_put(skb, 0, seq, &drbd_genl_family, 0, DRBD_DEVICE_STATE); + dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_DEVICE_STATE); if (!dh) goto nla_put_failure; dh->minor = device->minor; @@ -4716,10 +4757,10 @@ int notify_device_state(struct sk_buff *skb, if (nla_put_drbd_cfg_context(skb, device->resource, NULL, device) || nla_put_notification_header(skb, type) || ((type & ~NOTIFY_FLAGS) != NOTIFY_DESTROY && - device_info_to_skb(skb, device_info, true))) + device_info_to_skb(skb, device_info))) goto nla_put_failure; device_to_statistics(&device_statistics, device); - device_statistics_to_skb(skb, &device_statistics, !capable(CAP_SYS_ADMIN)); + device_statistics_to_skb(skb, &device_statistics); genlmsg_end(skb, dh); if (multicast) { err = drbd_genl_multicast_events(skb, GFP_NOWAIT); @@ -4758,7 +4799,7 @@ int notify_connection_state(struct sk_buff *skb, } err = -EMSGSIZE; - dh = genlmsg_put(skb, 0, seq, &drbd_genl_family, 0, DRBD_CONNECTION_STATE); + dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_CONNECTION_STATE); if (!dh) goto nla_put_failure; dh->minor = -1U; @@ -4766,10 +4807,10 @@ int notify_connection_state(struct sk_buff *skb, if (nla_put_drbd_cfg_context(skb, connection->resource, connection, NULL) || nla_put_notification_header(skb, type) || ((type & ~NOTIFY_FLAGS) != NOTIFY_DESTROY && - connection_info_to_skb(skb, connection_info, true))) + connection_info_to_skb(skb, connection_info))) goto nla_put_failure; connection_statistics.conn_congested = test_bit(NET_CONGESTED, &connection->flags); - connection_statistics_to_skb(skb, &connection_statistics, !capable(CAP_SYS_ADMIN)); + connection_statistics_to_skb(skb, &connection_statistics); genlmsg_end(skb, dh); if (multicast) { err = drbd_genl_multicast_events(skb, GFP_NOWAIT); @@ -4809,7 +4850,7 @@ int notify_peer_device_state(struct sk_buff *skb, } err = -EMSGSIZE; - dh = genlmsg_put(skb, 0, seq, &drbd_genl_family, 0, DRBD_PEER_DEVICE_STATE); + dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_PEER_DEVICE_STATE); if (!dh) goto nla_put_failure; dh->minor = -1U; @@ -4817,10 +4858,10 @@ int notify_peer_device_state(struct sk_buff *skb, if (nla_put_drbd_cfg_context(skb, resource, peer_device->connection, peer_device->device) || nla_put_notification_header(skb, type) || ((type & ~NOTIFY_FLAGS) != NOTIFY_DESTROY && - peer_device_info_to_skb(skb, peer_device_info, true))) + peer_device_info_to_skb(skb, peer_device_info))) goto nla_put_failure; peer_device_to_statistics(&peer_device_statistics, peer_device); - peer_device_statistics_to_skb(skb, &peer_device_statistics, !capable(CAP_SYS_ADMIN)); + peer_device_statistics_to_skb(skb, &peer_device_statistics); genlmsg_end(skb, dh); if (multicast) { err = drbd_genl_multicast_events(skb, GFP_NOWAIT); @@ -4859,7 +4900,7 @@ void notify_helper(enum drbd_notification_type type, goto fail; err = -EMSGSIZE; - dh = genlmsg_put(skb, 0, seq, &drbd_genl_family, 0, DRBD_HELPER); + dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_HELPER); if (!dh) goto fail; dh->minor = device ? device->minor : -1; @@ -4867,7 +4908,7 @@ void notify_helper(enum drbd_notification_type type, mutex_lock(¬ification_mutex); if (nla_put_drbd_cfg_context(skb, resource, connection, device) || nla_put_notification_header(skb, type) || - drbd_helper_info_to_skb(skb, &helper_info, true)) + drbd_helper_info_to_skb(skb, &helper_info)) goto unlock_fail; genlmsg_end(skb, dh); err = drbd_genl_multicast_events(skb, GFP_NOWAIT); @@ -4892,7 +4933,7 @@ static int notify_initial_state_done(struct sk_buff *skb, unsigned int seq) int err; err = -EMSGSIZE; - dh = genlmsg_put(skb, 0, seq, &drbd_genl_family, 0, DRBD_INITIAL_STATE_DONE); + dh = genlmsg_put(skb, 0, seq, &drbd_nl_family, 0, DRBD_ADM_INITIAL_STATE_DONE); if (!dh) goto nla_put_failure; dh->minor = -1U; @@ -4987,7 +5028,7 @@ out: return skb->len; } -int drbd_adm_get_initial_state(struct sk_buff *skb, struct netlink_callback *cb) +int drbd_nl_get_initial_state_dumpit(struct sk_buff *skb, struct netlink_callback *cb) { struct drbd_resource *resource; LIST_HEAD(head); @@ -5035,3 +5076,20 @@ int drbd_adm_get_initial_state(struct sk_buff *skb, struct netlink_callback *cb) cb->args[2] = cb->nlh->nlmsg_seq; return get_initial_state(skb, cb); } + +static const struct genl_multicast_group drbd_nl_mcgrps[] = { + [DRBD_NLGRP_EVENTS] = { .name = "events", }, +}; + +struct genl_family drbd_nl_family __ro_after_init = { + .name = "drbd", + .version = DRBD_FAMILY_VERSION, + .hdrsize = NLA_ALIGN(sizeof(struct drbd_genlmsghdr)), + .split_ops = drbd_nl_ops, + .n_split_ops = ARRAY_SIZE(drbd_nl_ops), + .mcgrps = drbd_nl_mcgrps, + .n_mcgrps = ARRAY_SIZE(drbd_nl_mcgrps), + .resv_start_op = 42, + .module = THIS_MODULE, + .netnsok = true, +}; diff --git a/drivers/block/drbd/drbd_nl_gen.c b/drivers/block/drbd/drbd_nl_gen.c new file mode 100644 index 000000000000..fb44b948cec8 --- /dev/null +++ b/drivers/block/drbd/drbd_nl_gen.c @@ -0,0 +1,2606 @@ +// SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) + +#include <net/netlink.h> +#include <net/genetlink.h> + +#include <linux/kernel.h> +#include <linux/slab.h> +#include "drbd_nl_gen.h" + +#include <uapi/linux/drbd_genl.h> +#include <linux/drbd.h> +#include <linux/drbd_limits.h> + +/* Common nested types */ +const struct nla_policy drbd_connection_info_nl_policy[DRBD_A_CONNECTION_INFO_CONN_ROLE + 1] = { + [DRBD_A_CONNECTION_INFO_CONN_CONNECTION_STATE] = { .type = NLA_U32, }, + [DRBD_A_CONNECTION_INFO_CONN_ROLE] = { .type = NLA_U32, }, +}; + +const struct nla_policy drbd_connection_statistics_nl_policy[DRBD_A_CONNECTION_STATISTICS_CONN_CONGESTED + 1] = { + [DRBD_A_CONNECTION_STATISTICS_CONN_CONGESTED] = { .type = NLA_U8, }, +}; + +const struct nla_policy drbd_detach_parms_nl_policy[DRBD_A_DETACH_PARMS_FORCE_DETACH + 1] = { + [DRBD_A_DETACH_PARMS_FORCE_DETACH] = { .type = NLA_U8, }, +}; + +const struct nla_policy drbd_device_info_nl_policy[DRBD_A_DEVICE_INFO_DEV_DISK_STATE + 1] = { + [DRBD_A_DEVICE_INFO_DEV_DISK_STATE] = { .type = NLA_U32, }, +}; + +const struct nla_policy drbd_device_statistics_nl_policy[DRBD_A_DEVICE_STATISTICS_HISTORY_UUIDS + 1] = { + [DRBD_A_DEVICE_STATISTICS_DEV_SIZE] = { .type = NLA_U64, }, + [DRBD_A_DEVICE_STATISTICS_DEV_READ] = { .type = NLA_U64, }, + [DRBD_A_DEVICE_STATISTICS_DEV_WRITE] = { .type = NLA_U64, }, + [DRBD_A_DEVICE_STATISTICS_DEV_AL_WRITES] = { .type = NLA_U64, }, + [DRBD_A_DEVICE_STATISTICS_DEV_BM_WRITES] = { .type = NLA_U64, }, + [DRBD_A_DEVICE_STATISTICS_DEV_UPPER_PENDING] = { .type = NLA_U32, }, + [DRBD_A_DEVICE_STATISTICS_DEV_LOWER_PENDING] = { .type = NLA_U32, }, + [DRBD_A_DEVICE_STATISTICS_DEV_UPPER_BLOCKED] = { .type = NLA_U8, }, + [DRBD_A_DEVICE_STATISTICS_DEV_LOWER_BLOCKED] = { .type = NLA_U8, }, + [DRBD_A_DEVICE_STATISTICS_DEV_AL_SUSPENDED] = { .type = NLA_U8, }, + [DRBD_A_DEVICE_STATISTICS_DEV_EXPOSED_DATA_UUID] = { .type = NLA_U64, }, + [DRBD_A_DEVICE_STATISTICS_DEV_CURRENT_UUID] = { .type = NLA_U64, }, + [DRBD_A_DEVICE_STATISTICS_DEV_DISK_FLAGS] = { .type = NLA_U32, }, + [DRBD_A_DEVICE_STATISTICS_HISTORY_UUIDS] = NLA_POLICY_MAX_LEN(DRBD_NL_HISTORY_UUIDS_SIZE), +}; + +const struct nla_policy drbd_disconnect_parms_nl_policy[DRBD_A_DISCONNECT_PARMS_FORCE_DISCONNECT + 1] = { + [DRBD_A_DISCONNECT_PARMS_FORCE_DISCONNECT] = { .type = NLA_U8, }, +}; + +const struct nla_policy drbd_disk_conf_nl_policy[DRBD_A_DISK_CONF_DISABLE_WRITE_SAME + 1] = { + [DRBD_A_DISK_CONF_BACKING_DEV] = { .type = NLA_NUL_STRING, .len = 128, }, + [DRBD_A_DISK_CONF_META_DEV] = { .type = NLA_NUL_STRING, .len = 128, }, + [DRBD_A_DISK_CONF_META_DEV_IDX] = { .type = NLA_U32, }, + [DRBD_A_DISK_CONF_DISK_SIZE] = { .type = NLA_U64, }, + [DRBD_A_DISK_CONF_MAX_BIO_BVECS] = { .type = NLA_U32, }, + [DRBD_A_DISK_CONF_ON_IO_ERROR] = { .type = NLA_U32, }, + [DRBD_A_DISK_CONF_FENCING] = { .type = NLA_U32, }, + [DRBD_A_DISK_CONF_RESYNC_RATE] = { .type = NLA_U32, }, + [DRBD_A_DISK_CONF_RESYNC_AFTER] = { .type = NLA_U32, }, + [DRBD_A_DISK_CONF_AL_EXTENTS] = { .type = NLA_U32, }, + [DRBD_A_DISK_CONF_C_PLAN_AHEAD] = { .type = NLA_U32, }, + [DRBD_A_DISK_CONF_C_DELAY_TARGET] = { .type = NLA_U32, }, + [DRBD_A_DISK_CONF_C_FILL_TARGET] = { .type = NLA_U32, }, + [DRBD_A_DISK_CONF_C_MAX_RATE] = { .type = NLA_U32, }, + [DRBD_A_DISK_CONF_C_MIN_RATE] = { .type = NLA_U32, }, + [DRBD_A_DISK_CONF_DISK_BARRIER] = { .type = NLA_U8, }, + [DRBD_A_DISK_CONF_DISK_FLUSHES] = { .type = NLA_U8, }, + [DRBD_A_DISK_CONF_DISK_DRAIN] = { .type = NLA_U8, }, + [DRBD_A_DISK_CONF_MD_FLUSHES] = { .type = NLA_U8, }, + [DRBD_A_DISK_CONF_DISK_TIMEOUT] = { .type = NLA_U32, }, + [DRBD_A_DISK_CONF_READ_BALANCING] = { .type = NLA_U32, }, + [DRBD_A_DISK_CONF_AL_UPDATES] = { .type = NLA_U8, }, + [DRBD_A_DISK_CONF_DISCARD_ZEROES_IF_ALIGNED] = { .type = NLA_U8, }, + [DRBD_A_DISK_CONF_RS_DISCARD_GRANULARITY] = { .type = NLA_U32, }, + [DRBD_A_DISK_CONF_DISABLE_WRITE_SAME] = { .type = NLA_U8, }, +}; + +const struct nla_policy drbd_drbd_cfg_context_nl_policy[DRBD_A_DRBD_CFG_CONTEXT_CTX_PEER_ADDR + 1] = { + [DRBD_A_DRBD_CFG_CONTEXT_CTX_VOLUME] = { .type = NLA_U32, }, + [DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME] = { .type = NLA_NUL_STRING, .len = 128, }, + [DRBD_A_DRBD_CFG_CONTEXT_CTX_MY_ADDR] = NLA_POLICY_MAX_LEN(128), + [DRBD_A_DRBD_CFG_CONTEXT_CTX_PEER_ADDR] = NLA_POLICY_MAX_LEN(128), +}; + +const struct nla_policy drbd_net_conf_nl_policy[DRBD_A_NET_CONF_SOCK_CHECK_TIMEO + 1] = { + [DRBD_A_NET_CONF_SHARED_SECRET] = { .type = NLA_NUL_STRING, .len = SHARED_SECRET_MAX, }, + [DRBD_A_NET_CONF_CRAM_HMAC_ALG] = { .type = NLA_NUL_STRING, .len = SHARED_SECRET_MAX, }, + [DRBD_A_NET_CONF_INTEGRITY_ALG] = { .type = NLA_NUL_STRING, .len = SHARED_SECRET_MAX, }, + [DRBD_A_NET_CONF_VERIFY_ALG] = { .type = NLA_NUL_STRING, .len = SHARED_SECRET_MAX, }, + [DRBD_A_NET_CONF_CSUMS_ALG] = { .type = NLA_NUL_STRING, .len = SHARED_SECRET_MAX, }, + [DRBD_A_NET_CONF_WIRE_PROTOCOL] = { .type = NLA_U32, }, + [DRBD_A_NET_CONF_CONNECT_INT] = { .type = NLA_U32, }, + [DRBD_A_NET_CONF_TIMEOUT] = { .type = NLA_U32, }, + [DRBD_A_NET_CONF_PING_INT] = { .type = NLA_U32, }, + [DRBD_A_NET_CONF_PING_TIMEO] = { .type = NLA_U32, }, + [DRBD_A_NET_CONF_SNDBUF_SIZE] = { .type = NLA_U32, }, + [DRBD_A_NET_CONF_RCVBUF_SIZE] = { .type = NLA_U32, }, + [DRBD_A_NET_CONF_KO_COUNT] = { .type = NLA_U32, }, + [DRBD_A_NET_CONF_MAX_BUFFERS] = { .type = NLA_U32, }, + [DRBD_A_NET_CONF_MAX_EPOCH_SIZE] = { .type = NLA_U32, }, + [DRBD_A_NET_CONF_UNPLUG_WATERMARK] = { .type = NLA_U32, }, + [DRBD_A_NET_CONF_AFTER_SB_0P] = { .type = NLA_U32, }, + [DRBD_A_NET_CONF_AFTER_SB_1P] = { .type = NLA_U32, }, + [DRBD_A_NET_CONF_AFTER_SB_2P] = { .type = NLA_U32, }, + [DRBD_A_NET_CONF_RR_CONFLICT] = { .type = NLA_U32, }, + [DRBD_A_NET_CONF_ON_CONGESTION] = { .type = NLA_U32, }, + [DRBD_A_NET_CONF_CONG_FILL] = { .type = NLA_U32, }, + [DRBD_A_NET_CONF_CONG_EXTENTS] = { .type = NLA_U32, }, + [DRBD_A_NET_CONF_TWO_PRIMARIES] = { .type = NLA_U8, }, + [DRBD_A_NET_CONF_DISCARD_MY_DATA] = { .type = NLA_U8, }, + [DRBD_A_NET_CONF_TCP_CORK] = { .type = NLA_U8, }, + [DRBD_A_NET_CONF_ALWAYS_ASBP] = { .type = NLA_U8, }, + [DRBD_A_NET_CONF_TENTATIVE] = { .type = NLA_U8, }, + [DRBD_A_NET_CONF_USE_RLE] = { .type = NLA_U8, }, + [DRBD_A_NET_CONF_CSUMS_AFTER_CRASH_ONLY] = { .type = NLA_U8, }, + [DRBD_A_NET_CONF_SOCK_CHECK_TIMEO] = { .type = NLA_U32, }, +}; + +const struct nla_policy drbd_new_c_uuid_parms_nl_policy[DRBD_A_NEW_C_UUID_PARMS_CLEAR_BM + 1] = { + [DRBD_A_NEW_C_UUID_PARMS_CLEAR_BM] = { .type = NLA_U8, }, +}; + +const struct nla_policy drbd_peer_device_info_nl_policy[DRBD_A_PEER_DEVICE_INFO_PEER_RESYNC_SUSP_DEPENDENCY + 1] = { + [DRBD_A_PEER_DEVICE_INFO_PEER_REPL_STATE] = { .type = NLA_U32, }, + [DRBD_A_PEER_DEVICE_INFO_PEER_DISK_STATE] = { .type = NLA_U32, }, + [DRBD_A_PEER_DEVICE_INFO_PEER_RESYNC_SUSP_USER] = { .type = NLA_U32, }, + [DRBD_A_PEER_DEVICE_INFO_PEER_RESYNC_SUSP_PEER] = { .type = NLA_U32, }, + [DRBD_A_PEER_DEVICE_INFO_PEER_RESYNC_SUSP_DEPENDENCY] = { .type = NLA_U32, }, +}; + +const struct nla_policy drbd_peer_device_statistics_nl_policy[DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_FLAGS + 1] = { + [DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_RECEIVED] = { .type = NLA_U64, }, + [DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_SENT] = { .type = NLA_U64, }, + [DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_PENDING] = { .type = NLA_U32, }, + [DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_UNACKED] = { .type = NLA_U32, }, + [DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_OUT_OF_SYNC] = { .type = NLA_U64, }, + [DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_RESYNC_FAILED] = { .type = NLA_U64, }, + [DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_BITMAP_UUID] = { .type = NLA_U64, }, + [DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_FLAGS] = { .type = NLA_U32, }, +}; + +const struct nla_policy drbd_res_opts_nl_policy[DRBD_A_RES_OPTS_ON_NO_DATA + 1] = { + [DRBD_A_RES_OPTS_CPU_MASK] = { .type = NLA_NUL_STRING, .len = DRBD_CPU_MASK_SIZE, }, + [DRBD_A_RES_OPTS_ON_NO_DATA] = { .type = NLA_U32, }, +}; + +const struct nla_policy drbd_resize_parms_nl_policy[DRBD_A_RESIZE_PARMS_AL_STRIPE_SIZE + 1] = { + [DRBD_A_RESIZE_PARMS_RESIZE_SIZE] = { .type = NLA_U64, }, + [DRBD_A_RESIZE_PARMS_RESIZE_FORCE] = { .type = NLA_U8, }, + [DRBD_A_RESIZE_PARMS_NO_RESYNC] = { .type = NLA_U8, }, + [DRBD_A_RESIZE_PARMS_AL_STRIPES] = { .type = NLA_U32, }, + [DRBD_A_RESIZE_PARMS_AL_STRIPE_SIZE] = { .type = NLA_U32, }, +}; + +const struct nla_policy drbd_resource_info_nl_policy[DRBD_A_RESOURCE_INFO_RES_SUSP_FEN + 1] = { + [DRBD_A_RESOURCE_INFO_RES_ROLE] = { .type = NLA_U32, }, + [DRBD_A_RESOURCE_INFO_RES_SUSP] = { .type = NLA_U8, }, + [DRBD_A_RESOURCE_INFO_RES_SUSP_NOD] = { .type = NLA_U8, }, + [DRBD_A_RESOURCE_INFO_RES_SUSP_FEN] = { .type = NLA_U8, }, +}; + +const struct nla_policy drbd_resource_statistics_nl_policy[DRBD_A_RESOURCE_STATISTICS_RES_STAT_WRITE_ORDERING + 1] = { + [DRBD_A_RESOURCE_STATISTICS_RES_STAT_WRITE_ORDERING] = { .type = NLA_U32, }, +}; + +const struct nla_policy drbd_set_role_parms_nl_policy[DRBD_A_SET_ROLE_PARMS_ASSUME_UPTODATE + 1] = { + [DRBD_A_SET_ROLE_PARMS_ASSUME_UPTODATE] = { .type = NLA_U8, }, +}; + +const struct nla_policy drbd_start_ov_parms_nl_policy[DRBD_A_START_OV_PARMS_OV_STOP_SECTOR + 1] = { + [DRBD_A_START_OV_PARMS_OV_START_SECTOR] = { .type = NLA_U64, }, + [DRBD_A_START_OV_PARMS_OV_STOP_SECTOR] = { .type = NLA_U64, }, +}; + +/* DRBD_ADM_GET_STATUS - do */ +static const struct nla_policy drbd_get_status_do_nl_policy[DRBD_NLA_CFG_CONTEXT + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), +}; + +/* DRBD_ADM_GET_STATUS - dump */ +static const struct nla_policy drbd_get_status_dump_nl_policy[DRBD_NLA_CFG_CONTEXT + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), +}; + +/* DRBD_ADM_NEW_MINOR - do */ +static const struct nla_policy drbd_new_minor_nl_policy[DRBD_NLA_CFG_CONTEXT + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), +}; + +/* DRBD_ADM_DEL_MINOR - do */ +static const struct nla_policy drbd_del_minor_nl_policy[DRBD_NLA_CFG_CONTEXT + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), +}; + +/* DRBD_ADM_NEW_RESOURCE - do */ +static const struct nla_policy drbd_new_resource_nl_policy[DRBD_NLA_RESOURCE_OPTS + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), + [DRBD_NLA_RESOURCE_OPTS] = NLA_POLICY_NESTED(drbd_res_opts_nl_policy), +}; + +/* DRBD_ADM_DEL_RESOURCE - do */ +static const struct nla_policy drbd_del_resource_nl_policy[DRBD_NLA_CFG_CONTEXT + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), +}; + +/* DRBD_ADM_RESOURCE_OPTS - do */ +static const struct nla_policy drbd_resource_opts_nl_policy[DRBD_NLA_RESOURCE_OPTS + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), + [DRBD_NLA_RESOURCE_OPTS] = NLA_POLICY_NESTED(drbd_res_opts_nl_policy), +}; + +/* DRBD_ADM_CONNECT - do */ +static const struct nla_policy drbd_connect_nl_policy[DRBD_NLA_NET_CONF + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), + [DRBD_NLA_NET_CONF] = NLA_POLICY_NESTED(drbd_net_conf_nl_policy), +}; + +/* DRBD_ADM_DISCONNECT - do */ +static const struct nla_policy drbd_disconnect_nl_policy[DRBD_NLA_DISCONNECT_PARMS + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), + [DRBD_NLA_DISCONNECT_PARMS] = NLA_POLICY_NESTED(drbd_disconnect_parms_nl_policy), +}; + +/* DRBD_ADM_ATTACH - do */ +static const struct nla_policy drbd_attach_nl_policy[DRBD_NLA_DISK_CONF + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), + [DRBD_NLA_DISK_CONF] = NLA_POLICY_NESTED(drbd_disk_conf_nl_policy), +}; + +/* DRBD_ADM_RESIZE - do */ +static const struct nla_policy drbd_resize_nl_policy[DRBD_NLA_RESIZE_PARMS + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), + [DRBD_NLA_RESIZE_PARMS] = NLA_POLICY_NESTED(drbd_resize_parms_nl_policy), +}; + +/* DRBD_ADM_PRIMARY - do */ +static const struct nla_policy drbd_primary_nl_policy[DRBD_NLA_SET_ROLE_PARMS + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), + [DRBD_NLA_SET_ROLE_PARMS] = NLA_POLICY_NESTED(drbd_set_role_parms_nl_policy), +}; + +/* DRBD_ADM_SECONDARY - do */ +static const struct nla_policy drbd_secondary_nl_policy[DRBD_NLA_SET_ROLE_PARMS + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), + [DRBD_NLA_SET_ROLE_PARMS] = NLA_POLICY_NESTED(drbd_set_role_parms_nl_policy), +}; + +/* DRBD_ADM_NEW_C_UUID - do */ +static const struct nla_policy drbd_new_c_uuid_nl_policy[DRBD_NLA_NEW_C_UUID_PARMS + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), + [DRBD_NLA_NEW_C_UUID_PARMS] = NLA_POLICY_NESTED(drbd_new_c_uuid_parms_nl_policy), +}; + +/* DRBD_ADM_START_OV - do */ +static const struct nla_policy drbd_start_ov_nl_policy[DRBD_NLA_START_OV_PARMS + 1] = { + [DRBD_NLA_START_OV_PARMS] = NLA_POLICY_NESTED(drbd_start_ov_parms_nl_policy), +}; + +/* DRBD_ADM_DETACH - do */ +static const struct nla_policy drbd_detach_nl_policy[DRBD_NLA_DETACH_PARMS + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), + [DRBD_NLA_DETACH_PARMS] = NLA_POLICY_NESTED(drbd_detach_parms_nl_policy), +}; + +/* DRBD_ADM_INVALIDATE - do */ +static const struct nla_policy drbd_invalidate_nl_policy[DRBD_NLA_CFG_CONTEXT + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), +}; + +/* DRBD_ADM_INVAL_PEER - do */ +static const struct nla_policy drbd_inval_peer_nl_policy[DRBD_NLA_CFG_CONTEXT + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), +}; + +/* DRBD_ADM_PAUSE_SYNC - do */ +static const struct nla_policy drbd_pause_sync_nl_policy[DRBD_NLA_CFG_CONTEXT + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), +}; + +/* DRBD_ADM_RESUME_SYNC - do */ +static const struct nla_policy drbd_resume_sync_nl_policy[DRBD_NLA_CFG_CONTEXT + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), +}; + +/* DRBD_ADM_SUSPEND_IO - do */ +static const struct nla_policy drbd_suspend_io_nl_policy[DRBD_NLA_CFG_CONTEXT + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), +}; + +/* DRBD_ADM_RESUME_IO - do */ +static const struct nla_policy drbd_resume_io_nl_policy[DRBD_NLA_CFG_CONTEXT + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), +}; + +/* DRBD_ADM_OUTDATE - do */ +static const struct nla_policy drbd_outdate_nl_policy[DRBD_NLA_CFG_CONTEXT + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), +}; + +/* DRBD_ADM_GET_TIMEOUT_TYPE - do */ +static const struct nla_policy drbd_get_timeout_type_nl_policy[DRBD_NLA_CFG_CONTEXT + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), +}; + +/* DRBD_ADM_DOWN - do */ +static const struct nla_policy drbd_down_nl_policy[DRBD_NLA_CFG_CONTEXT + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), +}; + +/* DRBD_ADM_CHG_DISK_OPTS - do */ +static const struct nla_policy drbd_chg_disk_opts_nl_policy[DRBD_NLA_DISK_CONF + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), + [DRBD_NLA_DISK_CONF] = NLA_POLICY_NESTED(drbd_disk_conf_nl_policy), +}; + +/* DRBD_ADM_CHG_NET_OPTS - do */ +static const struct nla_policy drbd_chg_net_opts_nl_policy[DRBD_NLA_NET_CONF + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), + [DRBD_NLA_NET_CONF] = NLA_POLICY_NESTED(drbd_net_conf_nl_policy), +}; + +/* DRBD_ADM_GET_RESOURCES - dump */ +static const struct nla_policy drbd_get_resources_nl_policy[DRBD_NLA_RESOURCE_STATISTICS + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), + [DRBD_NLA_RESOURCE_INFO] = NLA_POLICY_NESTED(drbd_resource_info_nl_policy), + [DRBD_NLA_RESOURCE_STATISTICS] = NLA_POLICY_NESTED(drbd_resource_statistics_nl_policy), +}; + +/* DRBD_ADM_GET_DEVICES - dump */ +static const struct nla_policy drbd_get_devices_nl_policy[DRBD_NLA_DEVICE_STATISTICS + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), + [DRBD_NLA_DEVICE_INFO] = NLA_POLICY_NESTED(drbd_device_info_nl_policy), + [DRBD_NLA_DEVICE_STATISTICS] = NLA_POLICY_NESTED(drbd_device_statistics_nl_policy), +}; + +/* DRBD_ADM_GET_CONNECTIONS - dump */ +static const struct nla_policy drbd_get_connections_nl_policy[DRBD_NLA_CONNECTION_STATISTICS + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), + [DRBD_NLA_CONNECTION_INFO] = NLA_POLICY_NESTED(drbd_connection_info_nl_policy), + [DRBD_NLA_CONNECTION_STATISTICS] = NLA_POLICY_NESTED(drbd_connection_statistics_nl_policy), +}; + +/* DRBD_ADM_GET_PEER_DEVICES - dump */ +static const struct nla_policy drbd_get_peer_devices_nl_policy[DRBD_NLA_PEER_DEVICE_STATISTICS + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), + [DRBD_NLA_PEER_DEVICE_INFO] = NLA_POLICY_NESTED(drbd_peer_device_info_nl_policy), + [DRBD_NLA_PEER_DEVICE_STATISTICS] = NLA_POLICY_NESTED(drbd_peer_device_statistics_nl_policy), +}; + +/* DRBD_ADM_GET_INITIAL_STATE - dump */ +static const struct nla_policy drbd_get_initial_state_nl_policy[DRBD_NLA_CFG_CONTEXT + 1] = { + [DRBD_NLA_CFG_CONTEXT] = NLA_POLICY_NESTED(drbd_drbd_cfg_context_nl_policy), +}; + +/* Ops table for drbd */ +const struct genl_split_ops drbd_nl_ops[32] = { + { + .cmd = DRBD_ADM_GET_STATUS, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_get_status_doit, + .post_doit = drbd_post_doit, + .policy = drbd_get_status_do_nl_policy, + .maxattr = DRBD_NLA_CFG_CONTEXT, + .flags = GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_GET_STATUS, + .dumpit = drbd_nl_get_status_dumpit, + .policy = drbd_get_status_dump_nl_policy, + .maxattr = DRBD_NLA_CFG_CONTEXT, + .flags = GENL_CMD_CAP_DUMP, + }, + { + .cmd = DRBD_ADM_NEW_MINOR, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_new_minor_doit, + .post_doit = drbd_post_doit, + .policy = drbd_new_minor_nl_policy, + .maxattr = DRBD_NLA_CFG_CONTEXT, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_DEL_MINOR, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_del_minor_doit, + .post_doit = drbd_post_doit, + .policy = drbd_del_minor_nl_policy, + .maxattr = DRBD_NLA_CFG_CONTEXT, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_NEW_RESOURCE, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_new_resource_doit, + .post_doit = drbd_post_doit, + .policy = drbd_new_resource_nl_policy, + .maxattr = DRBD_NLA_RESOURCE_OPTS, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_DEL_RESOURCE, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_del_resource_doit, + .post_doit = drbd_post_doit, + .policy = drbd_del_resource_nl_policy, + .maxattr = DRBD_NLA_CFG_CONTEXT, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_RESOURCE_OPTS, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_resource_opts_doit, + .post_doit = drbd_post_doit, + .policy = drbd_resource_opts_nl_policy, + .maxattr = DRBD_NLA_RESOURCE_OPTS, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_CONNECT, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_connect_doit, + .post_doit = drbd_post_doit, + .policy = drbd_connect_nl_policy, + .maxattr = DRBD_NLA_NET_CONF, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_DISCONNECT, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_disconnect_doit, + .post_doit = drbd_post_doit, + .policy = drbd_disconnect_nl_policy, + .maxattr = DRBD_NLA_DISCONNECT_PARMS, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_ATTACH, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_attach_doit, + .post_doit = drbd_post_doit, + .policy = drbd_attach_nl_policy, + .maxattr = DRBD_NLA_DISK_CONF, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_RESIZE, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_resize_doit, + .post_doit = drbd_post_doit, + .policy = drbd_resize_nl_policy, + .maxattr = DRBD_NLA_RESIZE_PARMS, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_PRIMARY, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_primary_doit, + .post_doit = drbd_post_doit, + .policy = drbd_primary_nl_policy, + .maxattr = DRBD_NLA_SET_ROLE_PARMS, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_SECONDARY, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_secondary_doit, + .post_doit = drbd_post_doit, + .policy = drbd_secondary_nl_policy, + .maxattr = DRBD_NLA_SET_ROLE_PARMS, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_NEW_C_UUID, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_new_c_uuid_doit, + .post_doit = drbd_post_doit, + .policy = drbd_new_c_uuid_nl_policy, + .maxattr = DRBD_NLA_NEW_C_UUID_PARMS, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_START_OV, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_start_ov_doit, + .post_doit = drbd_post_doit, + .policy = drbd_start_ov_nl_policy, + .maxattr = DRBD_NLA_START_OV_PARMS, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_DETACH, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_detach_doit, + .post_doit = drbd_post_doit, + .policy = drbd_detach_nl_policy, + .maxattr = DRBD_NLA_DETACH_PARMS, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_INVALIDATE, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_invalidate_doit, + .post_doit = drbd_post_doit, + .policy = drbd_invalidate_nl_policy, + .maxattr = DRBD_NLA_CFG_CONTEXT, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_INVAL_PEER, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_inval_peer_doit, + .post_doit = drbd_post_doit, + .policy = drbd_inval_peer_nl_policy, + .maxattr = DRBD_NLA_CFG_CONTEXT, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_PAUSE_SYNC, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_pause_sync_doit, + .post_doit = drbd_post_doit, + .policy = drbd_pause_sync_nl_policy, + .maxattr = DRBD_NLA_CFG_CONTEXT, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_RESUME_SYNC, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_resume_sync_doit, + .post_doit = drbd_post_doit, + .policy = drbd_resume_sync_nl_policy, + .maxattr = DRBD_NLA_CFG_CONTEXT, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_SUSPEND_IO, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_suspend_io_doit, + .post_doit = drbd_post_doit, + .policy = drbd_suspend_io_nl_policy, + .maxattr = DRBD_NLA_CFG_CONTEXT, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_RESUME_IO, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_resume_io_doit, + .post_doit = drbd_post_doit, + .policy = drbd_resume_io_nl_policy, + .maxattr = DRBD_NLA_CFG_CONTEXT, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_OUTDATE, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_outdate_doit, + .post_doit = drbd_post_doit, + .policy = drbd_outdate_nl_policy, + .maxattr = DRBD_NLA_CFG_CONTEXT, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_GET_TIMEOUT_TYPE, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_get_timeout_type_doit, + .post_doit = drbd_post_doit, + .policy = drbd_get_timeout_type_nl_policy, + .maxattr = DRBD_NLA_CFG_CONTEXT, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_DOWN, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_down_doit, + .post_doit = drbd_post_doit, + .policy = drbd_down_nl_policy, + .maxattr = DRBD_NLA_CFG_CONTEXT, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_CHG_DISK_OPTS, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_chg_disk_opts_doit, + .post_doit = drbd_post_doit, + .policy = drbd_chg_disk_opts_nl_policy, + .maxattr = DRBD_NLA_DISK_CONF, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_CHG_NET_OPTS, + .pre_doit = drbd_pre_doit, + .doit = drbd_nl_chg_net_opts_doit, + .post_doit = drbd_post_doit, + .policy = drbd_chg_net_opts_nl_policy, + .maxattr = DRBD_NLA_NET_CONF, + .flags = GENL_ADMIN_PERM | GENL_CMD_CAP_DO, + }, + { + .cmd = DRBD_ADM_GET_RESOURCES, + .dumpit = drbd_nl_get_resources_dumpit, + .policy = drbd_get_resources_nl_policy, + .maxattr = DRBD_NLA_RESOURCE_STATISTICS, + .flags = GENL_CMD_CAP_DUMP, + }, + { + .cmd = DRBD_ADM_GET_DEVICES, + .dumpit = drbd_nl_get_devices_dumpit, + .done = drbd_adm_dump_devices_done, + .policy = drbd_get_devices_nl_policy, + .maxattr = DRBD_NLA_DEVICE_STATISTICS, + .flags = GENL_CMD_CAP_DUMP, + }, + { + .cmd = DRBD_ADM_GET_CONNECTIONS, + .dumpit = drbd_nl_get_connections_dumpit, + .done = drbd_adm_dump_connections_done, + .policy = drbd_get_connections_nl_policy, + .maxattr = DRBD_NLA_CONNECTION_STATISTICS, + .flags = GENL_CMD_CAP_DUMP, + }, + { + .cmd = DRBD_ADM_GET_PEER_DEVICES, + .dumpit = drbd_nl_get_peer_devices_dumpit, + .done = drbd_adm_dump_peer_devices_done, + .policy = drbd_get_peer_devices_nl_policy, + .maxattr = DRBD_NLA_PEER_DEVICE_STATISTICS, + .flags = GENL_CMD_CAP_DUMP, + }, + { + .cmd = DRBD_ADM_GET_INITIAL_STATE, + .dumpit = drbd_nl_get_initial_state_dumpit, + .policy = drbd_get_initial_state_nl_policy, + .maxattr = DRBD_NLA_CFG_CONTEXT, + .flags = GENL_CMD_CAP_DUMP, + }, +}; + +static const struct genl_multicast_group drbd_nl_mcgrps[] = { + [DRBD_NLGRP_EVENTS] = { "events", }, +}; + +static int __drbd_cfg_context_from_attrs(struct drbd_cfg_context *s, + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + const int maxtype = DRBD_A_DRBD_CFG_CONTEXT_CTX_PEER_ADDR; + struct nlattr *tla = info->attrs[DRBD_NLA_CFG_CONTEXT]; + struct nlattr **ntb; + struct nlattr *nla; + int err = 0; + + if (ret_nested_attribute_table) + *ret_nested_attribute_table = NULL; + if (!tla) + return -ENOMSG; + ntb = kcalloc(DRBD_A_DRBD_CFG_CONTEXT_CTX_PEER_ADDR + 1, sizeof(*ntb), GFP_KERNEL); + if (!ntb) + return -ENOMEM; + err = nla_parse_nested_deprecated(ntb, maxtype, tla, drbd_drbd_cfg_context_nl_policy, NULL); + if (err) + goto out; + + nla = ntb[DRBD_A_DRBD_CFG_CONTEXT_CTX_VOLUME]; + if (nla && s) + s->ctx_volume = nla_get_u32(nla); + + nla = ntb[DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME]; + if (nla && s) + s->ctx_resource_name_len = nla_strscpy(s->ctx_resource_name, nla, 128); + + nla = ntb[DRBD_A_DRBD_CFG_CONTEXT_CTX_MY_ADDR]; + if (nla && s) + s->ctx_my_addr_len = nla_memcpy(s->ctx_my_addr, nla, 128); + + nla = ntb[DRBD_A_DRBD_CFG_CONTEXT_CTX_PEER_ADDR]; + if (nla && s) + s->ctx_peer_addr_len = nla_memcpy(s->ctx_peer_addr, nla, 128); + +out: + if (ret_nested_attribute_table && (!err || err == -ENOMSG)) + *ret_nested_attribute_table = ntb; + else + kfree(ntb); + return err; +} + +int drbd_cfg_context_from_attrs(struct drbd_cfg_context *s, + struct genl_info *info) +{ + return __drbd_cfg_context_from_attrs(s, NULL, info); +} + +int drbd_cfg_context_ntb_from_attrs( + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + return __drbd_cfg_context_from_attrs(NULL, ret_nested_attribute_table, info); +} + +static int __disk_conf_from_attrs(struct disk_conf *s, + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + const int maxtype = DRBD_A_DISK_CONF_DISABLE_WRITE_SAME; + struct nlattr *tla = info->attrs[DRBD_NLA_DISK_CONF]; + struct nlattr **ntb; + struct nlattr *nla; + int err = 0; + + if (ret_nested_attribute_table) + *ret_nested_attribute_table = NULL; + if (!tla) + return -ENOMSG; + ntb = kcalloc(DRBD_A_DISK_CONF_DISABLE_WRITE_SAME + 1, sizeof(*ntb), GFP_KERNEL); + if (!ntb) + return -ENOMEM; + err = nla_parse_nested_deprecated(ntb, maxtype, tla, drbd_disk_conf_nl_policy, NULL); + if (err) + goto out; + + nla = ntb[DRBD_A_DISK_CONF_BACKING_DEV]; + if (nla) { + if (s) + s->backing_dev_len = nla_strscpy(s->backing_dev, nla, 128); + } else { + pr_info("<< missing required attr: backing_dev\n"); + err = -ENOMSG; + } + + nla = ntb[DRBD_A_DISK_CONF_META_DEV]; + if (nla) { + if (s) + s->meta_dev_len = nla_strscpy(s->meta_dev, nla, 128); + } else { + pr_info("<< missing required attr: meta_dev\n"); + err = -ENOMSG; + } + + nla = ntb[DRBD_A_DISK_CONF_META_DEV_IDX]; + if (nla) { + if (s) + s->meta_dev_idx = nla_get_s32(nla); + } else { + pr_info("<< missing required attr: meta_dev_idx\n"); + err = -ENOMSG; + } + + nla = ntb[DRBD_A_DISK_CONF_DISK_SIZE]; + if (nla && s) + s->disk_size = nla_get_u64(nla); + + nla = ntb[DRBD_A_DISK_CONF_MAX_BIO_BVECS]; + if (nla && s) + s->max_bio_bvecs = nla_get_u32(nla); + + nla = ntb[DRBD_A_DISK_CONF_ON_IO_ERROR]; + if (nla && s) + s->on_io_error = nla_get_u32(nla); + + nla = ntb[DRBD_A_DISK_CONF_FENCING]; + if (nla && s) + s->fencing = nla_get_u32(nla); + + nla = ntb[DRBD_A_DISK_CONF_RESYNC_RATE]; + if (nla && s) + s->resync_rate = nla_get_u32(nla); + + nla = ntb[DRBD_A_DISK_CONF_RESYNC_AFTER]; + if (nla && s) + s->resync_after = nla_get_s32(nla); + + nla = ntb[DRBD_A_DISK_CONF_AL_EXTENTS]; + if (nla && s) + s->al_extents = nla_get_u32(nla); + + nla = ntb[DRBD_A_DISK_CONF_C_PLAN_AHEAD]; + if (nla && s) + s->c_plan_ahead = nla_get_u32(nla); + + nla = ntb[DRBD_A_DISK_CONF_C_DELAY_TARGET]; + if (nla && s) + s->c_delay_target = nla_get_u32(nla); + + nla = ntb[DRBD_A_DISK_CONF_C_FILL_TARGET]; + if (nla && s) + s->c_fill_target = nla_get_u32(nla); + + nla = ntb[DRBD_A_DISK_CONF_C_MAX_RATE]; + if (nla && s) + s->c_max_rate = nla_get_u32(nla); + + nla = ntb[DRBD_A_DISK_CONF_C_MIN_RATE]; + if (nla && s) + s->c_min_rate = nla_get_u32(nla); + + nla = ntb[DRBD_A_DISK_CONF_DISK_BARRIER]; + if (nla && s) + s->disk_barrier = nla_get_u8(nla); + + nla = ntb[DRBD_A_DISK_CONF_DISK_FLUSHES]; + if (nla && s) + s->disk_flushes = nla_get_u8(nla); + + nla = ntb[DRBD_A_DISK_CONF_DISK_DRAIN]; + if (nla && s) + s->disk_drain = nla_get_u8(nla); + + nla = ntb[DRBD_A_DISK_CONF_MD_FLUSHES]; + if (nla && s) + s->md_flushes = nla_get_u8(nla); + + nla = ntb[DRBD_A_DISK_CONF_DISK_TIMEOUT]; + if (nla && s) + s->disk_timeout = nla_get_u32(nla); + + nla = ntb[DRBD_A_DISK_CONF_READ_BALANCING]; + if (nla && s) + s->read_balancing = nla_get_u32(nla); + + nla = ntb[DRBD_A_DISK_CONF_AL_UPDATES]; + if (nla && s) + s->al_updates = nla_get_u8(nla); + + nla = ntb[DRBD_A_DISK_CONF_DISCARD_ZEROES_IF_ALIGNED]; + if (nla && s) + s->discard_zeroes_if_aligned = nla_get_u8(nla); + + nla = ntb[DRBD_A_DISK_CONF_RS_DISCARD_GRANULARITY]; + if (nla && s) + s->rs_discard_granularity = nla_get_u32(nla); + + nla = ntb[DRBD_A_DISK_CONF_DISABLE_WRITE_SAME]; + if (nla && s) + s->disable_write_same = nla_get_u8(nla); + +out: + if (ret_nested_attribute_table && (!err || err == -ENOMSG)) + *ret_nested_attribute_table = ntb; + else + kfree(ntb); + return err; +} + +int disk_conf_from_attrs(struct disk_conf *s, + struct genl_info *info) +{ + return __disk_conf_from_attrs(s, NULL, info); +} + +int disk_conf_ntb_from_attrs( + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + return __disk_conf_from_attrs(NULL, ret_nested_attribute_table, info); +} + +static int __res_opts_from_attrs(struct res_opts *s, + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + const int maxtype = DRBD_A_RES_OPTS_ON_NO_DATA; + struct nlattr *tla = info->attrs[DRBD_NLA_RESOURCE_OPTS]; + struct nlattr **ntb; + struct nlattr *nla; + int err = 0; + + if (ret_nested_attribute_table) + *ret_nested_attribute_table = NULL; + if (!tla) + return -ENOMSG; + ntb = kcalloc(DRBD_A_RES_OPTS_ON_NO_DATA + 1, sizeof(*ntb), GFP_KERNEL); + if (!ntb) + return -ENOMEM; + err = nla_parse_nested_deprecated(ntb, maxtype, tla, drbd_res_opts_nl_policy, NULL); + if (err) + goto out; + + nla = ntb[DRBD_A_RES_OPTS_CPU_MASK]; + if (nla && s) + s->cpu_mask_len = nla_strscpy(s->cpu_mask, nla, DRBD_CPU_MASK_SIZE); + + nla = ntb[DRBD_A_RES_OPTS_ON_NO_DATA]; + if (nla && s) + s->on_no_data = nla_get_u32(nla); + +out: + if (ret_nested_attribute_table && (!err || err == -ENOMSG)) + *ret_nested_attribute_table = ntb; + else + kfree(ntb); + return err; +} + +int res_opts_from_attrs(struct res_opts *s, + struct genl_info *info) +{ + return __res_opts_from_attrs(s, NULL, info); +} + +int res_opts_ntb_from_attrs( + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + return __res_opts_from_attrs(NULL, ret_nested_attribute_table, info); +} + +static int __net_conf_from_attrs(struct net_conf *s, + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + const int maxtype = DRBD_A_NET_CONF_SOCK_CHECK_TIMEO; + struct nlattr *tla = info->attrs[DRBD_NLA_NET_CONF]; + struct nlattr **ntb; + struct nlattr *nla; + int err = 0; + + if (ret_nested_attribute_table) + *ret_nested_attribute_table = NULL; + if (!tla) + return -ENOMSG; + ntb = kcalloc(DRBD_A_NET_CONF_SOCK_CHECK_TIMEO + 1, sizeof(*ntb), GFP_KERNEL); + if (!ntb) + return -ENOMEM; + err = nla_parse_nested_deprecated(ntb, maxtype, tla, drbd_net_conf_nl_policy, NULL); + if (err) + goto out; + + nla = ntb[DRBD_A_NET_CONF_SHARED_SECRET]; + if (nla && s) + s->shared_secret_len = nla_strscpy(s->shared_secret, nla, SHARED_SECRET_MAX); + + nla = ntb[DRBD_A_NET_CONF_CRAM_HMAC_ALG]; + if (nla && s) + s->cram_hmac_alg_len = nla_strscpy(s->cram_hmac_alg, nla, SHARED_SECRET_MAX); + + nla = ntb[DRBD_A_NET_CONF_INTEGRITY_ALG]; + if (nla && s) + s->integrity_alg_len = nla_strscpy(s->integrity_alg, nla, SHARED_SECRET_MAX); + + nla = ntb[DRBD_A_NET_CONF_VERIFY_ALG]; + if (nla && s) + s->verify_alg_len = nla_strscpy(s->verify_alg, nla, SHARED_SECRET_MAX); + + nla = ntb[DRBD_A_NET_CONF_CSUMS_ALG]; + if (nla && s) + s->csums_alg_len = nla_strscpy(s->csums_alg, nla, SHARED_SECRET_MAX); + + nla = ntb[DRBD_A_NET_CONF_WIRE_PROTOCOL]; + if (nla && s) + s->wire_protocol = nla_get_u32(nla); + + nla = ntb[DRBD_A_NET_CONF_CONNECT_INT]; + if (nla && s) + s->connect_int = nla_get_u32(nla); + + nla = ntb[DRBD_A_NET_CONF_TIMEOUT]; + if (nla && s) + s->timeout = nla_get_u32(nla); + + nla = ntb[DRBD_A_NET_CONF_PING_INT]; + if (nla && s) + s->ping_int = nla_get_u32(nla); + + nla = ntb[DRBD_A_NET_CONF_PING_TIMEO]; + if (nla && s) + s->ping_timeo = nla_get_u32(nla); + + nla = ntb[DRBD_A_NET_CONF_SNDBUF_SIZE]; + if (nla && s) + s->sndbuf_size = nla_get_u32(nla); + + nla = ntb[DRBD_A_NET_CONF_RCVBUF_SIZE]; + if (nla && s) + s->rcvbuf_size = nla_get_u32(nla); + + nla = ntb[DRBD_A_NET_CONF_KO_COUNT]; + if (nla && s) + s->ko_count = nla_get_u32(nla); + + nla = ntb[DRBD_A_NET_CONF_MAX_BUFFERS]; + if (nla && s) + s->max_buffers = nla_get_u32(nla); + + nla = ntb[DRBD_A_NET_CONF_MAX_EPOCH_SIZE]; + if (nla && s) + s->max_epoch_size = nla_get_u32(nla); + + nla = ntb[DRBD_A_NET_CONF_UNPLUG_WATERMARK]; + if (nla && s) + s->unplug_watermark = nla_get_u32(nla); + + nla = ntb[DRBD_A_NET_CONF_AFTER_SB_0P]; + if (nla && s) + s->after_sb_0p = nla_get_u32(nla); + + nla = ntb[DRBD_A_NET_CONF_AFTER_SB_1P]; + if (nla && s) + s->after_sb_1p = nla_get_u32(nla); + + nla = ntb[DRBD_A_NET_CONF_AFTER_SB_2P]; + if (nla && s) + s->after_sb_2p = nla_get_u32(nla); + + nla = ntb[DRBD_A_NET_CONF_RR_CONFLICT]; + if (nla && s) + s->rr_conflict = nla_get_u32(nla); + + nla = ntb[DRBD_A_NET_CONF_ON_CONGESTION]; + if (nla && s) + s->on_congestion = nla_get_u32(nla); + + nla = ntb[DRBD_A_NET_CONF_CONG_FILL]; + if (nla && s) + s->cong_fill = nla_get_u32(nla); + + nla = ntb[DRBD_A_NET_CONF_CONG_EXTENTS]; + if (nla && s) + s->cong_extents = nla_get_u32(nla); + + nla = ntb[DRBD_A_NET_CONF_TWO_PRIMARIES]; + if (nla && s) + s->two_primaries = nla_get_u8(nla); + + nla = ntb[DRBD_A_NET_CONF_DISCARD_MY_DATA]; + if (nla && s) + s->discard_my_data = nla_get_u8(nla); + + nla = ntb[DRBD_A_NET_CONF_TCP_CORK]; + if (nla && s) + s->tcp_cork = nla_get_u8(nla); + + nla = ntb[DRBD_A_NET_CONF_ALWAYS_ASBP]; + if (nla && s) + s->always_asbp = nla_get_u8(nla); + + nla = ntb[DRBD_A_NET_CONF_TENTATIVE]; + if (nla && s) + s->tentative = nla_get_u8(nla); + + nla = ntb[DRBD_A_NET_CONF_USE_RLE]; + if (nla && s) + s->use_rle = nla_get_u8(nla); + + nla = ntb[DRBD_A_NET_CONF_CSUMS_AFTER_CRASH_ONLY]; + if (nla && s) + s->csums_after_crash_only = nla_get_u8(nla); + + nla = ntb[DRBD_A_NET_CONF_SOCK_CHECK_TIMEO]; + if (nla && s) + s->sock_check_timeo = nla_get_u32(nla); + +out: + if (ret_nested_attribute_table && (!err || err == -ENOMSG)) + *ret_nested_attribute_table = ntb; + else + kfree(ntb); + return err; +} + +int net_conf_from_attrs(struct net_conf *s, + struct genl_info *info) +{ + return __net_conf_from_attrs(s, NULL, info); +} + +int net_conf_ntb_from_attrs( + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + return __net_conf_from_attrs(NULL, ret_nested_attribute_table, info); +} + +static int __set_role_parms_from_attrs(struct set_role_parms *s, + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + const int maxtype = DRBD_A_SET_ROLE_PARMS_ASSUME_UPTODATE; + struct nlattr *tla = info->attrs[DRBD_NLA_SET_ROLE_PARMS]; + struct nlattr **ntb; + struct nlattr *nla; + int err = 0; + + if (ret_nested_attribute_table) + *ret_nested_attribute_table = NULL; + if (!tla) + return -ENOMSG; + ntb = kcalloc(DRBD_A_SET_ROLE_PARMS_ASSUME_UPTODATE + 1, sizeof(*ntb), GFP_KERNEL); + if (!ntb) + return -ENOMEM; + err = nla_parse_nested_deprecated(ntb, maxtype, tla, drbd_set_role_parms_nl_policy, NULL); + if (err) + goto out; + + nla = ntb[DRBD_A_SET_ROLE_PARMS_ASSUME_UPTODATE]; + if (nla && s) + s->assume_uptodate = nla_get_u8(nla); + +out: + if (ret_nested_attribute_table && (!err || err == -ENOMSG)) + *ret_nested_attribute_table = ntb; + else + kfree(ntb); + return err; +} + +int set_role_parms_from_attrs(struct set_role_parms *s, + struct genl_info *info) +{ + return __set_role_parms_from_attrs(s, NULL, info); +} + +int set_role_parms_ntb_from_attrs( + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + return __set_role_parms_from_attrs(NULL, ret_nested_attribute_table, info); +} + +static int __resize_parms_from_attrs(struct resize_parms *s, + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + const int maxtype = DRBD_A_RESIZE_PARMS_AL_STRIPE_SIZE; + struct nlattr *tla = info->attrs[DRBD_NLA_RESIZE_PARMS]; + struct nlattr **ntb; + struct nlattr *nla; + int err = 0; + + if (ret_nested_attribute_table) + *ret_nested_attribute_table = NULL; + if (!tla) + return -ENOMSG; + ntb = kcalloc(DRBD_A_RESIZE_PARMS_AL_STRIPE_SIZE + 1, sizeof(*ntb), GFP_KERNEL); + if (!ntb) + return -ENOMEM; + err = nla_parse_nested_deprecated(ntb, maxtype, tla, drbd_resize_parms_nl_policy, NULL); + if (err) + goto out; + + nla = ntb[DRBD_A_RESIZE_PARMS_RESIZE_SIZE]; + if (nla && s) + s->resize_size = nla_get_u64(nla); + + nla = ntb[DRBD_A_RESIZE_PARMS_RESIZE_FORCE]; + if (nla && s) + s->resize_force = nla_get_u8(nla); + + nla = ntb[DRBD_A_RESIZE_PARMS_NO_RESYNC]; + if (nla && s) + s->no_resync = nla_get_u8(nla); + + nla = ntb[DRBD_A_RESIZE_PARMS_AL_STRIPES]; + if (nla && s) + s->al_stripes = nla_get_u32(nla); + + nla = ntb[DRBD_A_RESIZE_PARMS_AL_STRIPE_SIZE]; + if (nla && s) + s->al_stripe_size = nla_get_u32(nla); + +out: + if (ret_nested_attribute_table && (!err || err == -ENOMSG)) + *ret_nested_attribute_table = ntb; + else + kfree(ntb); + return err; +} + +int resize_parms_from_attrs(struct resize_parms *s, + struct genl_info *info) +{ + return __resize_parms_from_attrs(s, NULL, info); +} + +int resize_parms_ntb_from_attrs( + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + return __resize_parms_from_attrs(NULL, ret_nested_attribute_table, info); +} + +static int __start_ov_parms_from_attrs(struct start_ov_parms *s, + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + const int maxtype = DRBD_A_START_OV_PARMS_OV_STOP_SECTOR; + struct nlattr *tla = info->attrs[DRBD_NLA_START_OV_PARMS]; + struct nlattr **ntb; + struct nlattr *nla; + int err = 0; + + if (ret_nested_attribute_table) + *ret_nested_attribute_table = NULL; + if (!tla) + return -ENOMSG; + ntb = kcalloc(DRBD_A_START_OV_PARMS_OV_STOP_SECTOR + 1, sizeof(*ntb), GFP_KERNEL); + if (!ntb) + return -ENOMEM; + err = nla_parse_nested_deprecated(ntb, maxtype, tla, drbd_start_ov_parms_nl_policy, NULL); + if (err) + goto out; + + nla = ntb[DRBD_A_START_OV_PARMS_OV_START_SECTOR]; + if (nla && s) + s->ov_start_sector = nla_get_u64(nla); + + nla = ntb[DRBD_A_START_OV_PARMS_OV_STOP_SECTOR]; + if (nla && s) + s->ov_stop_sector = nla_get_u64(nla); + +out: + if (ret_nested_attribute_table && (!err || err == -ENOMSG)) + *ret_nested_attribute_table = ntb; + else + kfree(ntb); + return err; +} + +int start_ov_parms_from_attrs(struct start_ov_parms *s, + struct genl_info *info) +{ + return __start_ov_parms_from_attrs(s, NULL, info); +} + +int start_ov_parms_ntb_from_attrs( + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + return __start_ov_parms_from_attrs(NULL, ret_nested_attribute_table, info); +} + +static int __new_c_uuid_parms_from_attrs(struct new_c_uuid_parms *s, + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + const int maxtype = DRBD_A_NEW_C_UUID_PARMS_CLEAR_BM; + struct nlattr *tla = info->attrs[DRBD_NLA_NEW_C_UUID_PARMS]; + struct nlattr **ntb; + struct nlattr *nla; + int err = 0; + + if (ret_nested_attribute_table) + *ret_nested_attribute_table = NULL; + if (!tla) + return -ENOMSG; + ntb = kcalloc(DRBD_A_NEW_C_UUID_PARMS_CLEAR_BM + 1, sizeof(*ntb), GFP_KERNEL); + if (!ntb) + return -ENOMEM; + err = nla_parse_nested_deprecated(ntb, maxtype, tla, drbd_new_c_uuid_parms_nl_policy, NULL); + if (err) + goto out; + + nla = ntb[DRBD_A_NEW_C_UUID_PARMS_CLEAR_BM]; + if (nla && s) + s->clear_bm = nla_get_u8(nla); + +out: + if (ret_nested_attribute_table && (!err || err == -ENOMSG)) + *ret_nested_attribute_table = ntb; + else + kfree(ntb); + return err; +} + +int new_c_uuid_parms_from_attrs(struct new_c_uuid_parms *s, + struct genl_info *info) +{ + return __new_c_uuid_parms_from_attrs(s, NULL, info); +} + +int new_c_uuid_parms_ntb_from_attrs( + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + return __new_c_uuid_parms_from_attrs(NULL, ret_nested_attribute_table, info); +} + +static int __disconnect_parms_from_attrs(struct disconnect_parms *s, + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + const int maxtype = DRBD_A_DISCONNECT_PARMS_FORCE_DISCONNECT; + struct nlattr *tla = info->attrs[DRBD_NLA_DISCONNECT_PARMS]; + struct nlattr **ntb; + struct nlattr *nla; + int err = 0; + + if (ret_nested_attribute_table) + *ret_nested_attribute_table = NULL; + if (!tla) + return -ENOMSG; + ntb = kcalloc(DRBD_A_DISCONNECT_PARMS_FORCE_DISCONNECT + 1, sizeof(*ntb), GFP_KERNEL); + if (!ntb) + return -ENOMEM; + err = nla_parse_nested_deprecated(ntb, maxtype, tla, drbd_disconnect_parms_nl_policy, NULL); + if (err) + goto out; + + nla = ntb[DRBD_A_DISCONNECT_PARMS_FORCE_DISCONNECT]; + if (nla && s) + s->force_disconnect = nla_get_u8(nla); + +out: + if (ret_nested_attribute_table && (!err || err == -ENOMSG)) + *ret_nested_attribute_table = ntb; + else + kfree(ntb); + return err; +} + +int disconnect_parms_from_attrs(struct disconnect_parms *s, + struct genl_info *info) +{ + return __disconnect_parms_from_attrs(s, NULL, info); +} + +int disconnect_parms_ntb_from_attrs( + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + return __disconnect_parms_from_attrs(NULL, ret_nested_attribute_table, info); +} + +static int __detach_parms_from_attrs(struct detach_parms *s, + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + const int maxtype = DRBD_A_DETACH_PARMS_FORCE_DETACH; + struct nlattr *tla = info->attrs[DRBD_NLA_DETACH_PARMS]; + struct nlattr **ntb; + struct nlattr *nla; + int err = 0; + + if (ret_nested_attribute_table) + *ret_nested_attribute_table = NULL; + if (!tla) + return -ENOMSG; + ntb = kcalloc(DRBD_A_DETACH_PARMS_FORCE_DETACH + 1, sizeof(*ntb), GFP_KERNEL); + if (!ntb) + return -ENOMEM; + err = nla_parse_nested_deprecated(ntb, maxtype, tla, drbd_detach_parms_nl_policy, NULL); + if (err) + goto out; + + nla = ntb[DRBD_A_DETACH_PARMS_FORCE_DETACH]; + if (nla && s) + s->force_detach = nla_get_u8(nla); + +out: + if (ret_nested_attribute_table && (!err || err == -ENOMSG)) + *ret_nested_attribute_table = ntb; + else + kfree(ntb); + return err; +} + +int detach_parms_from_attrs(struct detach_parms *s, + struct genl_info *info) +{ + return __detach_parms_from_attrs(s, NULL, info); +} + +int detach_parms_ntb_from_attrs( + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + return __detach_parms_from_attrs(NULL, ret_nested_attribute_table, info); +} + +static int __resource_info_from_attrs(struct resource_info *s, + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + const int maxtype = DRBD_A_RESOURCE_INFO_RES_SUSP_FEN; + struct nlattr *tla = info->attrs[DRBD_NLA_RESOURCE_INFO]; + struct nlattr **ntb; + struct nlattr *nla; + int err = 0; + + if (ret_nested_attribute_table) + *ret_nested_attribute_table = NULL; + if (!tla) + return -ENOMSG; + ntb = kcalloc(DRBD_A_RESOURCE_INFO_RES_SUSP_FEN + 1, sizeof(*ntb), GFP_KERNEL); + if (!ntb) + return -ENOMEM; + err = nla_parse_nested_deprecated(ntb, maxtype, tla, drbd_resource_info_nl_policy, NULL); + if (err) + goto out; + + nla = ntb[DRBD_A_RESOURCE_INFO_RES_ROLE]; + if (nla && s) + s->res_role = nla_get_u32(nla); + + nla = ntb[DRBD_A_RESOURCE_INFO_RES_SUSP]; + if (nla && s) + s->res_susp = nla_get_u8(nla); + + nla = ntb[DRBD_A_RESOURCE_INFO_RES_SUSP_NOD]; + if (nla && s) + s->res_susp_nod = nla_get_u8(nla); + + nla = ntb[DRBD_A_RESOURCE_INFO_RES_SUSP_FEN]; + if (nla && s) + s->res_susp_fen = nla_get_u8(nla); + +out: + if (ret_nested_attribute_table && (!err || err == -ENOMSG)) + *ret_nested_attribute_table = ntb; + else + kfree(ntb); + return err; +} + +int resource_info_from_attrs(struct resource_info *s, + struct genl_info *info) +{ + return __resource_info_from_attrs(s, NULL, info); +} + +int resource_info_ntb_from_attrs( + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + return __resource_info_from_attrs(NULL, ret_nested_attribute_table, info); +} + +static int __device_info_from_attrs(struct device_info *s, + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + const int maxtype = DRBD_A_DEVICE_INFO_DEV_DISK_STATE; + struct nlattr *tla = info->attrs[DRBD_NLA_DEVICE_INFO]; + struct nlattr **ntb; + struct nlattr *nla; + int err = 0; + + if (ret_nested_attribute_table) + *ret_nested_attribute_table = NULL; + if (!tla) + return -ENOMSG; + ntb = kcalloc(DRBD_A_DEVICE_INFO_DEV_DISK_STATE + 1, sizeof(*ntb), GFP_KERNEL); + if (!ntb) + return -ENOMEM; + err = nla_parse_nested_deprecated(ntb, maxtype, tla, drbd_device_info_nl_policy, NULL); + if (err) + goto out; + + nla = ntb[DRBD_A_DEVICE_INFO_DEV_DISK_STATE]; + if (nla && s) + s->dev_disk_state = nla_get_u32(nla); + +out: + if (ret_nested_attribute_table && (!err || err == -ENOMSG)) + *ret_nested_attribute_table = ntb; + else + kfree(ntb); + return err; +} + +int device_info_from_attrs(struct device_info *s, + struct genl_info *info) +{ + return __device_info_from_attrs(s, NULL, info); +} + +int device_info_ntb_from_attrs( + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + return __device_info_from_attrs(NULL, ret_nested_attribute_table, info); +} + +static int __connection_info_from_attrs(struct connection_info *s, + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + const int maxtype = DRBD_A_CONNECTION_INFO_CONN_ROLE; + struct nlattr *tla = info->attrs[DRBD_NLA_CONNECTION_INFO]; + struct nlattr **ntb; + struct nlattr *nla; + int err = 0; + + if (ret_nested_attribute_table) + *ret_nested_attribute_table = NULL; + if (!tla) + return -ENOMSG; + ntb = kcalloc(DRBD_A_CONNECTION_INFO_CONN_ROLE + 1, sizeof(*ntb), GFP_KERNEL); + if (!ntb) + return -ENOMEM; + err = nla_parse_nested_deprecated(ntb, maxtype, tla, drbd_connection_info_nl_policy, NULL); + if (err) + goto out; + + nla = ntb[DRBD_A_CONNECTION_INFO_CONN_CONNECTION_STATE]; + if (nla && s) + s->conn_connection_state = nla_get_u32(nla); + + nla = ntb[DRBD_A_CONNECTION_INFO_CONN_ROLE]; + if (nla && s) + s->conn_role = nla_get_u32(nla); + +out: + if (ret_nested_attribute_table && (!err || err == -ENOMSG)) + *ret_nested_attribute_table = ntb; + else + kfree(ntb); + return err; +} + +int connection_info_from_attrs(struct connection_info *s, + struct genl_info *info) +{ + return __connection_info_from_attrs(s, NULL, info); +} + +int connection_info_ntb_from_attrs( + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + return __connection_info_from_attrs(NULL, ret_nested_attribute_table, info); +} + +static int __peer_device_info_from_attrs(struct peer_device_info *s, + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + const int maxtype = DRBD_A_PEER_DEVICE_INFO_PEER_RESYNC_SUSP_DEPENDENCY; + struct nlattr *tla = info->attrs[DRBD_NLA_PEER_DEVICE_INFO]; + struct nlattr **ntb; + struct nlattr *nla; + int err = 0; + + if (ret_nested_attribute_table) + *ret_nested_attribute_table = NULL; + if (!tla) + return -ENOMSG; + ntb = kcalloc(DRBD_A_PEER_DEVICE_INFO_PEER_RESYNC_SUSP_DEPENDENCY + 1, sizeof(*ntb), GFP_KERNEL); + if (!ntb) + return -ENOMEM; + err = nla_parse_nested_deprecated(ntb, maxtype, tla, drbd_peer_device_info_nl_policy, NULL); + if (err) + goto out; + + nla = ntb[DRBD_A_PEER_DEVICE_INFO_PEER_REPL_STATE]; + if (nla && s) + s->peer_repl_state = nla_get_u32(nla); + + nla = ntb[DRBD_A_PEER_DEVICE_INFO_PEER_DISK_STATE]; + if (nla && s) + s->peer_disk_state = nla_get_u32(nla); + + nla = ntb[DRBD_A_PEER_DEVICE_INFO_PEER_RESYNC_SUSP_USER]; + if (nla && s) + s->peer_resync_susp_user = nla_get_u32(nla); + + nla = ntb[DRBD_A_PEER_DEVICE_INFO_PEER_RESYNC_SUSP_PEER]; + if (nla && s) + s->peer_resync_susp_peer = nla_get_u32(nla); + + nla = ntb[DRBD_A_PEER_DEVICE_INFO_PEER_RESYNC_SUSP_DEPENDENCY]; + if (nla && s) + s->peer_resync_susp_dependency = nla_get_u32(nla); + +out: + if (ret_nested_attribute_table && (!err || err == -ENOMSG)) + *ret_nested_attribute_table = ntb; + else + kfree(ntb); + return err; +} + +int peer_device_info_from_attrs(struct peer_device_info *s, + struct genl_info *info) +{ + return __peer_device_info_from_attrs(s, NULL, info); +} + +int peer_device_info_ntb_from_attrs( + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + return __peer_device_info_from_attrs(NULL, ret_nested_attribute_table, info); +} + +static int __resource_statistics_from_attrs(struct resource_statistics *s, + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + const int maxtype = DRBD_A_RESOURCE_STATISTICS_RES_STAT_WRITE_ORDERING; + struct nlattr *tla = info->attrs[DRBD_NLA_RESOURCE_STATISTICS]; + struct nlattr **ntb; + struct nlattr *nla; + int err = 0; + + if (ret_nested_attribute_table) + *ret_nested_attribute_table = NULL; + if (!tla) + return -ENOMSG; + ntb = kcalloc(DRBD_A_RESOURCE_STATISTICS_RES_STAT_WRITE_ORDERING + 1, sizeof(*ntb), GFP_KERNEL); + if (!ntb) + return -ENOMEM; + err = nla_parse_nested_deprecated(ntb, maxtype, tla, drbd_resource_statistics_nl_policy, NULL); + if (err) + goto out; + + nla = ntb[DRBD_A_RESOURCE_STATISTICS_RES_STAT_WRITE_ORDERING]; + if (nla && s) + s->res_stat_write_ordering = nla_get_u32(nla); + +out: + if (ret_nested_attribute_table && (!err || err == -ENOMSG)) + *ret_nested_attribute_table = ntb; + else + kfree(ntb); + return err; +} + +int resource_statistics_from_attrs(struct resource_statistics *s, + struct genl_info *info) +{ + return __resource_statistics_from_attrs(s, NULL, info); +} + +int resource_statistics_ntb_from_attrs( + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + return __resource_statistics_from_attrs(NULL, ret_nested_attribute_table, info); +} + +static int __device_statistics_from_attrs(struct device_statistics *s, + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + const int maxtype = DRBD_A_DEVICE_STATISTICS_HISTORY_UUIDS; + struct nlattr *tla = info->attrs[DRBD_NLA_DEVICE_STATISTICS]; + struct nlattr **ntb; + struct nlattr *nla; + int err = 0; + + if (ret_nested_attribute_table) + *ret_nested_attribute_table = NULL; + if (!tla) + return -ENOMSG; + ntb = kcalloc(DRBD_A_DEVICE_STATISTICS_HISTORY_UUIDS + 1, sizeof(*ntb), GFP_KERNEL); + if (!ntb) + return -ENOMEM; + err = nla_parse_nested_deprecated(ntb, maxtype, tla, drbd_device_statistics_nl_policy, NULL); + if (err) + goto out; + + nla = ntb[DRBD_A_DEVICE_STATISTICS_DEV_SIZE]; + if (nla && s) + s->dev_size = nla_get_u64(nla); + + nla = ntb[DRBD_A_DEVICE_STATISTICS_DEV_READ]; + if (nla && s) + s->dev_read = nla_get_u64(nla); + + nla = ntb[DRBD_A_DEVICE_STATISTICS_DEV_WRITE]; + if (nla && s) + s->dev_write = nla_get_u64(nla); + + nla = ntb[DRBD_A_DEVICE_STATISTICS_DEV_AL_WRITES]; + if (nla && s) + s->dev_al_writes = nla_get_u64(nla); + + nla = ntb[DRBD_A_DEVICE_STATISTICS_DEV_BM_WRITES]; + if (nla && s) + s->dev_bm_writes = nla_get_u64(nla); + + nla = ntb[DRBD_A_DEVICE_STATISTICS_DEV_UPPER_PENDING]; + if (nla && s) + s->dev_upper_pending = nla_get_u32(nla); + + nla = ntb[DRBD_A_DEVICE_STATISTICS_DEV_LOWER_PENDING]; + if (nla && s) + s->dev_lower_pending = nla_get_u32(nla); + + nla = ntb[DRBD_A_DEVICE_STATISTICS_DEV_UPPER_BLOCKED]; + if (nla && s) + s->dev_upper_blocked = nla_get_u8(nla); + + nla = ntb[DRBD_A_DEVICE_STATISTICS_DEV_LOWER_BLOCKED]; + if (nla && s) + s->dev_lower_blocked = nla_get_u8(nla); + + nla = ntb[DRBD_A_DEVICE_STATISTICS_DEV_AL_SUSPENDED]; + if (nla && s) + s->dev_al_suspended = nla_get_u8(nla); + + nla = ntb[DRBD_A_DEVICE_STATISTICS_DEV_EXPOSED_DATA_UUID]; + if (nla && s) + s->dev_exposed_data_uuid = nla_get_u64(nla); + + nla = ntb[DRBD_A_DEVICE_STATISTICS_DEV_CURRENT_UUID]; + if (nla && s) + s->dev_current_uuid = nla_get_u64(nla); + + nla = ntb[DRBD_A_DEVICE_STATISTICS_DEV_DISK_FLAGS]; + if (nla && s) + s->dev_disk_flags = nla_get_u32(nla); + + nla = ntb[DRBD_A_DEVICE_STATISTICS_HISTORY_UUIDS]; + if (nla && s) + s->history_uuids_len = nla_memcpy(s->history_uuids, nla, DRBD_NL_HISTORY_UUIDS_SIZE); + +out: + if (ret_nested_attribute_table && (!err || err == -ENOMSG)) + *ret_nested_attribute_table = ntb; + else + kfree(ntb); + return err; +} + +int device_statistics_from_attrs(struct device_statistics *s, + struct genl_info *info) +{ + return __device_statistics_from_attrs(s, NULL, info); +} + +int device_statistics_ntb_from_attrs( + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + return __device_statistics_from_attrs(NULL, ret_nested_attribute_table, info); +} + +static int __connection_statistics_from_attrs(struct connection_statistics *s, + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + const int maxtype = DRBD_A_CONNECTION_STATISTICS_CONN_CONGESTED; + struct nlattr *tla = info->attrs[DRBD_NLA_CONNECTION_STATISTICS]; + struct nlattr **ntb; + struct nlattr *nla; + int err = 0; + + if (ret_nested_attribute_table) + *ret_nested_attribute_table = NULL; + if (!tla) + return -ENOMSG; + ntb = kcalloc(DRBD_A_CONNECTION_STATISTICS_CONN_CONGESTED + 1, sizeof(*ntb), GFP_KERNEL); + if (!ntb) + return -ENOMEM; + err = nla_parse_nested_deprecated(ntb, maxtype, tla, drbd_connection_statistics_nl_policy, NULL); + if (err) + goto out; + + nla = ntb[DRBD_A_CONNECTION_STATISTICS_CONN_CONGESTED]; + if (nla && s) + s->conn_congested = nla_get_u8(nla); + +out: + if (ret_nested_attribute_table && (!err || err == -ENOMSG)) + *ret_nested_attribute_table = ntb; + else + kfree(ntb); + return err; +} + +int connection_statistics_from_attrs(struct connection_statistics *s, + struct genl_info *info) +{ + return __connection_statistics_from_attrs(s, NULL, info); +} + +int connection_statistics_ntb_from_attrs( + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + return __connection_statistics_from_attrs(NULL, ret_nested_attribute_table, info); +} + +static int __peer_device_statistics_from_attrs(struct peer_device_statistics *s, + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + const int maxtype = DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_FLAGS; + struct nlattr *tla = info->attrs[DRBD_NLA_PEER_DEVICE_STATISTICS]; + struct nlattr **ntb; + struct nlattr *nla; + int err = 0; + + if (ret_nested_attribute_table) + *ret_nested_attribute_table = NULL; + if (!tla) + return -ENOMSG; + ntb = kcalloc(DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_FLAGS + 1, sizeof(*ntb), GFP_KERNEL); + if (!ntb) + return -ENOMEM; + err = nla_parse_nested_deprecated(ntb, maxtype, tla, drbd_peer_device_statistics_nl_policy, NULL); + if (err) + goto out; + + nla = ntb[DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_RECEIVED]; + if (nla && s) + s->peer_dev_received = nla_get_u64(nla); + + nla = ntb[DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_SENT]; + if (nla && s) + s->peer_dev_sent = nla_get_u64(nla); + + nla = ntb[DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_PENDING]; + if (nla && s) + s->peer_dev_pending = nla_get_u32(nla); + + nla = ntb[DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_UNACKED]; + if (nla && s) + s->peer_dev_unacked = nla_get_u32(nla); + + nla = ntb[DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_OUT_OF_SYNC]; + if (nla && s) + s->peer_dev_out_of_sync = nla_get_u64(nla); + + nla = ntb[DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_RESYNC_FAILED]; + if (nla && s) + s->peer_dev_resync_failed = nla_get_u64(nla); + + nla = ntb[DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_BITMAP_UUID]; + if (nla && s) + s->peer_dev_bitmap_uuid = nla_get_u64(nla); + + nla = ntb[DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_FLAGS]; + if (nla && s) + s->peer_dev_flags = nla_get_u32(nla); + +out: + if (ret_nested_attribute_table && (!err || err == -ENOMSG)) + *ret_nested_attribute_table = ntb; + else + kfree(ntb); + return err; +} + +int peer_device_statistics_from_attrs(struct peer_device_statistics *s, + struct genl_info *info) +{ + return __peer_device_statistics_from_attrs(s, NULL, info); +} + +int peer_device_statistics_ntb_from_attrs( + struct nlattr ***ret_nested_attribute_table, + struct genl_info *info) +{ + return __peer_device_statistics_from_attrs(NULL, ret_nested_attribute_table, info); +} + +int drbd_cfg_reply_to_skb(struct sk_buff *skb, struct drbd_cfg_reply *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_CFG_REPLY); + + if (!tla) + goto nla_put_failure; + + if (nla_put(skb, DRBD_A_DRBD_CFG_REPLY_INFO_TEXT, min_t(int, 0, + s->info_text_len + (s->info_text_len < 0)), s->info_text)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +int drbd_cfg_context_to_skb(struct sk_buff *skb, struct drbd_cfg_context *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_CFG_CONTEXT); + + if (!tla) + goto nla_put_failure; + + if (nla_put_u32(skb, DRBD_A_DRBD_CFG_CONTEXT_CTX_VOLUME, s->ctx_volume)) + goto nla_put_failure; + if (nla_put(skb, DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME, min_t(int, 128, + s->ctx_resource_name_len + (s->ctx_resource_name_len < 128)), s->ctx_resource_name)) + goto nla_put_failure; + if (nla_put(skb, DRBD_A_DRBD_CFG_CONTEXT_CTX_MY_ADDR, min_t(int, 128, + s->ctx_my_addr_len), s->ctx_my_addr)) + goto nla_put_failure; + if (nla_put(skb, DRBD_A_DRBD_CFG_CONTEXT_CTX_PEER_ADDR, min_t(int, 128, + s->ctx_peer_addr_len), s->ctx_peer_addr)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +int disk_conf_to_skb(struct sk_buff *skb, struct disk_conf *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_DISK_CONF); + + if (!tla) + goto nla_put_failure; + + if (nla_put(skb, DRBD_A_DISK_CONF_BACKING_DEV, min_t(int, 128, + s->backing_dev_len + (s->backing_dev_len < 128)), s->backing_dev)) + goto nla_put_failure; + if (nla_put(skb, DRBD_A_DISK_CONF_META_DEV, min_t(int, 128, + s->meta_dev_len + (s->meta_dev_len < 128)), s->meta_dev)) + goto nla_put_failure; + if (nla_put_s32(skb, DRBD_A_DISK_CONF_META_DEV_IDX, s->meta_dev_idx)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_DISK_CONF_DISK_SIZE, s->disk_size, 0)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_DISK_CONF_MAX_BIO_BVECS, s->max_bio_bvecs)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_DISK_CONF_ON_IO_ERROR, s->on_io_error)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_DISK_CONF_FENCING, s->fencing)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_DISK_CONF_RESYNC_RATE, s->resync_rate)) + goto nla_put_failure; + if (nla_put_s32(skb, DRBD_A_DISK_CONF_RESYNC_AFTER, s->resync_after)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_DISK_CONF_AL_EXTENTS, s->al_extents)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_DISK_CONF_C_PLAN_AHEAD, s->c_plan_ahead)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_DISK_CONF_C_DELAY_TARGET, s->c_delay_target)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_DISK_CONF_C_FILL_TARGET, s->c_fill_target)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_DISK_CONF_C_MAX_RATE, s->c_max_rate)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_DISK_CONF_C_MIN_RATE, s->c_min_rate)) + goto nla_put_failure; + if (nla_put_u8(skb, DRBD_A_DISK_CONF_DISK_BARRIER, s->disk_barrier)) + goto nla_put_failure; + if (nla_put_u8(skb, DRBD_A_DISK_CONF_DISK_FLUSHES, s->disk_flushes)) + goto nla_put_failure; + if (nla_put_u8(skb, DRBD_A_DISK_CONF_DISK_DRAIN, s->disk_drain)) + goto nla_put_failure; + if (nla_put_u8(skb, DRBD_A_DISK_CONF_MD_FLUSHES, s->md_flushes)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_DISK_CONF_DISK_TIMEOUT, s->disk_timeout)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_DISK_CONF_READ_BALANCING, s->read_balancing)) + goto nla_put_failure; + if (nla_put_u8(skb, DRBD_A_DISK_CONF_AL_UPDATES, s->al_updates)) + goto nla_put_failure; + if (nla_put_u8(skb, DRBD_A_DISK_CONF_DISCARD_ZEROES_IF_ALIGNED, s->discard_zeroes_if_aligned)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_DISK_CONF_RS_DISCARD_GRANULARITY, s->rs_discard_granularity)) + goto nla_put_failure; + if (nla_put_u8(skb, DRBD_A_DISK_CONF_DISABLE_WRITE_SAME, s->disable_write_same)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +int res_opts_to_skb(struct sk_buff *skb, struct res_opts *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_RESOURCE_OPTS); + + if (!tla) + goto nla_put_failure; + + if (nla_put(skb, DRBD_A_RES_OPTS_CPU_MASK, min_t(int, DRBD_CPU_MASK_SIZE, + s->cpu_mask_len + (s->cpu_mask_len < DRBD_CPU_MASK_SIZE)), s->cpu_mask)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_RES_OPTS_ON_NO_DATA, s->on_no_data)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +int net_conf_to_skb(struct sk_buff *skb, struct net_conf *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_NET_CONF); + + if (!tla) + goto nla_put_failure; + + if (nla_put(skb, DRBD_A_NET_CONF_SHARED_SECRET, min_t(int, SHARED_SECRET_MAX, + s->shared_secret_len + (s->shared_secret_len < SHARED_SECRET_MAX)), s->shared_secret)) + goto nla_put_failure; + if (nla_put(skb, DRBD_A_NET_CONF_CRAM_HMAC_ALG, min_t(int, SHARED_SECRET_MAX, + s->cram_hmac_alg_len + (s->cram_hmac_alg_len < SHARED_SECRET_MAX)), s->cram_hmac_alg)) + goto nla_put_failure; + if (nla_put(skb, DRBD_A_NET_CONF_INTEGRITY_ALG, min_t(int, SHARED_SECRET_MAX, + s->integrity_alg_len + (s->integrity_alg_len < SHARED_SECRET_MAX)), s->integrity_alg)) + goto nla_put_failure; + if (nla_put(skb, DRBD_A_NET_CONF_VERIFY_ALG, min_t(int, SHARED_SECRET_MAX, + s->verify_alg_len + (s->verify_alg_len < SHARED_SECRET_MAX)), s->verify_alg)) + goto nla_put_failure; + if (nla_put(skb, DRBD_A_NET_CONF_CSUMS_ALG, min_t(int, SHARED_SECRET_MAX, + s->csums_alg_len + (s->csums_alg_len < SHARED_SECRET_MAX)), s->csums_alg)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_NET_CONF_WIRE_PROTOCOL, s->wire_protocol)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_NET_CONF_CONNECT_INT, s->connect_int)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_NET_CONF_TIMEOUT, s->timeout)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_NET_CONF_PING_INT, s->ping_int)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_NET_CONF_PING_TIMEO, s->ping_timeo)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_NET_CONF_SNDBUF_SIZE, s->sndbuf_size)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_NET_CONF_RCVBUF_SIZE, s->rcvbuf_size)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_NET_CONF_KO_COUNT, s->ko_count)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_NET_CONF_MAX_BUFFERS, s->max_buffers)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_NET_CONF_MAX_EPOCH_SIZE, s->max_epoch_size)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_NET_CONF_UNPLUG_WATERMARK, s->unplug_watermark)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_NET_CONF_AFTER_SB_0P, s->after_sb_0p)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_NET_CONF_AFTER_SB_1P, s->after_sb_1p)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_NET_CONF_AFTER_SB_2P, s->after_sb_2p)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_NET_CONF_RR_CONFLICT, s->rr_conflict)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_NET_CONF_ON_CONGESTION, s->on_congestion)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_NET_CONF_CONG_FILL, s->cong_fill)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_NET_CONF_CONG_EXTENTS, s->cong_extents)) + goto nla_put_failure; + if (nla_put_u8(skb, DRBD_A_NET_CONF_TWO_PRIMARIES, s->two_primaries)) + goto nla_put_failure; + if (nla_put_u8(skb, DRBD_A_NET_CONF_DISCARD_MY_DATA, s->discard_my_data)) + goto nla_put_failure; + if (nla_put_u8(skb, DRBD_A_NET_CONF_TCP_CORK, s->tcp_cork)) + goto nla_put_failure; + if (nla_put_u8(skb, DRBD_A_NET_CONF_ALWAYS_ASBP, s->always_asbp)) + goto nla_put_failure; + if (nla_put_u8(skb, DRBD_A_NET_CONF_TENTATIVE, s->tentative)) + goto nla_put_failure; + if (nla_put_u8(skb, DRBD_A_NET_CONF_USE_RLE, s->use_rle)) + goto nla_put_failure; + if (nla_put_u8(skb, DRBD_A_NET_CONF_CSUMS_AFTER_CRASH_ONLY, s->csums_after_crash_only)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_NET_CONF_SOCK_CHECK_TIMEO, s->sock_check_timeo)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +int set_role_parms_to_skb(struct sk_buff *skb, struct set_role_parms *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_SET_ROLE_PARMS); + + if (!tla) + goto nla_put_failure; + + if (nla_put_u8(skb, DRBD_A_SET_ROLE_PARMS_ASSUME_UPTODATE, s->assume_uptodate)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +int resize_parms_to_skb(struct sk_buff *skb, struct resize_parms *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_RESIZE_PARMS); + + if (!tla) + goto nla_put_failure; + + if (nla_put_u64_64bit(skb, DRBD_A_RESIZE_PARMS_RESIZE_SIZE, s->resize_size, 0)) + goto nla_put_failure; + if (nla_put_u8(skb, DRBD_A_RESIZE_PARMS_RESIZE_FORCE, s->resize_force)) + goto nla_put_failure; + if (nla_put_u8(skb, DRBD_A_RESIZE_PARMS_NO_RESYNC, s->no_resync)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_RESIZE_PARMS_AL_STRIPES, s->al_stripes)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_RESIZE_PARMS_AL_STRIPE_SIZE, s->al_stripe_size)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +int state_info_to_skb(struct sk_buff *skb, struct state_info *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_STATE_INFO); + + if (!tla) + goto nla_put_failure; + + if (nla_put_u32(skb, DRBD_A_STATE_INFO_SIB_REASON, s->sib_reason)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_STATE_INFO_CURRENT_STATE, s->current_state)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_CAPACITY, s->capacity, 0)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_ED_UUID, s->ed_uuid, 0)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_STATE_INFO_PREV_STATE, s->prev_state)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_STATE_INFO_NEW_STATE, s->new_state)) + goto nla_put_failure; + if (nla_put(skb, DRBD_A_STATE_INFO_UUIDS, min_t(int, DRBD_NL_UUIDS_SIZE, + s->uuids_len), s->uuids)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_STATE_INFO_DISK_FLAGS, s->disk_flags)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BITS_TOTAL, s->bits_total, 0)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BITS_OOS, s->bits_oos, 0)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BITS_RS_TOTAL, s->bits_rs_total, 0)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BITS_RS_FAILED, s->bits_rs_failed, 0)) + goto nla_put_failure; + if (nla_put(skb, DRBD_A_STATE_INFO_HELPER, min_t(int, 32, + s->helper_len + (s->helper_len < 32)), s->helper)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_STATE_INFO_HELPER_EXIT_CODE, s->helper_exit_code)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_SEND_CNT, s->send_cnt, 0)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_RECV_CNT, s->recv_cnt, 0)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_READ_CNT, s->read_cnt, 0)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_WRIT_CNT, s->writ_cnt, 0)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_AL_WRIT_CNT, s->al_writ_cnt, 0)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_STATE_INFO_BM_WRIT_CNT, s->bm_writ_cnt, 0)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_STATE_INFO_AP_BIO_CNT, s->ap_bio_cnt)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_STATE_INFO_AP_PENDING_CNT, s->ap_pending_cnt)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_STATE_INFO_RS_PENDING_CNT, s->rs_pending_cnt)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +int start_ov_parms_to_skb(struct sk_buff *skb, struct start_ov_parms *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_START_OV_PARMS); + + if (!tla) + goto nla_put_failure; + + if (nla_put_u64_64bit(skb, DRBD_A_START_OV_PARMS_OV_START_SECTOR, s->ov_start_sector, 0)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_START_OV_PARMS_OV_STOP_SECTOR, s->ov_stop_sector, 0)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +int new_c_uuid_parms_to_skb(struct sk_buff *skb, struct new_c_uuid_parms *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_NEW_C_UUID_PARMS); + + if (!tla) + goto nla_put_failure; + + if (nla_put_u8(skb, DRBD_A_NEW_C_UUID_PARMS_CLEAR_BM, s->clear_bm)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +int timeout_parms_to_skb(struct sk_buff *skb, struct timeout_parms *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_TIMEOUT_PARMS); + + if (!tla) + goto nla_put_failure; + + if (nla_put_u32(skb, DRBD_A_TIMEOUT_PARMS_TIMEOUT_TYPE, s->timeout_type)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +int disconnect_parms_to_skb(struct sk_buff *skb, struct disconnect_parms *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_DISCONNECT_PARMS); + + if (!tla) + goto nla_put_failure; + + if (nla_put_u8(skb, DRBD_A_DISCONNECT_PARMS_FORCE_DISCONNECT, s->force_disconnect)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +int detach_parms_to_skb(struct sk_buff *skb, struct detach_parms *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_DETACH_PARMS); + + if (!tla) + goto nla_put_failure; + + if (nla_put_u8(skb, DRBD_A_DETACH_PARMS_FORCE_DETACH, s->force_detach)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +int resource_info_to_skb(struct sk_buff *skb, struct resource_info *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_RESOURCE_INFO); + + if (!tla) + goto nla_put_failure; + + if (nla_put_u32(skb, DRBD_A_RESOURCE_INFO_RES_ROLE, s->res_role)) + goto nla_put_failure; + if (nla_put_u8(skb, DRBD_A_RESOURCE_INFO_RES_SUSP, s->res_susp)) + goto nla_put_failure; + if (nla_put_u8(skb, DRBD_A_RESOURCE_INFO_RES_SUSP_NOD, s->res_susp_nod)) + goto nla_put_failure; + if (nla_put_u8(skb, DRBD_A_RESOURCE_INFO_RES_SUSP_FEN, s->res_susp_fen)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +int device_info_to_skb(struct sk_buff *skb, struct device_info *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_DEVICE_INFO); + + if (!tla) + goto nla_put_failure; + + if (nla_put_u32(skb, DRBD_A_DEVICE_INFO_DEV_DISK_STATE, s->dev_disk_state)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +int connection_info_to_skb(struct sk_buff *skb, struct connection_info *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_CONNECTION_INFO); + + if (!tla) + goto nla_put_failure; + + if (nla_put_u32(skb, DRBD_A_CONNECTION_INFO_CONN_CONNECTION_STATE, s->conn_connection_state)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_CONNECTION_INFO_CONN_ROLE, s->conn_role)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +int peer_device_info_to_skb(struct sk_buff *skb, struct peer_device_info *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_PEER_DEVICE_INFO); + + if (!tla) + goto nla_put_failure; + + if (nla_put_u32(skb, DRBD_A_PEER_DEVICE_INFO_PEER_REPL_STATE, s->peer_repl_state)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_PEER_DEVICE_INFO_PEER_DISK_STATE, s->peer_disk_state)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_PEER_DEVICE_INFO_PEER_RESYNC_SUSP_USER, s->peer_resync_susp_user)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_PEER_DEVICE_INFO_PEER_RESYNC_SUSP_PEER, s->peer_resync_susp_peer)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_PEER_DEVICE_INFO_PEER_RESYNC_SUSP_DEPENDENCY, s->peer_resync_susp_dependency)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +int resource_statistics_to_skb(struct sk_buff *skb, struct resource_statistics *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_RESOURCE_STATISTICS); + + if (!tla) + goto nla_put_failure; + + if (nla_put_u32(skb, DRBD_A_RESOURCE_STATISTICS_RES_STAT_WRITE_ORDERING, s->res_stat_write_ordering)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +int device_statistics_to_skb(struct sk_buff *skb, struct device_statistics *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_DEVICE_STATISTICS); + + if (!tla) + goto nla_put_failure; + + if (nla_put_u64_64bit(skb, DRBD_A_DEVICE_STATISTICS_DEV_SIZE, s->dev_size, 0)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_DEVICE_STATISTICS_DEV_READ, s->dev_read, 0)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_DEVICE_STATISTICS_DEV_WRITE, s->dev_write, 0)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_DEVICE_STATISTICS_DEV_AL_WRITES, s->dev_al_writes, 0)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_DEVICE_STATISTICS_DEV_BM_WRITES, s->dev_bm_writes, 0)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_DEVICE_STATISTICS_DEV_UPPER_PENDING, s->dev_upper_pending)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_DEVICE_STATISTICS_DEV_LOWER_PENDING, s->dev_lower_pending)) + goto nla_put_failure; + if (nla_put_u8(skb, DRBD_A_DEVICE_STATISTICS_DEV_UPPER_BLOCKED, s->dev_upper_blocked)) + goto nla_put_failure; + if (nla_put_u8(skb, DRBD_A_DEVICE_STATISTICS_DEV_LOWER_BLOCKED, s->dev_lower_blocked)) + goto nla_put_failure; + if (nla_put_u8(skb, DRBD_A_DEVICE_STATISTICS_DEV_AL_SUSPENDED, s->dev_al_suspended)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_DEVICE_STATISTICS_DEV_EXPOSED_DATA_UUID, s->dev_exposed_data_uuid, 0)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_DEVICE_STATISTICS_DEV_CURRENT_UUID, s->dev_current_uuid, 0)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_DEVICE_STATISTICS_DEV_DISK_FLAGS, s->dev_disk_flags)) + goto nla_put_failure; + if (nla_put(skb, DRBD_A_DEVICE_STATISTICS_HISTORY_UUIDS, min_t(int, DRBD_NL_HISTORY_UUIDS_SIZE, + s->history_uuids_len), s->history_uuids)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +int connection_statistics_to_skb(struct sk_buff *skb, struct connection_statistics *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_CONNECTION_STATISTICS); + + if (!tla) + goto nla_put_failure; + + if (nla_put_u8(skb, DRBD_A_CONNECTION_STATISTICS_CONN_CONGESTED, s->conn_congested)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +int peer_device_statistics_to_skb(struct sk_buff *skb, struct peer_device_statistics *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_PEER_DEVICE_STATISTICS); + + if (!tla) + goto nla_put_failure; + + if (nla_put_u64_64bit(skb, DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_RECEIVED, s->peer_dev_received, 0)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_SENT, s->peer_dev_sent, 0)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_PENDING, s->peer_dev_pending)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_UNACKED, s->peer_dev_unacked)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_OUT_OF_SYNC, s->peer_dev_out_of_sync, 0)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_RESYNC_FAILED, s->peer_dev_resync_failed, 0)) + goto nla_put_failure; + if (nla_put_u64_64bit(skb, DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_BITMAP_UUID, s->peer_dev_bitmap_uuid, 0)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_FLAGS, s->peer_dev_flags)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +int drbd_notification_header_to_skb(struct sk_buff *skb, struct drbd_notification_header *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_NOTIFICATION_HEADER); + + if (!tla) + goto nla_put_failure; + + if (nla_put_u32(skb, DRBD_A_DRBD_NOTIFICATION_HEADER_NH_TYPE, s->nh_type)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +int drbd_helper_info_to_skb(struct sk_buff *skb, struct drbd_helper_info *s) +{ + struct nlattr *tla = nla_nest_start(skb, DRBD_NLA_HELPER); + + if (!tla) + goto nla_put_failure; + + if (nla_put(skb, DRBD_A_DRBD_HELPER_INFO_HELPER_NAME, min_t(int, 32, + s->helper_name_len + (s->helper_name_len < 32)), s->helper_name)) + goto nla_put_failure; + if (nla_put_u32(skb, DRBD_A_DRBD_HELPER_INFO_HELPER_STATUS, s->helper_status)) + goto nla_put_failure; + + nla_nest_end(skb, tla); + return 0; + +nla_put_failure: + if (tla) + nla_nest_cancel(skb, tla); + return -EMSGSIZE; +} + +void set_disk_conf_defaults(struct disk_conf *x) +{ + x->on_io_error = DRBD_ON_IO_ERROR_DEF; + x->fencing = DRBD_FENCING_DEF; + x->resync_rate = DRBD_RESYNC_RATE_DEF; + x->resync_after = DRBD_MINOR_NUMBER_DEF; + x->al_extents = DRBD_AL_EXTENTS_DEF; + x->c_plan_ahead = DRBD_C_PLAN_AHEAD_DEF; + x->c_delay_target = DRBD_C_DELAY_TARGET_DEF; + x->c_fill_target = DRBD_C_FILL_TARGET_DEF; + x->c_max_rate = DRBD_C_MAX_RATE_DEF; + x->c_min_rate = DRBD_C_MIN_RATE_DEF; + x->disk_barrier = DRBD_DISK_BARRIER_DEF; + x->disk_flushes = DRBD_DISK_FLUSHES_DEF; + x->disk_drain = DRBD_DISK_DRAIN_DEF; + x->md_flushes = DRBD_MD_FLUSHES_DEF; + x->disk_timeout = DRBD_DISK_TIMEOUT_DEF; + x->read_balancing = DRBD_READ_BALANCING_DEF; + x->al_updates = DRBD_AL_UPDATES_DEF; + x->discard_zeroes_if_aligned = DRBD_DISCARD_ZEROES_IF_ALIGNED_DEF; + x->rs_discard_granularity = DRBD_RS_DISCARD_GRANULARITY_DEF; + x->disable_write_same = DRBD_DISABLE_WRITE_SAME_DEF; +} + +void set_res_opts_defaults(struct res_opts *x) +{ + memset(x->cpu_mask, 0, sizeof(x->cpu_mask)); + x->cpu_mask_len = 0; + x->on_no_data = DRBD_ON_NO_DATA_DEF; +} + +void set_net_conf_defaults(struct net_conf *x) +{ + memset(x->shared_secret, 0, sizeof(x->shared_secret)); + x->shared_secret_len = 0; + memset(x->cram_hmac_alg, 0, sizeof(x->cram_hmac_alg)); + x->cram_hmac_alg_len = 0; + memset(x->integrity_alg, 0, sizeof(x->integrity_alg)); + x->integrity_alg_len = 0; + memset(x->verify_alg, 0, sizeof(x->verify_alg)); + x->verify_alg_len = 0; + memset(x->csums_alg, 0, sizeof(x->csums_alg)); + x->csums_alg_len = 0; + x->wire_protocol = DRBD_PROTOCOL_DEF; + x->connect_int = DRBD_CONNECT_INT_DEF; + x->timeout = DRBD_TIMEOUT_DEF; + x->ping_int = DRBD_PING_INT_DEF; + x->ping_timeo = DRBD_PING_TIMEO_DEF; + x->sndbuf_size = DRBD_SNDBUF_SIZE_DEF; + x->rcvbuf_size = DRBD_RCVBUF_SIZE_DEF; + x->ko_count = DRBD_KO_COUNT_DEF; + x->max_buffers = DRBD_MAX_BUFFERS_DEF; + x->max_epoch_size = DRBD_MAX_EPOCH_SIZE_DEF; + x->unplug_watermark = DRBD_UNPLUG_WATERMARK_DEF; + x->after_sb_0p = DRBD_AFTER_SB_0P_DEF; + x->after_sb_1p = DRBD_AFTER_SB_1P_DEF; + x->after_sb_2p = DRBD_AFTER_SB_2P_DEF; + x->rr_conflict = DRBD_RR_CONFLICT_DEF; + x->on_congestion = DRBD_ON_CONGESTION_DEF; + x->cong_fill = DRBD_CONG_FILL_DEF; + x->cong_extents = DRBD_CONG_EXTENTS_DEF; + x->two_primaries = DRBD_ALLOW_TWO_PRIMARIES_DEF; + x->tcp_cork = DRBD_TCP_CORK_DEF; + x->always_asbp = DRBD_ALWAYS_ASBP_DEF; + x->use_rle = DRBD_USE_RLE_DEF; + x->csums_after_crash_only = DRBD_CSUMS_AFTER_CRASH_ONLY_DEF; + x->sock_check_timeo = DRBD_SOCKET_CHECK_TIMEO_DEF; +} + +void set_resize_parms_defaults(struct resize_parms *x) +{ + x->al_stripes = DRBD_AL_STRIPES_DEF; + x->al_stripe_size = DRBD_AL_STRIPE_SIZE_DEF; +} diff --git a/drivers/block/drbd/drbd_nl_gen.h b/drivers/block/drbd/drbd_nl_gen.h new file mode 100644 index 000000000000..f2140dd1ac4e --- /dev/null +++ b/drivers/block/drbd/drbd_nl_gen.h @@ -0,0 +1,395 @@ +/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */ + +#ifndef _LINUX_DRBD_GEN_H +#define _LINUX_DRBD_GEN_H + +#include <net/netlink.h> +#include <net/genetlink.h> + +#include <uapi/linux/drbd_genl.h> +#include <linux/drbd.h> +#include <linux/drbd_limits.h> + +/* Common nested types */ +extern const struct nla_policy drbd_connection_info_nl_policy[DRBD_A_CONNECTION_INFO_CONN_ROLE + 1]; +extern const struct nla_policy drbd_connection_statistics_nl_policy[DRBD_A_CONNECTION_STATISTICS_CONN_CONGESTED + 1]; +extern const struct nla_policy drbd_detach_parms_nl_policy[DRBD_A_DETACH_PARMS_FORCE_DETACH + 1]; +extern const struct nla_policy drbd_device_info_nl_policy[DRBD_A_DEVICE_INFO_DEV_DISK_STATE + 1]; +extern const struct nla_policy drbd_device_statistics_nl_policy[DRBD_A_DEVICE_STATISTICS_HISTORY_UUIDS + 1]; +extern const struct nla_policy drbd_disconnect_parms_nl_policy[DRBD_A_DISCONNECT_PARMS_FORCE_DISCONNECT + 1]; +extern const struct nla_policy drbd_disk_conf_nl_policy[DRBD_A_DISK_CONF_DISABLE_WRITE_SAME + 1]; +extern const struct nla_policy drbd_drbd_cfg_context_nl_policy[DRBD_A_DRBD_CFG_CONTEXT_CTX_PEER_ADDR + 1]; +extern const struct nla_policy drbd_net_conf_nl_policy[DRBD_A_NET_CONF_SOCK_CHECK_TIMEO + 1]; +extern const struct nla_policy drbd_new_c_uuid_parms_nl_policy[DRBD_A_NEW_C_UUID_PARMS_CLEAR_BM + 1]; +extern const struct nla_policy drbd_peer_device_info_nl_policy[DRBD_A_PEER_DEVICE_INFO_PEER_RESYNC_SUSP_DEPENDENCY + 1]; +extern const struct nla_policy drbd_peer_device_statistics_nl_policy[DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_FLAGS + 1]; +extern const struct nla_policy drbd_res_opts_nl_policy[DRBD_A_RES_OPTS_ON_NO_DATA + 1]; +extern const struct nla_policy drbd_resize_parms_nl_policy[DRBD_A_RESIZE_PARMS_AL_STRIPE_SIZE + 1]; +extern const struct nla_policy drbd_resource_info_nl_policy[DRBD_A_RESOURCE_INFO_RES_SUSP_FEN + 1]; +extern const struct nla_policy drbd_resource_statistics_nl_policy[DRBD_A_RESOURCE_STATISTICS_RES_STAT_WRITE_ORDERING + 1]; +extern const struct nla_policy drbd_set_role_parms_nl_policy[DRBD_A_SET_ROLE_PARMS_ASSUME_UPTODATE + 1]; +extern const struct nla_policy drbd_start_ov_parms_nl_policy[DRBD_A_START_OV_PARMS_OV_STOP_SECTOR + 1]; + +/* Ops table for drbd */ +extern const struct genl_split_ops drbd_nl_ops[32]; + +int drbd_pre_doit(const struct genl_split_ops *ops, struct sk_buff *skb, + struct genl_info *info); +void +drbd_post_doit(const struct genl_split_ops *ops, struct sk_buff *skb, + struct genl_info *info); +int drbd_adm_dump_devices_done(struct netlink_callback *cb); +int drbd_adm_dump_connections_done(struct netlink_callback *cb); +int drbd_adm_dump_peer_devices_done(struct netlink_callback *cb); + +int drbd_nl_get_status_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_get_status_dumpit(struct sk_buff *skb, struct netlink_callback *cb); +int drbd_nl_new_minor_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_del_minor_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_new_resource_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_del_resource_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_resource_opts_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_connect_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_disconnect_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_attach_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_resize_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_primary_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_secondary_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_new_c_uuid_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_start_ov_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_detach_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_invalidate_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_inval_peer_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_pause_sync_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_resume_sync_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_suspend_io_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_resume_io_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_outdate_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_get_timeout_type_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_down_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_chg_disk_opts_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_chg_net_opts_doit(struct sk_buff *skb, struct genl_info *info); +int drbd_nl_get_resources_dumpit(struct sk_buff *skb, + struct netlink_callback *cb); +int drbd_nl_get_devices_dumpit(struct sk_buff *skb, + struct netlink_callback *cb); +int drbd_nl_get_connections_dumpit(struct sk_buff *skb, + struct netlink_callback *cb); +int drbd_nl_get_peer_devices_dumpit(struct sk_buff *skb, + struct netlink_callback *cb); +int drbd_nl_get_initial_state_dumpit(struct sk_buff *skb, + struct netlink_callback *cb); + +enum { + DRBD_NLGRP_EVENTS, +}; + +struct drbd_cfg_reply { + char info_text[0]; + __u32 info_text_len; +}; + +struct drbd_cfg_context { + __u32 ctx_volume; + char ctx_resource_name[128]; + __u32 ctx_resource_name_len; + char ctx_my_addr[128]; + __u32 ctx_my_addr_len; + char ctx_peer_addr[128]; + __u32 ctx_peer_addr_len; +}; + +struct disk_conf { + char backing_dev[128]; + __u32 backing_dev_len; + char meta_dev[128]; + __u32 meta_dev_len; + __s32 meta_dev_idx; + __u64 disk_size; + __u32 max_bio_bvecs; + __u32 on_io_error; + __u32 fencing; + __u32 resync_rate; + __s32 resync_after; + __u32 al_extents; + __u32 c_plan_ahead; + __u32 c_delay_target; + __u32 c_fill_target; + __u32 c_max_rate; + __u32 c_min_rate; + unsigned char disk_barrier; + unsigned char disk_flushes; + unsigned char disk_drain; + unsigned char md_flushes; + __u32 disk_timeout; + __u32 read_balancing; + unsigned char al_updates; + unsigned char discard_zeroes_if_aligned; + __u32 rs_discard_granularity; + unsigned char disable_write_same; +}; + +struct res_opts { + char cpu_mask[DRBD_CPU_MASK_SIZE]; + __u32 cpu_mask_len; + __u32 on_no_data; +}; + +struct net_conf { + char shared_secret[SHARED_SECRET_MAX]; + __u32 shared_secret_len; + char cram_hmac_alg[SHARED_SECRET_MAX]; + __u32 cram_hmac_alg_len; + char integrity_alg[SHARED_SECRET_MAX]; + __u32 integrity_alg_len; + char verify_alg[SHARED_SECRET_MAX]; + __u32 verify_alg_len; + char csums_alg[SHARED_SECRET_MAX]; + __u32 csums_alg_len; + __u32 wire_protocol; + __u32 connect_int; + __u32 timeout; + __u32 ping_int; + __u32 ping_timeo; + __u32 sndbuf_size; + __u32 rcvbuf_size; + __u32 ko_count; + __u32 max_buffers; + __u32 max_epoch_size; + __u32 unplug_watermark; + __u32 after_sb_0p; + __u32 after_sb_1p; + __u32 after_sb_2p; + __u32 rr_conflict; + __u32 on_congestion; + __u32 cong_fill; + __u32 cong_extents; + unsigned char two_primaries; + unsigned char discard_my_data; + unsigned char tcp_cork; + unsigned char always_asbp; + unsigned char tentative; + unsigned char use_rle; + unsigned char csums_after_crash_only; + __u32 sock_check_timeo; +}; + +struct set_role_parms { + unsigned char assume_uptodate; +}; + +struct resize_parms { + __u64 resize_size; + unsigned char resize_force; + unsigned char no_resync; + __u32 al_stripes; + __u32 al_stripe_size; +}; + +struct state_info { + __u32 sib_reason; + __u32 current_state; + __u64 capacity; + __u64 ed_uuid; + __u32 prev_state; + __u32 new_state; + char uuids[DRBD_NL_UUIDS_SIZE]; + __u32 uuids_len; + __u32 disk_flags; + __u64 bits_total; + __u64 bits_oos; + __u64 bits_rs_total; + __u64 bits_rs_failed; + char helper[32]; + __u32 helper_len; + __u32 helper_exit_code; + __u64 send_cnt; + __u64 recv_cnt; + __u64 read_cnt; + __u64 writ_cnt; + __u64 al_writ_cnt; + __u64 bm_writ_cnt; + __u32 ap_bio_cnt; + __u32 ap_pending_cnt; + __u32 rs_pending_cnt; +}; + +struct start_ov_parms { + __u64 ov_start_sector; + __u64 ov_stop_sector; +}; + +struct new_c_uuid_parms { + unsigned char clear_bm; +}; + +struct timeout_parms { + __u32 timeout_type; +}; + +struct disconnect_parms { + unsigned char force_disconnect; +}; + +struct detach_parms { + unsigned char force_detach; +}; + +struct resource_info { + __u32 res_role; + unsigned char res_susp; + unsigned char res_susp_nod; + unsigned char res_susp_fen; +}; + +struct device_info { + __u32 dev_disk_state; +}; + +struct connection_info { + __u32 conn_connection_state; + __u32 conn_role; +}; + +struct peer_device_info { + __u32 peer_repl_state; + __u32 peer_disk_state; + __u32 peer_resync_susp_user; + __u32 peer_resync_susp_peer; + __u32 peer_resync_susp_dependency; +}; + +struct resource_statistics { + __u32 res_stat_write_ordering; +}; + +struct device_statistics { + __u64 dev_size; + __u64 dev_read; + __u64 dev_write; + __u64 dev_al_writes; + __u64 dev_bm_writes; + __u32 dev_upper_pending; + __u32 dev_lower_pending; + unsigned char dev_upper_blocked; + unsigned char dev_lower_blocked; + unsigned char dev_al_suspended; + __u64 dev_exposed_data_uuid; + __u64 dev_current_uuid; + __u32 dev_disk_flags; + char history_uuids[DRBD_NL_HISTORY_UUIDS_SIZE]; + __u32 history_uuids_len; +}; + +struct connection_statistics { + unsigned char conn_congested; +}; + +struct peer_device_statistics { + __u64 peer_dev_received; + __u64 peer_dev_sent; + __u32 peer_dev_pending; + __u32 peer_dev_unacked; + __u64 peer_dev_out_of_sync; + __u64 peer_dev_resync_failed; + __u64 peer_dev_bitmap_uuid; + __u32 peer_dev_flags; +}; + +struct drbd_notification_header { + __u32 nh_type; +}; + +struct drbd_helper_info { + char helper_name[32]; + __u32 helper_name_len; + __u32 helper_status; +}; + +int drbd_cfg_reply_to_skb(struct sk_buff *skb, struct drbd_cfg_reply *s); + +int drbd_cfg_context_from_attrs(struct drbd_cfg_context *s, struct genl_info *info); +int drbd_cfg_context_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info); +int drbd_cfg_context_to_skb(struct sk_buff *skb, struct drbd_cfg_context *s); + +int disk_conf_from_attrs(struct disk_conf *s, struct genl_info *info); +int disk_conf_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info); +int disk_conf_to_skb(struct sk_buff *skb, struct disk_conf *s); +void set_disk_conf_defaults(struct disk_conf *x); + +int res_opts_from_attrs(struct res_opts *s, struct genl_info *info); +int res_opts_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info); +int res_opts_to_skb(struct sk_buff *skb, struct res_opts *s); +void set_res_opts_defaults(struct res_opts *x); + +int net_conf_from_attrs(struct net_conf *s, struct genl_info *info); +int net_conf_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info); +int net_conf_to_skb(struct sk_buff *skb, struct net_conf *s); +void set_net_conf_defaults(struct net_conf *x); + +int set_role_parms_from_attrs(struct set_role_parms *s, struct genl_info *info); +int set_role_parms_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info); +int set_role_parms_to_skb(struct sk_buff *skb, struct set_role_parms *s); + +int resize_parms_from_attrs(struct resize_parms *s, struct genl_info *info); +int resize_parms_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info); +int resize_parms_to_skb(struct sk_buff *skb, struct resize_parms *s); +void set_resize_parms_defaults(struct resize_parms *x); + +int state_info_to_skb(struct sk_buff *skb, struct state_info *s); + +int start_ov_parms_from_attrs(struct start_ov_parms *s, struct genl_info *info); +int start_ov_parms_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info); +int start_ov_parms_to_skb(struct sk_buff *skb, struct start_ov_parms *s); + +int new_c_uuid_parms_from_attrs(struct new_c_uuid_parms *s, struct genl_info *info); +int new_c_uuid_parms_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info); +int new_c_uuid_parms_to_skb(struct sk_buff *skb, struct new_c_uuid_parms *s); + +int timeout_parms_to_skb(struct sk_buff *skb, struct timeout_parms *s); + +int disconnect_parms_from_attrs(struct disconnect_parms *s, struct genl_info *info); +int disconnect_parms_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info); +int disconnect_parms_to_skb(struct sk_buff *skb, struct disconnect_parms *s); + +int detach_parms_from_attrs(struct detach_parms *s, struct genl_info *info); +int detach_parms_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info); +int detach_parms_to_skb(struct sk_buff *skb, struct detach_parms *s); + +int resource_info_from_attrs(struct resource_info *s, struct genl_info *info); +int resource_info_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info); +int resource_info_to_skb(struct sk_buff *skb, struct resource_info *s); + +int device_info_from_attrs(struct device_info *s, struct genl_info *info); +int device_info_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info); +int device_info_to_skb(struct sk_buff *skb, struct device_info *s); + +int connection_info_from_attrs(struct connection_info *s, struct genl_info *info); +int connection_info_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info); +int connection_info_to_skb(struct sk_buff *skb, struct connection_info *s); + +int peer_device_info_from_attrs(struct peer_device_info *s, struct genl_info *info); +int peer_device_info_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info); +int peer_device_info_to_skb(struct sk_buff *skb, struct peer_device_info *s); + +int resource_statistics_from_attrs(struct resource_statistics *s, struct genl_info *info); +int resource_statistics_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info); +int resource_statistics_to_skb(struct sk_buff *skb, struct resource_statistics *s); + +int device_statistics_from_attrs(struct device_statistics *s, struct genl_info *info); +int device_statistics_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info); +int device_statistics_to_skb(struct sk_buff *skb, struct device_statistics *s); + +int connection_statistics_from_attrs(struct connection_statistics *s, struct genl_info *info); +int connection_statistics_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info); +int connection_statistics_to_skb(struct sk_buff *skb, struct connection_statistics *s); + +int peer_device_statistics_from_attrs(struct peer_device_statistics *s, struct genl_info *info); +int peer_device_statistics_ntb_from_attrs(struct nlattr ***ret_nested_attribute_table, struct genl_info *info); +int peer_device_statistics_to_skb(struct sk_buff *skb, struct peer_device_statistics *s); + +int drbd_notification_header_to_skb(struct sk_buff *skb, struct drbd_notification_header *s); + +int drbd_helper_info_to_skb(struct sk_buff *skb, struct drbd_helper_info *s); + +#endif /* _LINUX_DRBD_GEN_H */ diff --git a/drivers/block/drbd/drbd_proc.c b/drivers/block/drbd/drbd_proc.c index 1d0feafceadc..6d0c12c10260 100644 --- a/drivers/block/drbd/drbd_proc.c +++ b/drivers/block/drbd/drbd_proc.c @@ -228,7 +228,7 @@ int drbd_seq_show(struct seq_file *seq, void *v) }; seq_printf(seq, "version: " REL_VERSION " (api:%d/proto:%d-%d)\n%s\n", - GENL_MAGIC_VERSION, PRO_VERSION_MIN, PRO_VERSION_MAX, drbd_buildtag()); + DRBD_FAMILY_VERSION, PRO_VERSION_MIN, PRO_VERSION_MAX, drbd_buildtag()); /* cs .. connection state diff --git a/drivers/block/floppy.c b/drivers/block/floppy.c index 0509746f8aed..dca495be0683 100644 --- a/drivers/block/floppy.c +++ b/drivers/block/floppy.c @@ -5012,8 +5012,8 @@ MODULE_LICENSE("GPL"); /* This doesn't actually get used other than for module information */ static const struct pnp_device_id floppy_pnpids[] = { - {"PNP0700", 0}, - {} + { .id = "PNP0700" }, + { } }; MODULE_DEVICE_TABLE(pnp, floppy_pnpids); diff --git a/drivers/block/loop.c b/drivers/block/loop.c index 0000913f7efc..310de0463beb 100644 --- a/drivers/block/loop.c +++ b/drivers/block/loop.c @@ -342,23 +342,19 @@ static int lo_rw_aio(struct loop_device *lo, struct loop_cmd *cmd, { struct iov_iter iter; struct req_iterator rq_iter; - struct bio_vec *bvec; struct request *rq = blk_mq_rq_from_pdu(cmd); - struct bio *bio = rq->bio; struct file *file = lo->lo_backing_file; - struct bio_vec tmp; - unsigned int offset; unsigned int nr_bvec; int ret; nr_bvec = blk_rq_nr_bvec(rq); if (rq->bio != rq->biotail) { + struct bio_vec tmp, *bvec; - bvec = kmalloc_objs(struct bio_vec, nr_bvec, GFP_NOIO); - if (!bvec) + cmd->bvec = kmalloc_objs(*cmd->bvec, nr_bvec, GFP_NOIO); + if (!cmd->bvec) return -EIO; - cmd->bvec = bvec; /* * The bios of the request may be started from the middle of @@ -366,26 +362,26 @@ static int lo_rw_aio(struct loop_device *lo, struct loop_cmd *cmd, * copy bio->bi_iov_vec to new bvec. The rq_for_each_bvec * API will take care of all details for us. */ + bvec = cmd->bvec; rq_for_each_bvec(tmp, rq, rq_iter) { *bvec = tmp; bvec++; } - bvec = cmd->bvec; - offset = 0; + iov_iter_bvec(&iter, rw, cmd->bvec, nr_bvec, blk_rq_bytes(rq)); + iter.iov_offset = 0; } else { /* * Same here, this bio may be started from the middle of the * 'bvec' because of bio splitting, so offset from the bvec * must be passed to iov iterator */ - offset = bio->bi_iter.bi_bvec_done; - bvec = __bvec_iter_bvec(bio->bi_io_vec, bio->bi_iter); + iov_iter_bvec(&iter, rw, + __bvec_iter_bvec(rq->bio->bi_io_vec, rq->bio->bi_iter), + nr_bvec, blk_rq_bytes(rq)); + iter.iov_offset = rq->bio->bi_iter.bi_bvec_done; } atomic_set(&cmd->ref, 2); - iov_iter_bvec(&iter, rw, bvec, nr_bvec, blk_rq_bytes(rq)); - iter.iov_offset = offset; - cmd->iocb.ki_pos = pos; cmd->iocb.ki_filp = file; cmd->iocb.ki_ioprio = req_get_ioprio(rq); diff --git a/drivers/block/mtip32xx/mtip32xx.c b/drivers/block/mtip32xx/mtip32xx.c index 567192e371a8..f214a616386c 100644 --- a/drivers/block/mtip32xx/mtip32xx.c +++ b/drivers/block/mtip32xx/mtip32xx.c @@ -3340,7 +3340,7 @@ static void mtip_free_cmd(struct blk_mq_tag_set *set, struct request *rq, } static int mtip_init_cmd(struct blk_mq_tag_set *set, struct request *rq, - unsigned int hctx_idx, unsigned int numa_node) + unsigned int hctx_idx, int numa_node) { struct driver_data *dd = set->driver_data; struct mtip_cmd *cmd = blk_mq_rq_to_pdu(rq); @@ -3405,6 +3405,7 @@ static int mtip_block_initialize(struct driver_data *dd) .max_segment_size = 0x400000, }; int rv = 0, wait_for_rebuild = 0; + bool disk_added = false; sector_t capacity; unsigned int index = 0; @@ -3438,6 +3439,7 @@ static int mtip_block_initialize(struct driver_data *dd) dev_err(&dd->pdev->dev, "Unable to allocate request queue\n"); rv = -ENOMEM; + dd->disk = NULL; goto block_queue_alloc_init_error; } dd->queue = dd->disk->queue; @@ -3496,6 +3498,7 @@ skip_create_disk: rv = device_add_disk(&dd->pdev->dev, dd->disk, mtip_disk_attr_groups); if (rv) goto read_capacity_error; + disk_added = true; if (dd->mtip_svc_handler) { set_bit(MTIP_DDF_INIT_DONE_BIT, &dd->dd_flag); @@ -3511,7 +3514,9 @@ start_service_thread: dev_err(&dd->pdev->dev, "service thread failed to start\n"); dd->mtip_svc_handler = NULL; rv = -EFAULT; - goto kthread_run_error; + if (disk_added) + goto kthread_run_error; + goto read_capacity_error; } wake_up_process(dd->mtip_svc_handler); if (wait_for_rebuild == MTIP_FTL_REBUILD_MAGIC) @@ -3522,6 +3527,10 @@ start_service_thread: kthread_run_error: /* Delete our gendisk. This also removes the device from /dev */ del_gendisk(dd->disk); + mtip_hw_debugfs_exit(dd); + blk_mq_free_tag_set(&dd->tags); + mtip_hw_exit(dd); + return rv; read_capacity_error: init_hw_cmds_error: mtip_hw_debugfs_exit(dd); @@ -3529,6 +3538,7 @@ disk_index_error: ida_free(&rssd_index_ida, index); ida_get_error: put_disk(dd->disk); + dd->disk = NULL; block_queue_alloc_init_error: blk_mq_free_tag_set(&dd->tags); block_queue_alloc_tag_error: @@ -3839,7 +3849,10 @@ msi_initialize_err: } iomap_err: - kfree(dd); + if (dd->disk) + put_disk(dd->disk); + else + kfree(dd); pci_set_drvdata(pdev, NULL); return rv; done: diff --git a/drivers/block/n64cart.c b/drivers/block/n64cart.c index b9fdeff31caf..328da73b6f2c 100644 --- a/drivers/block/n64cart.c +++ b/drivers/block/n64cart.c @@ -12,6 +12,7 @@ #include <linux/init.h> #include <linux/module.h> #include <linux/platform_device.h> +#include <linux/string.h> enum { PI_DRAM_REG = 0, @@ -145,7 +146,7 @@ static int __init n64cart_probe(struct platform_device *pdev) disk->flags = GENHD_FL_NO_PART; disk->fops = &n64cart_fops; disk->private_data = &pdev->dev; - strcpy(disk->disk_name, "n64cart"); + strscpy(disk->disk_name, "n64cart"); set_capacity(disk, size >> SECTOR_SHIFT); set_disk_ro(disk, 1); diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c index fe63f3c55d0d..3a585a0c882a 100644 --- a/drivers/block/nbd.c +++ b/drivers/block/nbd.c @@ -1238,6 +1238,42 @@ static struct socket *nbd_get_socket(struct nbd_device *nbd, unsigned long fd, return sock; } +#ifdef CONFIG_DEBUG_LOCK_ALLOC +static struct lock_class_key nbd_key[3]; +static struct lock_class_key nbd_slock_key[3]; + +static void nbd_reclassify_socket(struct socket *sock) +{ + struct sock *sk = sock->sk; + + if (WARN_ON_ONCE(!sock_allow_reclassification(sk))) + return; + + switch (sk->sk_family) { + case AF_INET: + sock_lock_init_class_and_name(sk, "slock-AF_INET-NBD", + &nbd_slock_key[0], + "sk_lock-AF_INET-NBD", + &nbd_key[0]); + break; + case AF_INET6: + sock_lock_init_class_and_name(sk, "slock-AF_INET6-NBD", + &nbd_slock_key[1], + "sk_lock-AF_INET6-NBD", + &nbd_key[1]); + break; + case AF_UNIX: + sock_lock_init_class_and_name(sk, "slock-AF_UNIX-NBD", + &nbd_slock_key[2], + "sk_lock-AF_UNIX-NBD", + &nbd_key[2]); + break; + } +} +#else +static inline void nbd_reclassify_socket(struct socket *sock) {} +#endif + static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg, bool netlink) { @@ -1254,6 +1290,7 @@ static int nbd_add_socket(struct nbd_device *nbd, unsigned long arg, sock = nbd_get_socket(nbd, arg, &err); if (!sock) return err; + nbd_reclassify_socket(sock); /* * We need to make sure we don't get any errant requests while we're @@ -1888,7 +1925,7 @@ static void nbd_dbg_close(void) #endif static int nbd_init_request(struct blk_mq_tag_set *set, struct request *rq, - unsigned int hctx_idx, unsigned int numa_node) + unsigned int hctx_idx, int numa_node) { struct nbd_cmd *cmd = blk_mq_rq_to_pdu(rq); cmd->nbd = set->driver_data; diff --git a/drivers/block/rbd.c b/drivers/block/rbd.c index 6c1e7347e6a7..90c13de47cff 100644 --- a/drivers/block/rbd.c +++ b/drivers/block/rbd.c @@ -3672,7 +3672,7 @@ static void __rbd_lock(struct rbd_device *rbd_dev, const char *cookie) struct rbd_client_id cid = rbd_get_cid(rbd_dev); rbd_dev->lock_state = RBD_LOCK_STATE_LOCKED; - strcpy(rbd_dev->lock_cookie, cookie); + strscpy(rbd_dev->lock_cookie, cookie); rbd_set_owner_cid(rbd_dev, &cid); queue_work(rbd_dev->task_wq, &rbd_dev->acquired_lock_work); } @@ -6082,12 +6082,9 @@ static int rbd_dev_v2_snap_context(struct rbd_device *rbd_dev, /* * Make sure the reported number of snapshot ids wouldn't go - * beyond the end of our buffer. But before checking that, - * make sure the computed size of the snapshot context we - * allocate is representable in a size_t. + * beyond the end of our buffer. */ - if (snap_count > (SIZE_MAX - sizeof (struct ceph_snap_context)) - / sizeof (u64)) { + if (snap_count > RBD_MAX_SNAP_COUNT) { ret = -EINVAL; goto out; } diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c index 6c041eaebdb9..4f6d9e652187 100644 --- a/drivers/block/ublk_drv.c +++ b/drivers/block/ublk_drv.c @@ -359,7 +359,6 @@ static void ublk_buf_cleanup(struct ublk_device *ub); static void ublk_abort_queue(struct ublk_device *ub, struct ublk_queue *ubq); static inline struct request *__ublk_check_and_get_req(struct ublk_device *ub, u16 q_id, u16 tag, struct ublk_io *io); -static inline unsigned int ublk_req_build_flags(struct request *req); static void ublk_batch_dispatch(struct ublk_queue *ubq, const struct ublk_batch_io_data *data, struct ublk_batch_fetch_cmd *fcmd); @@ -471,6 +470,62 @@ static inline bool ublk_dev_support_integrity(const struct ublk_device *ub) return ub->dev_info.flags & UBLK_F_INTEGRITY; } +static inline unsigned int ublk_req_build_flags(struct request *req) +{ + unsigned flags = 0; + + if (req->cmd_flags & REQ_FAILFAST_DEV) + flags |= UBLK_IO_F_FAILFAST_DEV; + + if (req->cmd_flags & REQ_FAILFAST_TRANSPORT) + flags |= UBLK_IO_F_FAILFAST_TRANSPORT; + + if (req->cmd_flags & REQ_FAILFAST_DRIVER) + flags |= UBLK_IO_F_FAILFAST_DRIVER; + + if (req->cmd_flags & REQ_META) + flags |= UBLK_IO_F_META; + + if (req->cmd_flags & REQ_FUA) + flags |= UBLK_IO_F_FUA; + + if (req->cmd_flags & REQ_NOUNMAP) + flags |= UBLK_IO_F_NOUNMAP; + + if (req->cmd_flags & REQ_SWAP) + flags |= UBLK_IO_F_SWAP; + + if (blk_integrity_rq(req)) + flags |= UBLK_IO_F_INTEGRITY; + + return flags; +} + +static void ublk_init_iod(struct ublk_queue *ubq, struct request *req, + uint8_t ublk_op, uint32_t nr_sectors, + uint64_t start_sector) +{ + struct ublksrv_io_desc *iod = ublk_get_iod(ubq, req->tag); + struct ublk_io *io = &ubq->ios[req->tag]; + + iod->op_flags = ublk_op | ublk_req_build_flags(req); + iod->nr_sectors = nr_sectors; + iod->start_sector = start_sector; + + /* Try shmem zero-copy match before setting addr */ + if (ublk_support_shmem_zc(ubq) && blk_rq_has_data(req)) { + u32 buf_idx, buf_off; + + if (ublk_try_buf_match(ubq->dev, req, &buf_idx, &buf_off)) { + iod->op_flags |= UBLK_IO_F_SHMEM_ZC; + iod->addr = ublk_shmem_zc_addr(buf_idx, buf_off); + return; + } + } + + iod->addr = io->buf.addr; +} + #ifdef CONFIG_BLK_DEV_ZONED struct ublk_zoned_report_desc { @@ -652,8 +707,6 @@ out: static blk_status_t ublk_setup_iod_zoned(struct ublk_queue *ubq, struct request *req) { - struct ublksrv_io_desc *iod = ublk_get_iod(ubq, req->tag); - struct ublk_io *io = &ubq->ios[req->tag]; struct ublk_zoned_report_desc *desc; u32 ublk_op; @@ -683,9 +736,8 @@ static blk_status_t ublk_setup_iod_zoned(struct ublk_queue *ubq, ublk_op = desc->operation; switch (ublk_op) { case UBLK_IO_OP_REPORT_ZONES: - iod->op_flags = ublk_op | ublk_req_build_flags(req); - iod->nr_zones = desc->nr_zones; - iod->start_sector = desc->sector; + ublk_init_iod(ubq, req, ublk_op, desc->nr_zones, + desc->sector); return BLK_STS_OK; default: return BLK_STS_IOERR; @@ -697,11 +749,7 @@ static blk_status_t ublk_setup_iod_zoned(struct ublk_queue *ubq, return BLK_STS_IOERR; } - iod->op_flags = ublk_op | ublk_req_build_flags(req); - iod->nr_sectors = blk_rq_sectors(req); - iod->start_sector = blk_rq_pos(req); - iod->addr = io->buf.addr; - + ublk_init_iod(ubq, req, ublk_op, blk_rq_sectors(req), blk_rq_pos(req)); return BLK_STS_OK; } @@ -1189,11 +1237,6 @@ static inline struct ublk_queue *ublk_get_queue(struct ublk_device *dev, return dev->queues[qid]; } -static inline bool ublk_rq_has_data(const struct request *rq) -{ - return bio_has_data(rq->bio); -} - static inline struct ublksrv_io_desc * ublk_queue_cmd_buf(struct ublk_device *ub, int q_id) { @@ -1406,12 +1449,12 @@ static size_t ublk_copy_user_integrity(const struct request *req, static inline bool ublk_need_map_req(const struct request *req) { - return ublk_rq_has_data(req) && req_op(req) == REQ_OP_WRITE; + return blk_rq_has_data(req) && req_op(req) == REQ_OP_WRITE; } static inline bool ublk_need_unmap_req(const struct request *req) { - return ublk_rq_has_data(req) && + return blk_rq_has_data(req) && (req_op(req) == REQ_OP_READ || req_op(req) == REQ_OP_DRV_IN); } @@ -1460,41 +1503,8 @@ static unsigned int ublk_unmap_io(bool need_map, return rq_bytes; } -static inline unsigned int ublk_req_build_flags(struct request *req) -{ - unsigned flags = 0; - - if (req->cmd_flags & REQ_FAILFAST_DEV) - flags |= UBLK_IO_F_FAILFAST_DEV; - - if (req->cmd_flags & REQ_FAILFAST_TRANSPORT) - flags |= UBLK_IO_F_FAILFAST_TRANSPORT; - - if (req->cmd_flags & REQ_FAILFAST_DRIVER) - flags |= UBLK_IO_F_FAILFAST_DRIVER; - - if (req->cmd_flags & REQ_META) - flags |= UBLK_IO_F_META; - - if (req->cmd_flags & REQ_FUA) - flags |= UBLK_IO_F_FUA; - - if (req->cmd_flags & REQ_NOUNMAP) - flags |= UBLK_IO_F_NOUNMAP; - - if (req->cmd_flags & REQ_SWAP) - flags |= UBLK_IO_F_SWAP; - - if (blk_integrity_rq(req)) - flags |= UBLK_IO_F_INTEGRITY; - - return flags; -} - static blk_status_t ublk_setup_iod(struct ublk_queue *ubq, struct request *req) { - struct ublksrv_io_desc *iod = ublk_get_iod(ubq, req->tag); - struct ublk_io *io = &ubq->ios[req->tag]; u32 ublk_op; switch (req_op(req)) { @@ -1519,25 +1529,7 @@ static blk_status_t ublk_setup_iod(struct ublk_queue *ubq, struct request *req) return BLK_STS_IOERR; } - /* need to translate since kernel may change */ - iod->op_flags = ublk_op | ublk_req_build_flags(req); - iod->nr_sectors = blk_rq_sectors(req); - iod->start_sector = blk_rq_pos(req); - - /* Try shmem zero-copy match before setting addr */ - if (ublk_support_shmem_zc(ubq) && ublk_rq_has_data(req)) { - u32 buf_idx, buf_off; - - if (ublk_try_buf_match(ubq->dev, req, - &buf_idx, &buf_off)) { - iod->op_flags |= UBLK_IO_F_SHMEM_ZC; - iod->addr = ublk_shmem_zc_addr(buf_idx, buf_off); - return BLK_STS_OK; - } - } - - iod->addr = io->buf.addr; - + ublk_init_iod(ubq, req, ublk_op, blk_rq_sectors(req), blk_rq_pos(req)); return BLK_STS_OK; } @@ -1815,7 +1807,7 @@ static void ublk_dispatch_req(struct ublk_queue *ubq, struct request *req) if (!ublk_start_io(ubq, req, io)) return; - if (ublk_support_auto_buf_reg(ubq) && ublk_rq_has_data(req)) { + if (ublk_support_auto_buf_reg(ubq) && blk_rq_has_data(req)) { ublk_auto_buf_dispatch(ubq, req, io, io->cmd, issue_flags); } else { ublk_init_req_ref(ubq, io); @@ -1836,7 +1828,7 @@ static bool __ublk_batch_prep_dispatch(struct ublk_queue *ubq, if (!ublk_start_io(ubq, req, io)) return false; - if (ublk_support_auto_buf_reg(ubq) && ublk_rq_has_data(req)) { + if (ublk_support_auto_buf_reg(ubq) && blk_rq_has_data(req)) { res = ublk_auto_buf_register(ubq, req, io, cmd, data->issue_flags); @@ -2735,23 +2727,27 @@ static void ublk_start_cancel(struct ublk_device *ub) { struct gendisk *disk = ublk_get_disk(ub); - /* Our disk has been dead */ - if (!disk) - return; - mutex_lock(&ub->cancel_mutex); if (ub->canceling) goto out; - /* - * Now we are serialized with ublk_queue_rq() - * - * Make sure that ubq->canceling is set when queue is frozen, - * because ublk_queue_rq() has to rely on this flag for avoiding to - * touch completed uring_cmd - */ - blk_mq_quiesce_queue(disk->queue); - ublk_set_canceling(ub, true); - blk_mq_unquiesce_queue(disk->queue); + + if (disk) { + /* + * Quiesce to serialize with ublk_queue_rq(), ensuring + * ubq->canceling is visible when the queue resumes. + */ + blk_mq_quiesce_queue(disk->queue); + ublk_set_canceling(ub, true); + blk_mq_unquiesce_queue(disk->queue); + } else { + /* + * Disk not yet allocated by ublk_ctrl_start_dev(), so + * there is no request queue and ublk_queue_rq() cannot + * be running. Just set the flag; if start_dev proceeds + * later, new I/O will see canceling and be aborted. + */ + ublk_set_canceling(ub, true); + } out: mutex_unlock(&ub->cancel_mutex); ublk_put_disk(disk); @@ -3227,7 +3223,7 @@ ublk_daemon_register_io_buf(struct io_uring_cmd *cmd, return ublk_register_io_buf(cmd, ub, q_id, tag, io, index, issue_flags); - if (!ublk_dev_support_zero_copy(ub) || !ublk_rq_has_data(req)) + if (!ublk_dev_support_zero_copy(ub) || !blk_rq_has_data(req)) return -EINVAL; ret = io_buffer_register_bvec(cmd, req, ublk_io_release, index, @@ -3510,7 +3506,7 @@ static inline struct request *__ublk_check_and_get_req(struct ublk_device *ub, if (unlikely(!blk_mq_request_started(req) || req->tag != tag)) goto fail_put; - if (!ublk_rq_has_data(req)) + if (!blk_rq_has_data(req)) goto fail_put; return req; @@ -4085,7 +4081,7 @@ ublk_user_copy(struct kiocb *iocb, struct iov_iter *iter, int dir) return -EINVAL; req = io->req; - if (!ublk_rq_has_data(req)) + if (!blk_rq_has_data(req)) return -EINVAL; } else { req = __ublk_check_and_get_req(ub, q_id, tag, io); diff --git a/drivers/block/virtio_blk.c b/drivers/block/virtio_blk.c index b1c9a27fe00f..32bf3ba07a9d 100644 --- a/drivers/block/virtio_blk.c +++ b/drivers/block/virtio_blk.c @@ -689,6 +689,8 @@ static int virtblk_report_zones(struct gendisk *disk, sector_t sector, nz = min_t(u64, virtio64_to_cpu(vblk->vdev, report->nr_zones), nr_zones); + nz = min_t(u64, nz, + (buflen - sizeof(*report)) / sizeof(report->zones[0])); if (!nz) break; diff --git a/drivers/md/dm-raid.c b/drivers/md/dm-raid.c index c5dc083c7244..8f5a5e1342a9 100644 --- a/drivers/md/dm-raid.c +++ b/drivers/md/dm-raid.c @@ -3831,6 +3831,7 @@ static void raid_presuspend(struct dm_target *ti) * resume, raid_postsuspend() is too late. */ set_bit(RT_FLAG_RS_FROZEN, &rs->runtime_flags); + set_bit(MD_DM_SUSPENDING, &mddev->flags); if (!reshape_interrupted(mddev)) return; @@ -3847,13 +3848,16 @@ static void raid_presuspend(struct dm_target *ti) static void raid_presuspend_undo(struct dm_target *ti) { struct raid_set *rs = ti->private; + struct mddev *mddev = &rs->md; + clear_bit(MD_DM_SUSPENDING, &mddev->flags); clear_bit(RT_FLAG_RS_FROZEN, &rs->runtime_flags); } static void raid_postsuspend(struct dm_target *ti) { struct raid_set *rs = ti->private; + struct mddev *mddev = &rs->md; if (!test_and_set_bit(RT_FLAG_RS_SUSPENDED, &rs->runtime_flags)) { /* @@ -3864,6 +3868,8 @@ static void raid_postsuspend(struct dm_target *ti) mddev_suspend(&rs->md, false); rs->md.ro = MD_RDONLY; } + clear_bit(MD_DM_SUSPENDING, &mddev->flags); + } static void attempt_restore_of_faulty_devices(struct raid_set *rs) diff --git a/drivers/md/dm-rq.c b/drivers/md/dm-rq.c index 9703b3ae364e..9a386254d836 100644 --- a/drivers/md/dm-rq.c +++ b/drivers/md/dm-rq.c @@ -462,7 +462,7 @@ static void dm_start_request(struct mapped_device *md, struct request *orig) } static int dm_mq_init_request(struct blk_mq_tag_set *set, struct request *rq, - unsigned int hctx_idx, unsigned int numa_node) + unsigned int hctx_idx, int numa_node) { struct mapped_device *md = set->driver_data; struct dm_rq_target_io *tio = blk_mq_rq_to_pdu(rq); diff --git a/drivers/md/md.c b/drivers/md/md.c index 8b568eee8743..096bb64e87bd 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -395,17 +395,24 @@ static bool is_suspended(struct mddev *mddev, struct bio *bio) bool md_handle_request(struct mddev *mddev, struct bio *bio) { check_suspended: - if (is_suspended(mddev, bio)) { - /* Bail out if REQ_NOWAIT is set for the bio */ - if (bio->bi_opf & REQ_NOWAIT) { - bio_wouldblock_error(bio); - return true; + if (unlikely(md_cloned_bio(mddev, bio))) { + /* + * This bio is an MD cloned bio and already holds an + * active_io reference, so percpu_ref_get() is safe here. + */ + percpu_ref_get(&mddev->active_io); + } else { + if (is_suspended(mddev, bio)) { + /* Bail out if REQ_NOWAIT is set for the bio */ + if (bio->bi_opf & REQ_NOWAIT) { + bio_wouldblock_error(bio); + return true; + } + wait_event(mddev->sb_wait, !is_suspended(mddev, bio)); } - wait_event(mddev->sb_wait, !is_suspended(mddev, bio)); + if (!percpu_ref_tryget_live(&mddev->active_io)) + goto check_suspended; } - if (!percpu_ref_tryget_live(&mddev->active_io)) - goto check_suspended; - if (!mddev->pers->make_request(mddev, bio)) { percpu_ref_put(&mddev->active_io); if (mddev_is_dm(mddev) && mddev->pers->prepare_suspend) @@ -4414,9 +4421,10 @@ raid_disks_store(struct mddev *mddev, const char *buf, size_t len) err = mddev_suspend_and_lock(mddev); if (err) return err; - if (mddev->pers) - err = update_raid_disks(mddev, n); - else if (mddev->reshape_position != MaxSector) { + if (mddev->pers) { + if (n != mddev->raid_disks) + err = update_raid_disks(mddev, n); + } else if (mddev->reshape_position != MaxSector) { struct md_rdev *rdev; int olddisks = mddev->raid_disks - mddev->delta_disks; diff --git a/drivers/md/md.h b/drivers/md/md.h index 52c378086046..d8daf0f75cbb 100644 --- a/drivers/md/md.h +++ b/drivers/md/md.h @@ -346,6 +346,7 @@ struct md_cluster_operations; * @MD_HAS_SUPERBLOCK: There is persistence sb in member disks. * @MD_FAILLAST_DEV: Allow last rdev to be removed. * @MD_SERIALIZE_POLICY: Enforce write IO is not reordered, just used by raid1. + * @MD_DM_SUSPENDING: This DM raid device is suspending. * * change UNSUPPORTED_MDDEV_FLAGS for each array type if new flag is added */ @@ -365,6 +366,7 @@ enum mddev_flags { MD_HAS_SUPERBLOCK, MD_FAILLAST_DEV, MD_SERIALIZE_POLICY, + MD_DM_SUSPENDING, }; enum mddev_sb_flags { @@ -1042,6 +1044,11 @@ void mddev_update_io_opt(struct mddev *mddev, unsigned int nr_stripes); extern const struct block_device_operations md_fops; +static inline bool md_cloned_bio(struct mddev *mddev, struct bio *bio) +{ + return bio->bi_pool == &mddev->io_clone_set; +} + /* * MD devices can be used undeneath by DM, in which case ->gendisk is NULL. */ diff --git a/drivers/md/raid0.c b/drivers/md/raid0.c index 5e38a51e349a..2c000b3a5f49 100644 --- a/drivers/md/raid0.c +++ b/drivers/md/raid0.c @@ -14,6 +14,7 @@ #include <linux/seq_file.h> #include <linux/module.h> #include <linux/slab.h> +#include <linux/string_choices.h> #include <trace/events/block.h> #include "md.h" #include "raid0.h" @@ -43,7 +44,7 @@ static void dump_zones(struct mddev *mddev) int raid_disks = conf->strip_zone[0].nb_dev; pr_debug("md: RAID0 configuration for %s - %d zone%s\n", mdname(mddev), - conf->nr_strip_zones, conf->nr_strip_zones==1?"":"s"); + conf->nr_strip_zones, str_plural(conf->nr_strip_zones)); for (j = 0; j < conf->nr_strip_zones; j++) { char line[200]; int len = 0; @@ -392,6 +393,7 @@ static int raid0_set_limits(struct mddev *mddev) lim.io_opt = lim.io_min * mddev->raid_disks; lim.chunk_sectors = mddev->chunk_sectors; lim.features |= BLK_FEAT_ATOMIC_WRITES; + lim.features |= BLK_FEAT_PCI_P2PDMA; err = mddev_stack_rdev_limits(mddev, &lim, MDDEV_STACK_INTEGRITY); if (err) return err; diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index 64d970e2ef50..5b9368bd9e70 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -1343,11 +1343,18 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio, bool r1bio_existed = !!r1_bio; /* - * If r1_bio is set, we are blocking the raid1d thread - * so there is a tiny risk of deadlock. So ask for + * An md cloned bio indicates we are in the error path. + * This is more reliable than checking r1_bio, which might + * be NULL even in the error path if a failed bio was split. + */ + bool err_path = md_cloned_bio(mddev, bio); + + /* + * If we are in the error path, we are blocking the raid1d + * thread so there is a tiny risk of deadlock. So ask for * emergency memory if needed. */ - gfp_t gfp = r1_bio ? (GFP_NOIO | __GFP_HIGH) : GFP_NOIO; + gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) : GFP_NOIO; /* * Still need barrier for READ in case that whole @@ -1411,7 +1418,7 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio, } r1_bio->read_disk = rdisk; - if (!r1bio_existed) { + if (likely(!md_cloned_bio(mddev, bio))) { md_account_bio(mddev, &bio); r1_bio->master_bio = bio; } @@ -1596,8 +1603,10 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio, * complexity of supporting that is not worth * the benefit. */ - if (bio->bi_opf & REQ_ATOMIC) + if (bio->bi_opf & REQ_ATOMIC) { + rdev_dec_pending(rdev, mddev); goto err_handle; + } good_sectors = first_bad - r1_bio->sector; if (good_sectors < max_sectors) @@ -2411,11 +2420,6 @@ static void fix_read_error(struct r1conf *conf, struct r1bio *r1_bio) struct mddev *mddev = conf->mddev; struct md_rdev *rdev = conf->mirrors[read_disk].rdev; - if (exceed_read_errors(mddev, rdev)) { - r1_bio->bios[r1_bio->read_disk] = IO_BLOCKED; - return; - } - while(sectors) { int s = sectors; int d = read_disk; @@ -2627,35 +2631,36 @@ static void handle_write_finished(struct r1conf *conf, struct r1bio *r1_bio) static void handle_read_error(struct r1conf *conf, struct r1bio *r1_bio) { + struct md_rdev *rdev = conf->mirrors[r1_bio->read_disk].rdev; + struct bio *bio = r1_bio->bios[r1_bio->read_disk]; struct mddev *mddev = conf->mddev; - struct bio *bio; - struct md_rdev *rdev; sector_t sector; clear_bit(R1BIO_ReadError, &r1_bio->state); - /* we got a read error. Maybe the drive is bad. Maybe just - * the block and we can fix it. - * We freeze all other IO, and try reading the block from - * other devices. When we find one, we re-write - * and check it that fixes the read error. - * This is all done synchronously while the array is - * frozen - */ - bio = r1_bio->bios[r1_bio->read_disk]; bio_put(bio); r1_bio->bios[r1_bio->read_disk] = NULL; - rdev = conf->mirrors[r1_bio->read_disk].rdev; - if (mddev->ro == 0 - && !test_bit(FailFast, &rdev->flags)) { - freeze_array(conf, 1); - fix_read_error(conf, r1_bio); - unfreeze_array(conf); - } else if (mddev->ro == 0 && test_bit(FailFast, &rdev->flags)) { + /* + * We got a read error. Maybe the drive is bad. Maybe just the block + * and we can fix it. + * + * If allowed, freeze all other IO, and try reading the block from other + * devices. If we find one, we re-write and check it that fixes the + * read error. This is all done synchronously while the array is + * frozen. + */ + if (mddev->ro) { + r1_bio->bios[r1_bio->read_disk] = IO_BLOCKED; + } else if (test_bit(FailFast, &rdev->flags)) { md_error(mddev, rdev); } else { - r1_bio->bios[r1_bio->read_disk] = IO_BLOCKED; + freeze_array(conf, 1); + if (exceed_read_errors(mddev, rdev)) + r1_bio->bios[r1_bio->read_disk] = IO_BLOCKED; + else + fix_read_error(conf, r1_bio); + unfreeze_array(conf); } rdev_dec_pending(rdev, conf->mddev); @@ -3208,6 +3213,7 @@ static int raid1_set_limits(struct mddev *mddev) lim.max_hw_wzeroes_unmap_sectors = 0; lim.logical_block_size = mddev->logical_block_size; lim.features |= BLK_FEAT_ATOMIC_WRITES; + lim.features |= BLK_FEAT_PCI_P2PDMA; err = mddev_stack_rdev_limits(mddev, &lim, MDDEV_STACK_INTEGRITY); if (err) return err; diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index 39085e7dd6d2..cee5a253a281 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -1146,7 +1146,7 @@ static bool regular_request_wait(struct mddev *mddev, struct r10conf *conf, } static void raid10_read_request(struct mddev *mddev, struct bio *bio, - struct r10bio *r10_bio, bool io_accounting) + struct r10bio *r10_bio) { struct r10conf *conf = mddev->private; struct bio *read_bio; @@ -1155,7 +1155,20 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio, char b[BDEVNAME_SIZE]; int slot = r10_bio->read_slot; struct md_rdev *err_rdev = NULL; - gfp_t gfp = GFP_NOIO; + + /* + * An md cloned bio indicates we are in the error path. + * This is more reliable than checking slot, which might + * be -1 even in the error path if a failed bio was split. + */ + bool err_path = md_cloned_bio(mddev, bio); + + /* + * If we are in the error path, we are blocking the raid10d + * thread so there is a tiny risk of deadlock. So ask for + * emergency memory if needed. + */ + gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) : GFP_NOIO; if (slot >= 0 && r10_bio->devs[slot].rdev) { /* @@ -1166,11 +1179,6 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio, * we lose the device name in error messages. */ int disk; - /* - * As we are blocking raid10, it is a little safer to - * use __GFP_HIGH. - */ - gfp = GFP_NOIO | __GFP_HIGH; disk = r10_bio->devs[slot].devnum; err_rdev = conf->mirrors[disk].rdev; @@ -1218,7 +1226,7 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio, } slot = r10_bio->read_slot; - if (io_accounting) { + if (likely(!md_cloned_bio(mddev, bio))) { md_account_bio(mddev, &bio); r10_bio->master_bio = bio; } @@ -1544,7 +1552,7 @@ static void __make_request(struct mddev *mddev, struct bio *bio, int sectors) conf->geo.raid_disks); if (bio_data_dir(bio) == READ) - raid10_read_request(mddev, bio, r10_bio, true); + raid10_read_request(mddev, bio, r10_bio); else raid10_write_request(mddev, bio, r10_bio); } @@ -1727,6 +1735,7 @@ retry_discard: r10_bio->mddev = mddev; r10_bio->state = 0; r10_bio->sectors = 0; + r10_bio->read_slot = -1; memset(r10_bio->devs, 0, sizeof(r10_bio->devs[0]) * geo->raid_disks); wait_blocked_dev(mddev, r10_bio); @@ -2858,7 +2867,7 @@ static void handle_read_error(struct mddev *mddev, struct r10bio *r10_bio) rdev_dec_pending(rdev, mddev); r10_bio->state = 0; - raid10_read_request(mddev, r10_bio->master_bio, r10_bio, false); + raid10_read_request(mddev, r10_bio->master_bio, r10_bio); /* * allow_barrier after re-submit to ensure no sync io * can be issued while regular io pending. @@ -3941,6 +3950,7 @@ static int raid10_set_queue_limits(struct mddev *mddev) lim.chunk_sectors = mddev->chunk_sectors; lim.io_opt = lim.io_min * raid10_nr_stripes(conf); lim.features |= BLK_FEAT_ATOMIC_WRITES; + lim.features |= BLK_FEAT_PCI_P2PDMA; err = mddev_stack_rdev_limits(mddev, &lim, MDDEV_STACK_INTEGRITY); if (err) return err; diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index 0d76e82f4506..65ae7d8930fc 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -6042,8 +6042,11 @@ out_release: raid5_release_stripe(sh); out: if (ret == STRIPE_SCHEDULE_AND_RETRY && reshape_interrupted(mddev)) { - bi->bi_status = BLK_STS_RESOURCE; - ret = STRIPE_WAIT_RESHAPE; + if (!mddev_is_dm(mddev) || + test_bit(MD_DM_SUSPENDING, &mddev->flags)) { + bi->bi_status = BLK_STS_RESOURCE; + ret = STRIPE_WAIT_RESHAPE; + } pr_err_ratelimited("dm-raid456: io across reshape position while reshape can't make progress"); } return ret; diff --git a/drivers/mmc/core/queue.c b/drivers/mmc/core/queue.c index c9028e4a7e56..5d187e063da8 100644 --- a/drivers/mmc/core/queue.c +++ b/drivers/mmc/core/queue.c @@ -208,7 +208,7 @@ static unsigned short mmc_get_max_segments(struct mmc_host *host) } static int mmc_mq_init_request(struct blk_mq_tag_set *set, struct request *req, - unsigned int hctx_idx, unsigned int numa_node) + unsigned int hctx_idx, int numa_node) { struct mmc_queue_req *mq_rq = req_to_mmc_queue_req(req); struct mmc_queue *mq = set->driver_data; diff --git a/drivers/mtd/ubi/block.c b/drivers/mtd/ubi/block.c index 8880a783c3bc..29c0d6941a81 100644 --- a/drivers/mtd/ubi/block.c +++ b/drivers/mtd/ubi/block.c @@ -312,7 +312,7 @@ static blk_status_t ubiblock_queue_rq(struct blk_mq_hw_ctx *hctx, static int ubiblock_init_request(struct blk_mq_tag_set *set, struct request *req, unsigned int hctx_idx, - unsigned int numa_node) + int numa_node) { struct ubiblock_pdu *pdu = blk_mq_rq_to_pdu(req); diff --git a/drivers/nvme/host/apple.c b/drivers/nvme/host/apple.c index c692fc73babf..f9327feb87d0 100644 --- a/drivers/nvme/host/apple.c +++ b/drivers/nvme/host/apple.c @@ -819,7 +819,7 @@ static int apple_nvme_init_hctx(struct blk_mq_hw_ctx *hctx, void *data, static int apple_nvme_init_request(struct blk_mq_tag_set *set, struct request *req, unsigned int hctx_idx, - unsigned int numa_node) + int numa_node) { struct apple_nvme_queue *q = set->driver_data; struct apple_nvme *anv = queue_to_apple_nvme(q); @@ -858,7 +858,7 @@ static void apple_nvme_disable(struct apple_nvme *anv, bool shutdown) * doing a safe shutdown. */ if (!dead && shutdown && freeze) - nvme_wait_freeze_timeout(&anv->ctrl, NVME_IO_TIMEOUT); + nvme_wait_freeze_timeout(&anv->ctrl); nvme_quiesce_io_queues(&anv->ctrl); diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index c3032d6ad6b1..3b7a8f7a3542 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -323,6 +323,7 @@ static void nvme_retry_req(struct request *req) { unsigned long delay = 0; u16 crd; + struct nvme_ns *ns = req->q->queuedata; /* The mask and shift result must be <= 3 */ crd = (nvme_req(req)->status & NVME_STATUS_CRD) >> 11; @@ -330,6 +331,9 @@ static void nvme_retry_req(struct request *req) delay = nvme_req(req)->ctrl->crdt[crd - 1] * 100; nvme_req(req)->retries++; + if (ns) + atomic_long_inc(&ns->retries); + blk_mq_requeue_request(req, false); blk_mq_delay_kick_requeue_list(req->q, delay); } @@ -434,11 +438,19 @@ static inline void nvme_end_req_zoned(struct request *req) static inline void __nvme_end_req(struct request *req) { - if (unlikely(nvme_req(req)->status && !(req->rq_flags & RQF_QUIET))) { + struct nvme_ns *ns = req->q->queuedata; + struct nvme_request *nr = nvme_req(req); + + if (unlikely(nr->status && !(req->rq_flags & RQF_QUIET))) { if (blk_rq_is_passthrough(req)) nvme_log_err_passthru(req); else nvme_log_error(req); + + if (ns) + atomic_long_inc(&ns->errors); + else + atomic_long_inc(&nr->ctrl->errors); } nvme_end_req_zoned(req); nvme_trace_bio_complete(req); @@ -584,6 +596,7 @@ bool nvme_change_ctrl_state(struct nvme_ctrl *ctrl, case NVME_CTRL_NEW: case NVME_CTRL_LIVE: changed = true; + atomic_long_inc(&ctrl->nr_reset); fallthrough; default: break; @@ -729,10 +742,8 @@ void nvme_init_request(struct request *req, struct nvme_command *cmd) struct nvme_ns *ns = req->q->disk->private_data; logging_enabled = ns->head->passthru_err_log_enabled; - req->timeout = NVME_IO_TIMEOUT; } else { /* no queuedata implies admin queue */ logging_enabled = nr->ctrl->passthru_err_log_enabled; - req->timeout = NVME_ADMIN_TIMEOUT; } if (!logging_enabled) @@ -2263,7 +2274,7 @@ static int nvme_query_fdp_granularity(struct nvme_ctrl *ctrl, } n = le16_to_cpu(h->numfdpc) + 1; - if (fdp_idx > n) { + if (fdp_idx >= n) { dev_warn(ctrl->device, "FDP index:%d out of range:%d\n", fdp_idx, n); /* Proceed without registering FDP streams */ @@ -2275,14 +2286,16 @@ static int nvme_query_fdp_granularity(struct nvme_ctrl *ctrl, desc = log; end = log + size - sizeof(*h); for (i = 0; i < fdp_idx; i++) { - log += le16_to_cpu(desc->dsze); - desc = log; - if (log >= end) { + u16 dsze = le16_to_cpu(desc->dsze); + + if (!dsze || log + dsze > end) { dev_warn(ctrl->device, - "FDP invalid config descriptor list\n"); + "FDP invalid config descriptor at index %d\n", i); ret = 0; goto out; } + log += dsze; + desc = log; } if (le32_to_cpu(desc->nrg) > 1) { @@ -2409,12 +2422,22 @@ static int nvme_update_ns_info_block(struct nvme_ns *ns, goto out; } + if (id->lbaf[lbaf].ds < SECTOR_SHIFT || + check_shl_overflow(le64_to_cpu(id->nsze), + id->lbaf[lbaf].ds - SECTOR_SHIFT, + &capacity)) { + dev_warn_once(ns->ctrl->device, + "invalid LBA data size %u, skipping namespace\n", + id->lbaf[lbaf].ds); + ret = -ENODEV; + goto out; + } + lim = queue_limits_start_update(ns->disk->queue); memflags = blk_mq_freeze_queue(ns->disk->queue); ns->head->lba_shift = id->lbaf[lbaf].ds; ns->head->nuse = le64_to_cpu(id->nuse); - capacity = nvme_lba_to_sect(ns->head, le64_to_cpu(id->nsze)); nvme_set_ctrl_limits(ns->ctrl, &lim, false); nvme_configure_metadata(ns->ctrl, ns->head, id, nvm, info); nvme_set_chunk_sectors(ns, id, &lim); @@ -2483,6 +2506,14 @@ out: return ret; } +static void nvme_stack_zone_resources(struct queue_limits *t, + const struct queue_limits *b) +{ + t->max_open_zones = min_not_zero(t->max_open_zones, b->max_open_zones); + t->max_active_zones = + min_not_zero(t->max_active_zones, b->max_active_zones); +} + static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_ns_info *info) { bool unsupported = false; @@ -2549,6 +2580,8 @@ static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_ns_info *info) lim.io_opt = ns_lim->io_opt; queue_limits_stack_bdev(&lim, ns->disk->part0, 0, ns->head->disk->disk_name); + if (lim.features & BLK_FEAT_ZONED) + nvme_stack_zone_resources(&lim, ns_lim); if (unsupported) ns->head->disk->flags |= GENHD_FL_HIDDEN; else @@ -2559,7 +2592,7 @@ static int nvme_update_ns_info(struct nvme_ns *ns, struct nvme_ns_info *info) set_capacity_and_notify(ns->head->disk, get_capacity(ns->disk)); set_disk_ro(ns->head->disk, nvme_ns_is_readonly(ns, info)); - nvme_mpath_revalidate_paths(ns); + nvme_mpath_revalidate_paths(ns->head); blk_mq_unfreeze_queue(ns->head->disk->queue, memflags); } @@ -3926,7 +3959,7 @@ static struct nvme_ns_head *nvme_alloc_ns_head(struct nvme_ctrl *ctrl, int ret = -ENOMEM; #ifdef CONFIG_NVME_MULTIPATH - size += num_possible_nodes() * sizeof(struct nvme_ns *); + size += nr_node_ids * sizeof(struct nvme_ns *); #endif head = kzalloc(size, GFP_KERNEL); @@ -4209,6 +4242,7 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info) mutex_unlock(&ctrl->namespaces_lock); goto out_unlink_ns; } + blk_queue_rq_timeout(ns->queue, ctrl->io_timeout); nvme_ns_add_to_ctrl_list(ns); mutex_unlock(&ctrl->namespaces_lock); synchronize_srcu(&ctrl->srcu); @@ -4894,12 +4928,7 @@ int nvme_alloc_admin_tag_set(struct nvme_ctrl *ctrl, struct blk_mq_tag_set *set, if (ret) return ret; - /* - * If a previous admin queue exists (e.g., from before a reset), - * put it now before allocating a new one to avoid orphaning it. - */ - if (ctrl->admin_q) - blk_put_queue(ctrl->admin_q); + WARN_ON_ONCE(ctrl->admin_q); ctrl->admin_q = blk_mq_alloc_queue(set, NULL, NULL); if (IS_ERR(ctrl->admin_q)) { @@ -4937,10 +4966,8 @@ void nvme_remove_admin_tag_set(struct nvme_ctrl *ctrl) */ nvme_stop_keep_alive(ctrl); blk_mq_destroy_queue(ctrl->admin_q); - if (ctrl->ops->flags & NVME_F_FABRICS) { + if (ctrl->fabrics_q) blk_mq_destroy_queue(ctrl->fabrics_q); - blk_put_queue(ctrl->fabrics_q); - } blk_mq_free_tag_set(ctrl->admin_tagset); } EXPORT_SYMBOL_GPL(nvme_remove_admin_tag_set); @@ -5082,6 +5109,8 @@ static void nvme_free_ctrl(struct device *dev) if (ctrl->admin_q) blk_put_queue(ctrl->admin_q); + if (ctrl->fabrics_q) + blk_put_queue(ctrl->fabrics_q); if (!subsys || ctrl->instance != subsys->instance) ida_free(&nvme_instance_ida, ctrl->instance); nvme_free_cels(ctrl); @@ -5146,6 +5175,8 @@ int nvme_init_ctrl(struct nvme_ctrl *ctrl, struct device *dev, memset(&ctrl->ka_cmd, 0, sizeof(ctrl->ka_cmd)); ctrl->ka_cmd.common.opcode = nvme_admin_keep_alive; ctrl->ka_last_check_time = jiffies; + ctrl->admin_timeout = NVME_ADMIN_TIMEOUT; + ctrl->io_timeout = NVME_IO_TIMEOUT; BUILD_BUG_ON(NVME_DSM_MAX_RANGES * sizeof(struct nvme_dsm_range) > PAGE_SIZE); @@ -5252,8 +5283,9 @@ void nvme_unfreeze(struct nvme_ctrl *ctrl) } EXPORT_SYMBOL_GPL(nvme_unfreeze); -int nvme_wait_freeze_timeout(struct nvme_ctrl *ctrl, long timeout) +int nvme_wait_freeze_timeout(struct nvme_ctrl *ctrl) { + long timeout = ctrl->io_timeout; struct nvme_ns *ns; int srcu_idx; diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c index e4f4528fe2a2..2c9a6d3c9797 100644 --- a/drivers/nvme/host/fc.c +++ b/drivers/nvme/host/fc.c @@ -2109,7 +2109,7 @@ out_on_error: static int nvme_fc_init_request(struct blk_mq_tag_set *set, struct request *rq, - unsigned int hctx_idx, unsigned int numa_node) + unsigned int hctx_idx, int numa_node) { struct nvme_fc_ctrl *ctrl = to_fc_ctrl(set->driver_data); struct nvme_fcp_op_w_sgl *op = blk_mq_rq_to_pdu(rq); @@ -3148,6 +3148,8 @@ nvme_fc_create_association(struct nvme_fc_ctrl *ctrl) goto out_term_aen_ops; } + /* accumulate reconnect attempts before resetting it to zero */ + atomic_long_add(ctrl->ctrl.nr_reconnects, &ctrl->ctrl.acc_reconnects); ctrl->ctrl.nr_reconnects = 0; nvme_start_ctrl(&ctrl->ctrl); @@ -3470,6 +3472,7 @@ nvme_fc_alloc_ctrl(struct device *dev, struct nvmf_ctrl_options *opts, ctrl->ctrl.opts = opts; ctrl->ctrl.nr_reconnects = 0; + atomic_long_set(&ctrl->ctrl.acc_reconnects, 0); INIT_LIST_HEAD(&ctrl->ctrl_list); ctrl->lport = lport; ctrl->rport = rport; diff --git a/drivers/nvme/host/ioctl.c b/drivers/nvme/host/ioctl.c index 08889b20e5d8..664216eece4a 100644 --- a/drivers/nvme/host/ioctl.c +++ b/drivers/nvme/host/ioctl.c @@ -102,8 +102,17 @@ static struct request *nvme_alloc_user_request(struct request_queue *q, struct nvme_command *cmd, blk_opf_t rq_flags, blk_mq_req_flags_t blk_flags) { + struct nvme_ns *ns = q->queuedata; struct request *req; + /* + * The NVME_MPATH flag is set only for IO commands sent to a namespace + * with a multipath enabled head. The request is not eligible for + * failover as passthrough requests also append REQ_FAILFAST_DRIVER. + */ + if (ns && nvme_ns_head_multipath(ns->head)) + rq_flags |= REQ_NVME_MPATH; + req = blk_mq_alloc_request(q, nvme_req_op(cmd) | rq_flags, blk_flags); if (IS_ERR(req)) return req; diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c index 263161cb8ac0..e033ede953cc 100644 --- a/drivers/nvme/host/multipath.c +++ b/drivers/nvme/host/multipath.c @@ -73,19 +73,29 @@ static const char *nvme_iopolicy_names[] = { static int iopolicy = NVME_IOPOLICY_NUMA; +static int nvme_iopolicy_parse(const char *str) +{ + int i; + + for (i = 0; i < ARRAY_SIZE(nvme_iopolicy_names); i++) { + if (sysfs_streq(str, nvme_iopolicy_names[i])) + return i; + } + return -EINVAL; +} + static int nvme_set_iopolicy(const char *val, const struct kernel_param *kp) { + int policy; + if (!val) return -EINVAL; - if (!strncmp(val, "numa", 4)) - iopolicy = NVME_IOPOLICY_NUMA; - else if (!strncmp(val, "round-robin", 11)) - iopolicy = NVME_IOPOLICY_RR; - else if (!strncmp(val, "queue-depth", 11)) - iopolicy = NVME_IOPOLICY_QD; - else - return -EINVAL; + policy = nvme_iopolicy_parse(val); + if (policy < 0) + return policy; + + iopolicy = policy; return 0; } @@ -142,6 +152,7 @@ void nvme_failover_req(struct request *req) struct bio *bio; nvme_mpath_clear_current_path(ns); + atomic_long_inc(&ns->failover); /* * If we got back an ANA error, we know the controller is alive but not @@ -175,9 +186,12 @@ void nvme_mpath_start_request(struct request *rq) nvme_req(rq)->flags |= NVME_MPATH_CNT_ACTIVE; } - if (!blk_queue_io_stat(disk->queue) || blk_rq_is_passthrough(rq) || + if (!blk_queue_io_stat(disk->queue) || (nvme_req(rq)->flags & NVME_MPATH_IO_STATS)) return; + if (blk_rq_is_passthrough(rq) && + !blk_rq_passthrough_stats(rq, disk->queue)) + return; nvme_req(rq)->flags |= NVME_MPATH_IO_STATS; nvme_req(rq)->start_time = bdev_start_io_acct(disk->part0, req_op(rq), @@ -254,10 +268,10 @@ void nvme_mpath_clear_ctrl_paths(struct nvme_ctrl *ctrl) srcu_read_unlock(&ctrl->srcu, srcu_idx); } -void nvme_mpath_revalidate_paths(struct nvme_ns *ns) +void nvme_mpath_revalidate_paths(struct nvme_ns_head *head) { - struct nvme_ns_head *head = ns->head; sector_t capacity = get_capacity(head->disk); + struct nvme_ns *ns; int node; int srcu_idx; @@ -511,6 +525,12 @@ static void nvme_ns_head_submit_bio(struct bio *bio) ns = nvme_find_path(head); if (likely(ns)) { bio_set_dev(bio, ns->disk->part0); + /* + * Use BIO_REMAPPED to skip bio_check_eod() when this bio + * enters submit_bio_noacct() for the per-path device. The EOD + * check already passed on the multipath head. + */ + bio_set_flag(bio, BIO_REMAPPED); bio->bi_opf |= REQ_NVME_MPATH; trace_block_bio_remap(bio, disk_devt(ns->head->disk), bio->bi_iter.bi_sector); @@ -521,10 +541,12 @@ static void nvme_ns_head_submit_bio(struct bio *bio) spin_lock_irq(&head->requeue_lock); bio_list_add(&head->requeue_list, bio); spin_unlock_irq(&head->requeue_lock); + atomic_long_inc(&head->io_requeue_no_usable_path_count); } else { dev_warn_ratelimited(dev, "no available path - failing I/O\n"); bio_io_error(bio); + atomic_long_inc(&head->io_fail_no_available_path_count); } srcu_read_unlock(&head->srcu, srcu_idx); @@ -730,7 +752,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head) blk_set_stacking_limits(&lim); lim.dma_alignment = 3; lim.features |= BLK_FEAT_IO_STAT | BLK_FEAT_NOWAIT | - BLK_FEAT_POLL | BLK_FEAT_ATOMIC_WRITES; + BLK_FEAT_POLL | BLK_FEAT_ATOMIC_WRITES | BLK_FEAT_PCI_P2PDMA; if (head->ids.csi == NVME_CSI_ZNS) lim.features |= BLK_FEAT_ZONED; @@ -1039,16 +1061,14 @@ static ssize_t nvme_subsys_iopolicy_store(struct device *dev, { struct nvme_subsystem *subsys = container_of(dev, struct nvme_subsystem, dev); - int i; + int policy; - for (i = 0; i < ARRAY_SIZE(nvme_iopolicy_names); i++) { - if (sysfs_streq(buf, nvme_iopolicy_names[i])) { - nvme_subsys_iopolicy_update(subsys, i); - return count; - } - } + policy = nvme_iopolicy_parse(buf); + if (policy < 0) + return policy; - return -EINVAL; + nvme_subsys_iopolicy_update(subsys, policy); + return count; } SUBSYS_ATTR_RW(iopolicy, S_IRUGO | S_IWUSR, nvme_subsys_iopolicy_show, nvme_subsys_iopolicy_store); @@ -1151,6 +1171,90 @@ static ssize_t delayed_removal_secs_store(struct device *dev, DEVICE_ATTR_RW(delayed_removal_secs); +static ssize_t multipath_failover_count_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nvme_ns *ns = nvme_get_ns_from_dev(dev); + + return sysfs_emit(buf, "%lu\n", atomic_long_read(&ns->failover)); +} + +static ssize_t multipath_failover_count_store(struct device *dev, + struct device_attribute *attr, const char *buf, size_t count) +{ + unsigned long failover; + int ret; + struct nvme_ns *ns = nvme_get_ns_from_dev(dev); + + ret = kstrtoul(buf, 0, &failover); + if (ret) + return -EINVAL; + + atomic_long_set(&ns->failover, failover); + + return count; +} + +DEVICE_ATTR_RW(multipath_failover_count); + +static ssize_t io_requeue_no_usable_path_count_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct gendisk *disk = dev_to_disk(dev); + struct nvme_ns_head *head = disk->private_data; + + return sysfs_emit(buf, "%lu\n", + atomic_long_read(&head->io_requeue_no_usable_path_count)); +} + +static ssize_t io_requeue_no_usable_path_count_store(struct device *dev, + struct device_attribute *attr, const char *buf, size_t count) +{ + int err; + unsigned long requeue_cnt; + struct gendisk *disk = dev_to_disk(dev); + struct nvme_ns_head *head = disk->private_data; + + err = kstrtoul(buf, 0, &requeue_cnt); + if (err) + return -EINVAL; + + atomic_long_set(&head->io_requeue_no_usable_path_count, requeue_cnt); + + return count; +} + +DEVICE_ATTR_RW(io_requeue_no_usable_path_count); + +static ssize_t io_fail_no_available_path_count_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct gendisk *disk = dev_to_disk(dev); + struct nvme_ns_head *head = disk->private_data; + + return sysfs_emit(buf, "%lu\n", + atomic_long_read(&head->io_fail_no_available_path_count)); +} + +static ssize_t io_fail_no_available_path_count_store(struct device *dev, + struct device_attribute *attr, const char *buf, size_t count) +{ + int err; + unsigned long fail_cnt; + struct gendisk *disk = dev_to_disk(dev); + struct nvme_ns_head *head = disk->private_data; + + err = kstrtoul(buf, 0, &fail_cnt); + if (err) + return -EINVAL; + + atomic_long_set(&head->io_fail_no_available_path_count, fail_cnt); + + return count; +} + +DEVICE_ATTR_RW(io_fail_no_available_path_count); + static int nvme_lookup_ana_group_desc(struct nvme_ctrl *ctrl, struct nvme_ana_group_desc *desc, void *data) { diff --git a/drivers/nvme/host/nvme.h b/drivers/nvme/host/nvme.h index ccd5e05dac98..b367c67dcb37 100644 --- a/drivers/nvme/host/nvme.h +++ b/drivers/nvme/host/nvme.h @@ -370,6 +370,8 @@ struct nvme_ctrl { u16 mtfa; u32 ctrl_config; u32 queue_count; + u32 admin_timeout; + u32 io_timeout; u64 cap; u32 max_hw_sectors; @@ -413,6 +415,8 @@ struct nvme_ctrl { unsigned long ka_last_check_time; struct work_struct fw_act_work; unsigned long events; + atomic_long_t errors; + atomic_long_t nr_reset; #ifdef CONFIG_NVME_MULTIPATH /* asymmetric namespace access: */ @@ -454,6 +458,8 @@ struct nvme_ctrl { u16 icdoff; u16 maxcmd; int nr_reconnects; + /* accumulate reconenct attempts, as nr_reconnects can reset to zero */ + atomic_long_t acc_reconnects; unsigned long flags; struct nvmf_ctrl_options *opts; @@ -563,6 +569,8 @@ struct nvme_ns_head { unsigned long flags; struct delayed_work remove_work; unsigned int delayed_removal_secs; + atomic_long_t io_requeue_no_usable_path_count; + atomic_long_t io_fail_no_available_path_count; #define NVME_NSHEAD_DISK_LIVE 0 #define NVME_NSHEAD_QUEUE_IF_NO_PATH 1 struct nvme_ns __rcu *current_path[]; @@ -589,7 +597,10 @@ struct nvme_ns { #ifdef CONFIG_NVME_MULTIPATH enum nvme_ana_state ana_state; u32 ana_grpid; + atomic_long_t failover; #endif + atomic_long_t retries; + atomic_long_t errors; struct list_head siblings; struct kref kref; struct nvme_ns_head *head; @@ -900,7 +911,7 @@ void nvme_sync_queues(struct nvme_ctrl *ctrl); void nvme_sync_io_queues(struct nvme_ctrl *ctrl); void nvme_unfreeze(struct nvme_ctrl *ctrl); void nvme_wait_freeze(struct nvme_ctrl *ctrl); -int nvme_wait_freeze_timeout(struct nvme_ctrl *ctrl, long timeout); +int nvme_wait_freeze_timeout(struct nvme_ctrl *ctrl); void nvme_start_freeze(struct nvme_ctrl *ctrl); static inline enum req_op nvme_req_op(struct nvme_command *cmd) @@ -1012,6 +1023,7 @@ extern const struct attribute_group nvme_ns_mpath_attr_group; extern const struct pr_ops nvme_pr_ops; extern const struct block_device_operations nvme_ns_head_ops; extern const struct attribute_group nvme_dev_attrs_group; +extern const struct attribute_group nvme_dev_diag_attrs_group; extern const struct attribute_group *nvme_subsys_attrs_groups[]; extern const struct attribute_group *nvme_dev_attr_groups[]; extern const struct block_device_operations nvme_bdev_ops; @@ -1041,7 +1053,7 @@ void nvme_mpath_update(struct nvme_ctrl *ctrl); void nvme_mpath_uninit(struct nvme_ctrl *ctrl); void nvme_mpath_stop(struct nvme_ctrl *ctrl); bool nvme_mpath_clear_current_path(struct nvme_ns *ns); -void nvme_mpath_revalidate_paths(struct nvme_ns *ns); +void nvme_mpath_revalidate_paths(struct nvme_ns_head *head); void nvme_mpath_clear_ctrl_paths(struct nvme_ctrl *ctrl); void nvme_mpath_remove_disk(struct nvme_ns_head *head); void nvme_mpath_start_request(struct request *rq); @@ -1061,6 +1073,9 @@ extern struct device_attribute dev_attr_ana_state; extern struct device_attribute dev_attr_queue_depth; extern struct device_attribute dev_attr_numa_nodes; extern struct device_attribute dev_attr_delayed_removal_secs; +extern struct device_attribute dev_attr_multipath_failover_count; +extern struct device_attribute dev_attr_io_requeue_no_usable_path_count; +extern struct device_attribute dev_attr_io_fail_no_available_path_count; extern struct device_attribute subsys_attr_iopolicy; static inline bool nvme_disk_is_ns_head(struct gendisk *disk) @@ -1106,7 +1121,7 @@ static inline bool nvme_mpath_clear_current_path(struct nvme_ns *ns) { return false; } -static inline void nvme_mpath_revalidate_paths(struct nvme_ns *ns) +static inline void nvme_mpath_revalidate_paths(struct nvme_ns_head *head) { } static inline void nvme_mpath_clear_ctrl_paths(struct nvme_ctrl *ctrl) diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index b5f846200678..8438c904ec49 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -587,11 +587,16 @@ static bool nvme_dbbuf_update_and_check_event(u16 value, __le32 *dbbuf_db, } static struct nvme_descriptor_pools * -nvme_setup_descriptor_pools(struct nvme_dev *dev, unsigned numa_node) +nvme_setup_descriptor_pools(struct nvme_dev *dev, int numa_node) { - struct nvme_descriptor_pools *pools = &dev->descriptor_pools[numa_node]; + struct nvme_descriptor_pools *pools; size_t small_align = NVME_SMALL_POOL_SIZE; + if (numa_node == NUMA_NO_NODE) + numa_node = 0; + + pools = &dev->descriptor_pools[numa_node]; + if (pools->small) return pools; /* already initialized */ @@ -660,7 +665,7 @@ static int nvme_init_hctx(struct blk_mq_hw_ctx *hctx, void *data, static int nvme_pci_init_request(struct blk_mq_tag_set *set, struct request *req, unsigned int hctx_idx, - unsigned int numa_node) + int numa_node) { struct nvme_iod *iod = blk_mq_rq_to_pdu(req); @@ -2838,6 +2843,7 @@ static const struct attribute_group nvme_pci_dev_attrs_group = { static const struct attribute_group *nvme_pci_dev_attr_groups[] = { &nvme_dev_attrs_group, &nvme_pci_dev_attrs_group, + &nvme_dev_diag_attrs_group, NULL, }; @@ -3122,7 +3128,7 @@ static bool __nvme_delete_io_queues(struct nvme_dev *dev, u8 opcode) unsigned long timeout; retry: - timeout = NVME_ADMIN_TIMEOUT; + timeout = dev->ctrl.admin_timeout; while (nr_queues > 0) { if (nvme_delete_queue(&dev->queues[nr_queues], opcode)) break; @@ -3304,7 +3310,7 @@ static void nvme_dev_disable(struct nvme_dev *dev, bool shutdown) * if doing a safe shutdown. */ if (!dead && shutdown) - nvme_wait_freeze_timeout(&dev->ctrl, NVME_IO_TIMEOUT); + nvme_wait_freeze_timeout(&dev->ctrl); } nvme_quiesce_io_queues(&dev->ctrl); diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c index f77c960f7632..6909e3542794 100644 --- a/drivers/nvme/host/rdma.c +++ b/drivers/nvme/host/rdma.c @@ -292,7 +292,7 @@ static void nvme_rdma_exit_request(struct blk_mq_tag_set *set, static int nvme_rdma_init_request(struct blk_mq_tag_set *set, struct request *rq, unsigned int hctx_idx, - unsigned int numa_node) + int numa_node) { struct nvme_rdma_ctrl *ctrl = to_rdma_ctrl(set->driver_data); struct nvme_rdma_request *req = blk_mq_rq_to_pdu(rq); @@ -888,7 +888,7 @@ static int nvme_rdma_configure_io_queues(struct nvme_rdma_ctrl *ctrl, bool new) if (!new) { nvme_start_freeze(&ctrl->ctrl); nvme_unquiesce_io_queues(&ctrl->ctrl); - if (!nvme_wait_freeze_timeout(&ctrl->ctrl, NVME_IO_TIMEOUT)) { + if (!nvme_wait_freeze_timeout(&ctrl->ctrl)) { /* * If we timed out waiting for freeze we are likely to * be stuck. Fail the controller initialization just @@ -1110,6 +1110,8 @@ static void nvme_rdma_reconnect_ctrl_work(struct work_struct *work) dev_info(ctrl->ctrl.device, "Successfully reconnected (%d attempts)\n", ctrl->ctrl.nr_reconnects); + /* accumulate reconnect attempts before resetting it to zero */ + atomic_long_add(ctrl->ctrl.nr_reconnects, &ctrl->ctrl.acc_reconnects); ctrl->ctrl.nr_reconnects = 0; return; diff --git a/drivers/nvme/host/sysfs.c b/drivers/nvme/host/sysfs.c index e59758616f27..933a5adfb7af 100644 --- a/drivers/nvme/host/sysfs.c +++ b/drivers/nvme/host/sysfs.c @@ -6,6 +6,7 @@ */ #include <linux/nvme-auth.h> +#include <linux/blkdev.h> #include "nvme.h" #include "fabrics.h" @@ -335,14 +336,7 @@ static bool multipath_sysfs_group_visible(struct kobject *kobj) return nvme_disk_is_ns_head(dev_to_disk(dev)); } - -static bool multipath_sysfs_attr_visible(struct kobject *kobj, - struct attribute *attr, int n) -{ - return false; -} - -DEFINE_SYSFS_GROUP_VISIBLE(multipath_sysfs) +DEFINE_SIMPLE_SYSFS_GROUP_VISIBLE(multipath_sysfs) const struct attribute_group nvme_ns_mpath_attr_group = { .name = "multipath", @@ -351,11 +345,114 @@ const struct attribute_group nvme_ns_mpath_attr_group = { }; #endif +static ssize_t command_retries_count_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nvme_ns *ns = nvme_get_ns_from_dev(dev); + + return sysfs_emit(buf, "%lu\n", atomic_long_read(&ns->retries)); +} + +static ssize_t command_retries_count_store(struct device *dev, + struct device_attribute *attr, const char *buf, size_t count) +{ + unsigned long retries; + int err; + struct nvme_ns *ns = nvme_get_ns_from_dev(dev); + + err = kstrtoul(buf, 0, &retries); + if (err) + return -EINVAL; + + atomic_long_set(&ns->retries, retries); + + return count; +} +static DEVICE_ATTR_RW(command_retries_count); + +static ssize_t nvme_io_errors_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nvme_ns *ns = nvme_get_ns_from_dev(dev); + + return sysfs_emit(buf, "%lu\n", atomic_long_read(&ns->errors)); +} + +static ssize_t nvme_io_errors_store(struct device *dev, + struct device_attribute *attr, const char *buf, size_t count) +{ + unsigned long errors; + int err; + struct nvme_ns *ns = nvme_get_ns_from_dev(dev); + + err = kstrtoul(buf, 0, &errors); + if (err) + return -EINVAL; + + atomic_long_set(&ns->errors, errors); + + return count; +} + +struct device_attribute dev_attr_io_errors = + __ATTR(command_error_count, 0644, + nvme_io_errors_show, nvme_io_errors_store); + +static struct attribute *nvme_ns_diag_attrs[] = { + &dev_attr_command_retries_count.attr, + &dev_attr_io_errors.attr, +#ifdef CONFIG_NVME_MULTIPATH + &dev_attr_multipath_failover_count.attr, + &dev_attr_io_requeue_no_usable_path_count.attr, + &dev_attr_io_fail_no_available_path_count.attr, +#endif + NULL, +}; + +static umode_t nvme_ns_diag_attrs_are_visible(struct kobject *kobj, + struct attribute *a, int n) +{ + struct device *dev = container_of(kobj, struct device, kobj); + + if (a == &dev_attr_command_retries_count.attr) { + if (nvme_disk_is_ns_head(dev_to_disk(dev))) + return 0; + } + if (a == &dev_attr_io_errors.attr) { + struct gendisk *disk = dev_to_disk(dev); + + if (nvme_disk_is_ns_head(disk)) + return 0; + } +#ifdef CONFIG_NVME_MULTIPATH + if (a == &dev_attr_multipath_failover_count.attr) { + if (nvme_disk_is_ns_head(dev_to_disk(dev))) + return 0; + } + if (a == &dev_attr_io_requeue_no_usable_path_count.attr) { + if (!nvme_disk_is_ns_head(dev_to_disk(dev))) + return 0; + } + if (a == &dev_attr_io_fail_no_available_path_count.attr) { + if (!nvme_disk_is_ns_head(dev_to_disk(dev))) + return 0; + } +#endif + return a->mode; +} + +const struct attribute_group nvme_ns_diag_attr_group = { + .name = "diag", + .attrs = nvme_ns_diag_attrs, + .is_visible = nvme_ns_diag_attrs_are_visible, +}; + const struct attribute_group *nvme_ns_attr_groups[] = { &nvme_ns_attr_group, #ifdef CONFIG_NVME_MULTIPATH &nvme_ns_mpath_attr_group, #endif + &nvme_ns_diag_attr_group, NULL, }; @@ -623,6 +720,92 @@ static ssize_t quirks_show(struct device *dev, struct device_attribute *attr, } static DEVICE_ATTR_RO(quirks); +static ssize_t nvme_admin_timeout_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nvme_ctrl *ctrl = dev_get_drvdata(dev); + + return sysfs_emit(buf, "%u\n", + jiffies_to_msecs(ctrl->admin_timeout)); +} + +static ssize_t nvme_admin_timeout_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + struct nvme_ctrl *ctrl = dev_get_drvdata(dev); + u32 timeout; + int err; + + /* + * Wait until the controller reaches the LIVE state to be sure that + * admin_q and fabrics_q are properly initialized. + */ + if (!test_bit(NVME_CTRL_STARTED_ONCE, &ctrl->flags)) + return -EBUSY; + + err = kstrtou32(buf, 10, &timeout); + if (err || !timeout) + return -EINVAL; + + ctrl->admin_timeout = msecs_to_jiffies(timeout); + + blk_queue_rq_timeout(ctrl->admin_q, ctrl->admin_timeout); + if (ctrl->fabrics_q) + blk_queue_rq_timeout(ctrl->fabrics_q, ctrl->admin_timeout); + + return count; +} + +static DEVICE_ATTR(admin_timeout, S_IRUGO | S_IWUSR, + nvme_admin_timeout_show, nvme_admin_timeout_store); + +static ssize_t nvme_io_timeout_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nvme_ctrl *ctrl = dev_get_drvdata(dev); + + return sysfs_emit(buf, "%u\n", jiffies_to_msecs(ctrl->io_timeout)); +} + +static ssize_t nvme_io_timeout_store(struct device *dev, + struct device_attribute *attr, + const char *buf, size_t count) +{ + struct nvme_ctrl *ctrl = dev_get_drvdata(dev); + struct nvme_ns *ns; + u32 timeout; + int err; + + /* + * Wait until the controller reaches the LIVE state to be sure that + * connect_q is properly initialized. + */ + if (!test_bit(NVME_CTRL_STARTED_ONCE, &ctrl->flags)) + return -EBUSY; + + err = kstrtou32(buf, 10, &timeout); + if (err || !timeout) + return -EINVAL; + + /* Take the namespaces_lock to avoid racing against nvme_alloc_ns() */ + mutex_lock(&ctrl->namespaces_lock); + + ctrl->io_timeout = msecs_to_jiffies(timeout); + list_for_each_entry(ns, &ctrl->namespaces, list) + blk_queue_rq_timeout(ns->queue, ctrl->io_timeout); + + mutex_unlock(&ctrl->namespaces_lock); + + if (ctrl->connect_q) + blk_queue_rq_timeout(ctrl->connect_q, ctrl->io_timeout); + + return count; +} + +static DEVICE_ATTR(io_timeout, S_IRUGO | S_IWUSR, + nvme_io_timeout_show, nvme_io_timeout_store); + #ifdef CONFIG_NVME_HOST_AUTH static ssize_t nvme_ctrl_dhchap_secret_show(struct device *dev, struct device_attribute *attr, char *buf) @@ -765,6 +948,8 @@ static struct attribute *nvme_dev_attrs[] = { &dev_attr_cntrltype.attr, &dev_attr_dctype.attr, &dev_attr_quirks.attr, + &dev_attr_admin_timeout.attr, + &dev_attr_io_timeout.attr, #ifdef CONFIG_NVME_HOST_AUTH &dev_attr_dhchap_secret.attr, &dev_attr_dhchap_ctrl_secret.attr, @@ -937,11 +1122,121 @@ static const struct attribute_group nvme_tls_attrs_group = { }; #endif +static ssize_t nvme_adm_errors_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nvme_ctrl *ctrl = dev_get_drvdata(dev); + + return sysfs_emit(buf, "%lu\n", + (unsigned long)atomic_long_read(&ctrl->errors)); +} + +static ssize_t nvme_adm_errors_store(struct device *dev, + struct device_attribute *attr, const char *buf, size_t count) +{ + unsigned long errors; + int err; + struct nvme_ctrl *ctrl = dev_get_drvdata(dev); + + err = kstrtoul(buf, 0, &errors); + if (err) + return -EINVAL; + + atomic_long_set(&ctrl->errors, errors); + + return count; +} + +struct device_attribute dev_attr_adm_errors = + __ATTR(command_error_count, 0644, + nvme_adm_errors_show, nvme_adm_errors_store); + +static ssize_t reset_count_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nvme_ctrl *ctrl = dev_get_drvdata(dev); + + return sysfs_emit(buf, "%lu\n", atomic_long_read(&ctrl->nr_reset)); +} + +static ssize_t reset_count_store(struct device *dev, + struct device_attribute *attr, const char *buf, size_t count) +{ + int err; + unsigned long reset_cnt; + struct nvme_ctrl *ctrl = dev_get_drvdata(dev); + + err = kstrtoul(buf, 0, &reset_cnt); + if (err) + return -EINVAL; + + atomic_long_set(&ctrl->nr_reset, reset_cnt); + + return count; +} + +static ssize_t reconnect_count_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct nvme_ctrl *ctrl = dev_get_drvdata(dev); + + return sysfs_emit(buf, "%lu\n", + atomic_long_read(&ctrl->acc_reconnects) + + ctrl->nr_reconnects); +} + +static ssize_t reconnect_count_store(struct device *dev, + struct device_attribute *attr, const char *buf, size_t count) +{ + int err; + unsigned long reconnect_cnt; + struct nvme_ctrl *ctrl = dev_get_drvdata(dev); + + err = kstrtoul(buf, 0, &reconnect_cnt); + if (err) + return -EINVAL; + + atomic_long_set(&ctrl->acc_reconnects, reconnect_cnt); + + return count; +} + +static DEVICE_ATTR_RW(reconnect_count); + +static DEVICE_ATTR_RW(reset_count); + +static struct attribute *nvme_dev_diag_attrs[] = { + &dev_attr_adm_errors.attr, + &dev_attr_reset_count.attr, + &dev_attr_reconnect_count.attr, + NULL, +}; + +static umode_t nvme_dev_diag_attrs_are_visible(struct kobject *kobj, + struct attribute *a, int n) +{ + struct device *dev = container_of(kobj, struct device, kobj); + struct nvme_ctrl *ctrl = dev_get_drvdata(dev); + + if (a == &dev_attr_reconnect_count.attr && !ctrl->opts) + return 0; + + return a->mode; +} + +const struct attribute_group nvme_dev_diag_attrs_group = { + .name = "diag", + .attrs = nvme_dev_diag_attrs, + .is_visible = nvme_dev_diag_attrs_are_visible, +}; +EXPORT_SYMBOL_GPL(nvme_dev_diag_attrs_group); + const struct attribute_group *nvme_dev_attr_groups[] = { &nvme_dev_attrs_group, #ifdef CONFIG_NVME_TCP_TLS &nvme_tls_attrs_group, #endif + &nvme_dev_diag_attrs_group, NULL, }; diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 68a1d7640494..ba5c7b3e2a7c 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -56,44 +56,6 @@ MODULE_PARM_DESC(tls_handshake_timeout, static atomic_t nvme_tcp_cpu_queues[NR_CPUS]; -#ifdef CONFIG_DEBUG_LOCK_ALLOC -/* lockdep can detect a circular dependency of the form - * sk_lock -> mmap_lock (page fault) -> fs locks -> sk_lock - * because dependencies are tracked for both nvme-tcp and user contexts. Using - * a separate class prevents lockdep from conflating nvme-tcp socket use with - * user-space socket API use. - */ -static struct lock_class_key nvme_tcp_sk_key[2]; -static struct lock_class_key nvme_tcp_slock_key[2]; - -static void nvme_tcp_reclassify_socket(struct socket *sock) -{ - struct sock *sk = sock->sk; - - if (WARN_ON_ONCE(!sock_allow_reclassification(sk))) - return; - - switch (sk->sk_family) { - case AF_INET: - sock_lock_init_class_and_name(sk, "slock-AF_INET-NVME", - &nvme_tcp_slock_key[0], - "sk_lock-AF_INET-NVME", - &nvme_tcp_sk_key[0]); - break; - case AF_INET6: - sock_lock_init_class_and_name(sk, "slock-AF_INET6-NVME", - &nvme_tcp_slock_key[1], - "sk_lock-AF_INET6-NVME", - &nvme_tcp_sk_key[1]); - break; - default: - WARN_ON_ONCE(1); - } -} -#else -static void nvme_tcp_reclassify_socket(struct socket *sock) { } -#endif - enum nvme_tcp_send_state { NVME_TCP_SEND_CMD_PDU = 0, NVME_TCP_SEND_H2C_PDU, @@ -180,6 +142,11 @@ struct nvme_tcp_queue { void (*state_change)(struct sock *); void (*data_ready)(struct sock *); void (*write_space)(struct sock *); + +#ifdef CONFIG_DEBUG_LOCK_ALLOC + struct lock_class_key nvme_tcp_sk_key; + struct lock_class_key nvme_tcp_slock_key; +#endif }; struct nvme_tcp_ctrl { @@ -207,6 +174,39 @@ static const struct blk_mq_ops nvme_tcp_mq_ops; static const struct blk_mq_ops nvme_tcp_admin_mq_ops; static int nvme_tcp_try_send(struct nvme_tcp_queue *queue); +#ifdef CONFIG_DEBUG_LOCK_ALLOC +/* lockdep can detect a circular dependency of the form + * sk_lock -> mmap_lock (page fault) -> fs locks -> sk_lock + * because dependencies are tracked for both nvme-tcp and user contexts. Using + * a separate class prevents lockdep from conflating nvme-tcp socket use with + * user-space socket API use. + */ +static void nvme_tcp_reclassify_socket(struct nvme_tcp_queue *queue) +{ + struct sock *sk = queue->sock->sk; + + if (WARN_ON_ONCE(!sock_allow_reclassification(sk))) + return; + + switch (sk->sk_family) { + case AF_INET: + sock_lock_init_class_and_name(sk, "slock-AF_INET-NVME", + &queue->nvme_tcp_slock_key, + "sk_lock-AF_INET-NVME", + &queue->nvme_tcp_sk_key); + break; + case AF_INET6: + sock_lock_init_class_and_name(sk, "slock-AF_INET6-NVME", + &queue->nvme_tcp_slock_key, + "sk_lock-AF_INET6-NVME", + &queue->nvme_tcp_sk_key); + break; + default: + WARN_ON_ONCE(1); + } +} +#endif + static inline struct nvme_tcp_ctrl *to_tcp_ctrl(struct nvme_ctrl *ctrl) { return container_of(ctrl, struct nvme_tcp_ctrl, ctrl); @@ -340,32 +340,25 @@ static void nvme_tcp_init_iter(struct nvme_tcp_request *req, unsigned int dir) { struct request *rq = blk_mq_rq_from_pdu(req); - struct bio_vec *vec; - unsigned int size; - int nr_bvec; - size_t offset; if (rq->rq_flags & RQF_SPECIAL_PAYLOAD) { - vec = &rq->special_vec; - nr_bvec = 1; - size = blk_rq_payload_bytes(rq); - offset = 0; + iov_iter_bvec(&req->iter, dir, &rq->special_vec, 1, + blk_rq_payload_bytes(rq)); + req->iter.iov_offset = 0; } else { struct bio *bio = req->curr_bio; struct bvec_iter bi; struct bio_vec bv; + int nr_bvec = 0; - vec = __bvec_iter_bvec(bio->bi_io_vec, bio->bi_iter); - nr_bvec = 0; - bio_for_each_bvec(bv, bio, bi) { + bio_for_each_bvec(bv, bio, bi) nr_bvec++; - } - size = bio->bi_iter.bi_size; - offset = bio->bi_iter.bi_bvec_done; - } - iov_iter_bvec(&req->iter, dir, vec, nr_bvec, size); - req->iter.iov_offset = offset; + iov_iter_bvec(&req->iter, dir, + __bvec_iter_bvec(bio->bi_io_vec, bio->bi_iter), nr_bvec, + bio->bi_iter.bi_size); + req->iter.iov_offset = bio->bi_iter.bi_bvec_done; + } } static inline void nvme_tcp_advance_req(struct nvme_tcp_request *req, @@ -548,7 +541,7 @@ static void nvme_tcp_exit_request(struct blk_mq_tag_set *set, static int nvme_tcp_init_request(struct blk_mq_tag_set *set, struct request *rq, unsigned int hctx_idx, - unsigned int numa_node) + int numa_node) { struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(set->driver_data); struct nvme_tcp_request *req = blk_mq_rq_to_pdu(rq); @@ -1468,6 +1461,11 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) kfree(queue->pdu); mutex_destroy(&queue->send_mutex); mutex_destroy(&queue->queue_lock); + +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_unregister_key(&queue->nvme_tcp_sk_key); + lockdep_unregister_key(&queue->nvme_tcp_slock_key); +#endif } static int nvme_tcp_init_connection(struct nvme_tcp_queue *queue) @@ -1813,7 +1811,12 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid, } sk_net_refcnt_upgrade(queue->sock->sk); - nvme_tcp_reclassify_socket(queue->sock); + +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_register_key(&queue->nvme_tcp_sk_key); + lockdep_register_key(&queue->nvme_tcp_slock_key); + nvme_tcp_reclassify_socket(queue); +#endif /* Single syn retry */ tcp_sock_set_syncnt(queue->sock->sk, 1); @@ -1918,6 +1921,10 @@ err_sock: /* Use sync variant - see nvme_tcp_free_queue() for explanation */ __fput_sync(queue->sock->file); queue->sock = NULL; +#ifdef CONFIG_DEBUG_LOCK_ALLOC + lockdep_unregister_key(&queue->nvme_tcp_sk_key); + lockdep_unregister_key(&queue->nvme_tcp_slock_key); +#endif err_destroy_mutex: mutex_destroy(&queue->send_mutex); mutex_destroy(&queue->queue_lock); @@ -2208,7 +2215,7 @@ static int nvme_tcp_configure_io_queues(struct nvme_ctrl *ctrl, bool new) if (!new) { nvme_start_freeze(ctrl); nvme_unquiesce_io_queues(ctrl); - if (!nvme_wait_freeze_timeout(ctrl, NVME_IO_TIMEOUT)) { + if (!nvme_wait_freeze_timeout(ctrl)) { /* * If we timed out waiting for freeze we are likely to * be stuck. Fail the controller initialization just @@ -2475,6 +2482,8 @@ static void nvme_tcp_reconnect_ctrl_work(struct work_struct *work) dev_info(ctrl->device, "Successfully reconnected (attempt %d/%d)\n", ctrl->nr_reconnects, ctrl->opts->max_reconnects); + /* accumulate reconnect attempts before resetting it to zero */ + atomic_long_add(ctrl->nr_reconnects, &ctrl->acc_reconnects); ctrl->nr_reconnects = 0; return; @@ -3053,6 +3062,8 @@ static int __init nvme_tcp_init_module(void) if (wq_unbound) wq_flags |= WQ_UNBOUND; + else + wq_flags |= WQ_PERCPU; nvme_tcp_wq = alloc_workqueue("nvme_tcp_wq", wq_flags, 0); if (!nvme_tcp_wq) diff --git a/drivers/nvme/target/discovery.c b/drivers/nvme/target/discovery.c index e9b35549e254..114869d16a1f 100644 --- a/drivers/nvme/target/discovery.c +++ b/drivers/nvme/target/discovery.c @@ -166,6 +166,7 @@ static void nvmet_execute_disc_get_log_page(struct nvmet_req *req) u64 offset = nvmet_get_log_page_offset(req->cmd); size_t data_len = nvmet_get_log_page_len(req->cmd); size_t alloc_len; + size_t copy_len; struct nvmet_subsys_link *p; struct nvmet_port *r; u32 numrec = 0; @@ -242,7 +243,27 @@ static void nvmet_execute_disc_get_log_page(struct nvmet_req *req) up_read(&nvmet_config_sem); - status = nvmet_copy_to_sgl(req, 0, buffer + offset, data_len); + /* + * Validate the host-supplied log page offset before copying out. + * Without this check, the host controls a 64-bit byte offset into + * a small kzalloc'd buffer: a value past the log page lets the + * subsequent memcpy read adjacent kernel heap, and a value aimed + * at unmapped kernel memory faults the in-kernel copy and crashes + * the target host. The Discovery controller is unauthenticated, + * so the bug is reachable from any reachable fabric peer. + */ + if (offset > alloc_len) { + req->error_loc = + offsetof(struct nvme_get_log_page_command, lpo); + status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR; + goto out_free_buffer; + } + + copy_len = min_t(size_t, data_len, alloc_len - offset); + status = nvmet_copy_to_sgl(req, 0, buffer + offset, copy_len); + if (!status && copy_len < data_len) + status = nvmet_zero_sgl(req, copy_len, data_len - copy_len); +out_free_buffer: kfree(buffer); out: nvmet_req_complete(req, status); diff --git a/drivers/nvme/target/fabrics-cmd-auth.c b/drivers/nvme/target/fabrics-cmd-auth.c index f1e613e7c63e..0a85acf1e5c7 100644 --- a/drivers/nvme/target/fabrics-cmd-auth.c +++ b/drivers/nvme/target/fabrics-cmd-auth.c @@ -132,13 +132,22 @@ static u8 nvmet_auth_negotiate(struct nvmet_req *req, void *d) return 0; } -static u8 nvmet_auth_reply(struct nvmet_req *req, void *d) +static u8 nvmet_auth_reply(struct nvmet_req *req, void *d, u32 tl) { struct nvmet_ctrl *ctrl = req->sq->ctrl; struct nvmf_auth_dhchap_reply_data *data = d; - u16 dhvlen = le16_to_cpu(data->dhvlen); + u16 dhvlen; u8 *response; + if (tl < sizeof(*data)) + return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD; + + dhvlen = le16_to_cpu(data->dhvlen); + + /* Validate that hl and dhvlen fit within the transfer length */ + if (sizeof(*data) + 2 * (size_t)data->hl + dhvlen > tl) + return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD; + pr_debug("%s: ctrl %d qid %d: data hl %d cvalid %d dhvlen %u\n", __func__, ctrl->cntlid, req->sq->qid, data->hl, data->cvalid, dhvlen); @@ -338,7 +347,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req) switch (data->auth_id) { case NVME_AUTH_DHCHAP_MESSAGE_REPLY: - dhchap_status = nvmet_auth_reply(req, d); + dhchap_status = nvmet_auth_reply(req, d, tl); if (dhchap_status == 0) req->sq->dhchap_step = NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1; diff --git a/drivers/nvme/target/loop.c b/drivers/nvme/target/loop.c index d98d0cdc5d6f..fcb1f8186fdd 100644 --- a/drivers/nvme/target/loop.c +++ b/drivers/nvme/target/loop.c @@ -202,7 +202,7 @@ static int nvme_loop_init_iod(struct nvme_loop_ctrl *ctrl, static int nvme_loop_init_request(struct blk_mq_tag_set *set, struct request *req, unsigned int hctx_idx, - unsigned int numa_node) + int numa_node) { struct nvme_loop_ctrl *ctrl = to_loop_ctrl(set->driver_data); struct nvme_loop_iod *iod = blk_mq_rq_to_pdu(req); @@ -274,7 +274,6 @@ static void nvme_loop_destroy_admin_queue(struct nvme_loop_ctrl *ctrl) nvmet_sq_destroy(&ctrl->queues[0].nvme_sq); nvmet_cq_put(&ctrl->queues[0].nvme_cq); - nvme_remove_admin_tag_set(&ctrl->ctrl); } static void nvme_loop_free_ctrl(struct nvme_ctrl *nctrl) @@ -375,25 +374,18 @@ static int nvme_loop_configure_admin_queue(struct nvme_loop_ctrl *ctrl) } ctrl->ctrl.queue_count = 1; - error = nvme_alloc_admin_tag_set(&ctrl->ctrl, &ctrl->admin_tag_set, - &nvme_loop_admin_mq_ops, - sizeof(struct nvme_loop_iod) + - NVME_INLINE_SG_CNT * sizeof(struct scatterlist)); - if (error) - goto out_free_sq; - /* reset stopped state for the fresh admin queue */ clear_bit(NVME_CTRL_ADMIN_Q_STOPPED, &ctrl->ctrl.flags); error = nvmf_connect_admin_queue(&ctrl->ctrl); if (error) - goto out_cleanup_tagset; + goto out_free_sq; set_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags); error = nvme_enable_ctrl(&ctrl->ctrl); if (error) - goto out_cleanup_tagset; + goto out_free_sq; ctrl->ctrl.max_hw_sectors = (NVME_LOOP_MAX_SEGMENTS - 1) << PAGE_SECTORS_SHIFT; @@ -402,14 +394,12 @@ static int nvme_loop_configure_admin_queue(struct nvme_loop_ctrl *ctrl) error = nvme_init_ctrl_finish(&ctrl->ctrl, false); if (error) - goto out_cleanup_tagset; + goto out_free_sq; return 0; -out_cleanup_tagset: - clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags); - nvme_remove_admin_tag_set(&ctrl->ctrl); out_free_sq: + clear_bit(NVME_LOOP_Q_LIVE, &ctrl->queues[0].flags); nvmet_sq_destroy(&ctrl->queues[0].nvme_sq); nvmet_cq_put(&ctrl->queues[0].nvme_cq); return error; @@ -432,6 +422,7 @@ static void nvme_loop_shutdown_ctrl(struct nvme_loop_ctrl *ctrl) static void nvme_loop_delete_ctrl_host(struct nvme_ctrl *ctrl) { nvme_loop_shutdown_ctrl(to_loop_ctrl(ctrl)); + nvme_remove_admin_tag_set(ctrl); } static void nvme_loop_delete_ctrl(struct nvmet_ctrl *nctrl) @@ -494,6 +485,7 @@ out_destroy_admin: nvme_cancel_admin_tagset(&ctrl->ctrl); nvme_loop_destroy_admin_queue(ctrl); out_disable: + nvme_remove_admin_tag_set(&ctrl->ctrl); dev_warn(ctrl->ctrl.device, "Removing after reset failure\n"); nvme_uninit_ctrl(&ctrl->ctrl); } @@ -594,10 +586,17 @@ static struct nvme_ctrl *nvme_loop_create_ctrl(struct device *dev, if (!ctrl->queues) goto out_uninit_ctrl; - ret = nvme_loop_configure_admin_queue(ctrl); + ret = nvme_alloc_admin_tag_set(&ctrl->ctrl, &ctrl->admin_tag_set, + &nvme_loop_admin_mq_ops, + sizeof(struct nvme_loop_iod) + + NVME_INLINE_SG_CNT * sizeof(struct scatterlist)); if (ret) goto out_free_queues; + ret = nvme_loop_configure_admin_queue(ctrl); + if (ret) + goto out_remove_admin_tagset; + if (opts->queue_size > ctrl->ctrl.maxcmd) { /* warn if maxcmd is lower than queue_size */ dev_warn(ctrl->ctrl.device, @@ -633,6 +632,8 @@ out_remove_admin_queue: nvme_quiesce_admin_queue(&ctrl->ctrl); nvme_cancel_admin_tagset(&ctrl->ctrl); nvme_loop_destroy_admin_queue(ctrl); +out_remove_admin_tagset: + nvme_remove_admin_tag_set(&ctrl->ctrl); out_free_queues: kfree(ctrl->queues); out_uninit_ctrl: diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c index e6e2c3f9afdf..ac26f4f774c4 100644 --- a/drivers/nvme/target/rdma.c +++ b/drivers/nvme/target/rdma.c @@ -1598,8 +1598,10 @@ static int nvmet_rdma_queue_connect(struct rdma_cm_id *cm_id, pending++; } mutex_unlock(&nvmet_rdma_queue_mutex); - if (pending > NVMET_RDMA_BACKLOG) - return NVME_SC_CONNECT_CTRL_BUSY; + if (pending > NVMET_RDMA_BACKLOG) { + ret = NVME_SC_CONNECT_CTRL_BUSY; + goto put_device; + } } ret = nvmet_rdma_cm_accept(cm_id, queue, &event->param.conn); diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c index 20f150d17a96..15c52f1f95f1 100644 --- a/drivers/nvme/target/tcp.c +++ b/drivers/nvme/target/tcp.c @@ -1844,10 +1844,11 @@ static void nvmet_tcp_tls_handshake_done(void *data, int status, if (!status) status = nvmet_tcp_tls_key_lookup(queue, peerid); + if (!status) + status = nvmet_tcp_set_queue_sock(queue); + if (status) nvmet_tcp_schedule_release_queue(queue); - else - nvmet_tcp_set_queue_sock(queue); kref_put(&queue->kref, nvmet_tcp_release_queue); } @@ -1999,6 +2000,12 @@ out_free_connect: nvmet_tcp_free_cmd(&queue->connect); out_ida_remove: ida_free(&nvmet_tcp_queue_ida, queue->idx); + /* + * Drain the page fragment cache if any allocations were done. + * The first allocation using pf_cache is nvmet_tcp_alloc_cmd() + * for queue->connect after ida_alloc(). + */ + page_frag_cache_drain(&queue->pf_cache); out_sock: fput(queue->sock->file); out_free_queue: diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c index 85eef401925a..22e2e3223440 100644 --- a/drivers/scsi/scsi_lib.c +++ b/drivers/scsi/scsi_lib.c @@ -1973,7 +1973,7 @@ out_put_budget: } static int scsi_mq_init_request(struct blk_mq_tag_set *set, struct request *rq, - unsigned int hctx_idx, unsigned int numa_node) + unsigned int hctx_idx, int numa_node) { struct Scsi_Host *shost = set->driver_data; struct scsi_cmnd *cmd = blk_mq_rq_to_pdu(rq); diff --git a/fs/direct-io.c b/fs/direct-io.c index 2267f5ae7f77..9e4976716985 100644 --- a/fs/direct-io.c +++ b/fs/direct-io.c @@ -502,12 +502,8 @@ static blk_status_t dio_bio_complete(struct dio *dio, struct bio *bio) const enum req_op dio_op = dio->opf & REQ_OP_MASK; bool should_dirty = dio_op == REQ_OP_READ && dio->should_dirty; - if (err) { - if (err == BLK_STS_AGAIN && (bio->bi_opf & REQ_NOWAIT)) - dio->io_error = -EAGAIN; - else - dio->io_error = -EIO; - } + if (err) + dio->io_error = -EIO; if (dio->is_async && should_dirty) { bio_check_pages_dirty(bio); /* transfers ownership */ @@ -1178,13 +1174,10 @@ ssize_t __blockdev_direct_IO(struct kiocb *iocb, struct inode *inode, dio->is_async = true; dio->inode = inode; - if (iov_iter_rw(iter) == WRITE) { + if (iov_iter_rw(iter) == WRITE) dio->opf = REQ_OP_WRITE | REQ_SYNC | REQ_IDLE; - if (iocb->ki_flags & IOCB_NOWAIT) - dio->opf |= REQ_NOWAIT; - } else { + else dio->opf = REQ_OP_READ; - } /* * For AIO O_(D)SYNC writes we need to defer completions to a workqueue diff --git a/include/linux/bio.h b/include/linux/bio.h index 8300d5565e36..8f33f717b14f 100644 --- a/include/linux/bio.h +++ b/include/linux/bio.h @@ -283,7 +283,7 @@ static inline void bio_first_folio(struct folio_iter *fi, struct bio *bio, return; } - fi->folio = page_folio(bvec->bv_page); + fi->folio = bvec_folio(bvec); fi->offset = bvec->bv_offset + PAGE_SIZE * folio_page_idx(fi->folio, bvec->bv_page); fi->_seg_count = bvec->bv_len; @@ -347,7 +347,6 @@ enum { }; extern int bioset_init(struct bio_set *, unsigned int, unsigned int, int flags); extern void bioset_exit(struct bio_set *); -extern int biovec_init_pool(mempool_t *pool, int pool_entries); struct bio *bio_alloc_bioset(struct block_device *bdev, unsigned short nr_vecs, blk_opf_t opf, gfp_t gfp, struct bio_set *bs); @@ -371,17 +370,27 @@ void submit_bio(struct bio *bio); extern void bio_endio(struct bio *); -static inline void bio_io_error(struct bio *bio) +/** + * bio_endio_status - end I/O on a bio with a specific status + * @bio: bio + * @status: status to set + * + * Set @bio->bi_status to @status and call bio_endio(). + **/ +static inline void bio_endio_status(struct bio *bio, blk_status_t status) { - bio->bi_status = BLK_STS_IOERR; + bio->bi_status = status; bio_endio(bio); } +static inline void bio_io_error(struct bio *bio) +{ + bio_endio_status(bio, BLK_STS_IOERR); +} + static inline void bio_wouldblock_error(struct bio *bio) { - bio_set_flag(bio, BIO_QUIET); - bio->bi_status = BLK_STS_AGAIN; - bio_endio(bio); + bio_endio_status(bio, BLK_STS_AGAIN); } /* @@ -479,17 +488,10 @@ int bio_iov_iter_bounce(struct bio *bio, struct iov_iter *iter, size_t maxlen, size_t minsize); void bio_iov_iter_unbounce(struct bio *bio, bool is_error, bool mark_dirty); -extern void bio_copy_data_iter(struct bio *dst, struct bvec_iter *dst_iter, - struct bio *src, struct bvec_iter *src_iter); extern void bio_copy_data(struct bio *dst, struct bio *src); extern void bio_free_pages(struct bio *bio); +void zero_fill_bio(struct bio *bio); void guard_bio_eod(struct bio *bio); -void zero_fill_bio_iter(struct bio *bio, struct bvec_iter iter); - -static inline void zero_fill_bio(struct bio *bio) -{ - zero_fill_bio_iter(bio, bio->bi_iter); -} static inline void bio_release_pages(struct bio *bio, bool mark_dirty) { diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h index 18a2388ba581..af878597afb8 100644 --- a/include/linux/blk-mq.h +++ b/include/linux/blk-mq.h @@ -428,7 +428,7 @@ struct blk_mq_hw_ctx { struct blk_mq_tags *sched_tags; /** @numa_node: NUMA node the storage adapter has been connected to. */ - unsigned int numa_node; + int numa_node; /** @queue_num: Index of this hardware queue. */ unsigned int queue_num; @@ -653,7 +653,7 @@ struct blk_mq_ops { * flush request. */ int (*init_request)(struct blk_mq_tag_set *set, struct request *, - unsigned int, unsigned int); + unsigned int, int); /** * @exit_request: Ditto for exit/teardown. */ @@ -1104,6 +1104,7 @@ struct req_iterator { /* * blk_rq_pos() : the current sector * blk_rq_bytes() : bytes left in the entire request + * blk_rq_has_data() : whether the request carries data * blk_rq_cur_bytes() : bytes left in the current segment * blk_rq_sectors() : sectors left in the entire request * blk_rq_cur_sectors() : sectors left in the current segment @@ -1119,6 +1120,14 @@ static inline unsigned int blk_rq_bytes(const struct request *rq) return rq->__data_len; } +static inline bool blk_rq_has_data(const struct request *rq) +{ + return blk_rq_bytes(rq) && + req_op(rq) != REQ_OP_DISCARD && + req_op(rq) != REQ_OP_SECURE_ERASE && + req_op(rq) != REQ_OP_WRITE_ZEROES; +} + static inline int blk_rq_cur_bytes(const struct request *rq) { if (!rq->bio) @@ -1243,4 +1252,44 @@ static inline int blk_rq_map_sg(struct request *rq, struct scatterlist *sglist) } void blk_dump_rq_flags(struct request *, char *); +/** + * blk_rq_passthrough_stats - check if this request should account stats + * @rq: request to check + * @q: the queue accumulating the stats + * + * Note, @q does not necessarily need to be the request_queue that provides + * @rq. + * + * Return: true if stats should be accounted. + */ +static inline bool blk_rq_passthrough_stats(struct request *rq, + struct request_queue *q) +{ + struct bio *bio = rq->bio; + + if (!blk_queue_passthrough_stat(q)) + return false; + + /* Requests without a bio do not transfer data. */ + if (!bio) + return false; + + /* + * Stats are accumulated in the bdev, so must have one attached to a + * bio to track stats. Most drivers do not set the bdev for passthrough + * requests, but nvme is one that will set it. + */ + if (!bio->bi_bdev) + return false; + + /* + * We don't know what a passthrough command does, but we know the + * payload size and data direction. Ensuring the size is aligned to the + * block size filters out most commands with payloads that don't + * represent sector access. + */ + if (blk_rq_bytes(rq) & (bdev_logical_block_size(bio->bi_bdev) - 1)) + return false; + return true; +} #endif /* BLK_MQ_H */ diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h index 890128cdea1c..5070851cf924 100644 --- a/include/linux/blkdev.h +++ b/include/linux/blkdev.h @@ -176,6 +176,7 @@ struct gendisk { #define GD_SUPPRESS_PART_SCAN 5 #define GD_OWNS_QUEUE 6 #define GD_ZONE_APPEND_USED 7 +#define GD_ERROR_INJECT 8 struct mutex open_mutex; /* open/close mutex */ unsigned open_partitions; /* number of open partitions */ @@ -227,6 +228,11 @@ struct gendisk { */ struct blk_independent_access_ranges *ia_ranges; +#ifdef CONFIG_BLK_ERROR_INJECTION + struct mutex error_injection_lock; + struct list_head error_injection_list; +#endif + struct mutex rqos_state_mutex; /* rqos state change mutex */ }; @@ -1040,7 +1046,6 @@ extern const char *blk_op_str(enum req_op op); int blk_status_to_errno(blk_status_t status); blk_status_t errno_to_blk_status(int errno); -const char *blk_status_to_str(blk_status_t status); /* only poll the hardware once, don't continue until a completion was found */ #define BLK_POLL_ONESHOT (1 << 0) @@ -1093,15 +1098,17 @@ static inline unsigned int blk_boundary_sectors_left(sector_t offset, */ static inline struct queue_limits queue_limits_start_update(struct request_queue *q) + __acquires(&q->limits_lock) { mutex_lock(&q->limits_lock); return q->limits; } int queue_limits_commit_update_frozen(struct request_queue *q, - struct queue_limits *lim); + struct queue_limits *lim) __releases(&q->limits_lock); int queue_limits_commit_update(struct request_queue *q, - struct queue_limits *lim); -int queue_limits_set(struct request_queue *q, struct queue_limits *lim); + struct queue_limits *lim) __releases(&q->limits_lock); +int queue_limits_set(struct request_queue *q, struct queue_limits *lim) + __must_not_hold(&q->limits_lock); int blk_validate_limits(struct queue_limits *lim); /** @@ -1113,6 +1120,7 @@ int blk_validate_limits(struct queue_limits *lim); * starting update. */ static inline void queue_limits_cancel_update(struct request_queue *q) + __releases(&q->limits_lock) { mutex_unlock(&q->limits_lock); } @@ -1744,22 +1752,26 @@ void blkdev_show(struct seq_file *seqf, off_t offset); #endif struct blk_holder_ops { - void (*mark_dead)(struct block_device *bdev, bool surprise); + void (*mark_dead)(struct block_device *bdev, bool surprise) + __releases(&bdev->bd_holder_lock); /* * Sync the file system mounted on the block device. */ - void (*sync)(struct block_device *bdev); + void (*sync)(struct block_device *bdev) + __releases(&bdev->bd_holder_lock); /* * Freeze the file system mounted on the block device. */ - int (*freeze)(struct block_device *bdev); + int (*freeze)(struct block_device *bdev) + __releases(&bdev->bd_holder_lock); /* * Thaw the file system mounted on the block device. */ - int (*thaw)(struct block_device *bdev); + int (*thaw)(struct block_device *bdev) + __releases(&bdev->bd_holder_lock); }; /* diff --git a/include/linux/bvec.h b/include/linux/bvec.h index d36dd476feda..92837e2743f1 100644 --- a/include/linux/bvec.h +++ b/include/linux/bvec.h @@ -74,6 +74,21 @@ static inline void bvec_set_virt(struct bio_vec *bv, void *vaddr, bvec_set_page(bv, virt_to_page(vaddr), len, offset_in_page(vaddr)); } +/** + * bvec_folio - Return the first folio referenced by this bvec + * @bv: bvec to access + * + * A bvec can contain non-folio memory, so this should only be called by + * the creator of the bvec; drivers have no business looking at the owner + * of the memory. It may not even be the right interface for the caller + * to use as a bvec can span multiple folios. You may be better off using + * something like bio_for_each_folio_all() which iterates over all folios. + */ +static inline struct folio *bvec_folio(const struct bio_vec *bv) +{ + return page_folio(bv->bv_page); +} + struct bvec_iter { /* * Current device address in 512 byte sectors. Only updated by the bio @@ -104,51 +119,78 @@ struct bvec_iter_all { unsigned done; }; -/* - * various member access, note that bio_data should of course not be used - * on highmem page vectors - */ -#define __bvec_iter_bvec(bvec, iter) (&(bvec)[(iter).bi_idx]) +static __always_inline const struct bio_vec * +__bvec_iter_bvec(const struct bio_vec *bvecs, const struct bvec_iter iter) +{ + return bvecs + iter.bi_idx; +} /* multi-page (mp_bvec) helpers */ -#define mp_bvec_iter_page(bvec, iter) \ - (__bvec_iter_bvec((bvec), (iter))->bv_page) +static __always_inline struct page * +mp_bvec_iter_page(const struct bio_vec *bvecs, const struct bvec_iter iter) +{ + return __bvec_iter_bvec(bvecs, iter)->bv_page; +} -#define mp_bvec_iter_len(bvec, iter) \ - min((iter).bi_size, \ - __bvec_iter_bvec((bvec), (iter))->bv_len - (iter).bi_bvec_done) +static __always_inline unsigned int +mp_bvec_iter_len(const struct bio_vec *bvecs, const struct bvec_iter iter) +{ + return min(__bvec_iter_bvec(bvecs, iter)->bv_len - iter.bi_bvec_done, + iter.bi_size); +} -#define mp_bvec_iter_offset(bvec, iter) \ - (__bvec_iter_bvec((bvec), (iter))->bv_offset + (iter).bi_bvec_done) +static __always_inline unsigned int +mp_bvec_iter_offset(const struct bio_vec *bvecs, const struct bvec_iter iter) +{ + return __bvec_iter_bvec(bvecs, iter)->bv_offset + iter.bi_bvec_done; +} -#define mp_bvec_iter_page_idx(bvec, iter) \ - (mp_bvec_iter_offset((bvec), (iter)) / PAGE_SIZE) +static __always_inline unsigned int +mp_bvec_iter_page_idx(const struct bio_vec *bvecs, const struct bvec_iter iter) +{ + return mp_bvec_iter_offset(bvecs, iter) / PAGE_SIZE; +} -#define mp_bvec_iter_bvec(bvec, iter) \ -((struct bio_vec) { \ - .bv_page = mp_bvec_iter_page((bvec), (iter)), \ - .bv_len = mp_bvec_iter_len((bvec), (iter)), \ - .bv_offset = mp_bvec_iter_offset((bvec), (iter)), \ -}) +static __always_inline struct bio_vec +mp_bvec_iter_bvec(const struct bio_vec *bvecs, const struct bvec_iter iter) +{ + return (struct bio_vec) { + .bv_page = mp_bvec_iter_page(bvecs, iter), + .bv_len = mp_bvec_iter_len(bvecs, iter), + .bv_offset = mp_bvec_iter_offset(bvecs, iter), + }; +} /* For building single-page bvec in flight */ - #define bvec_iter_offset(bvec, iter) \ - (mp_bvec_iter_offset((bvec), (iter)) % PAGE_SIZE) +static __always_inline unsigned int +bvec_iter_offset(const struct bio_vec *bvecs, const struct bvec_iter iter) +{ + return mp_bvec_iter_offset(bvecs, iter) % PAGE_SIZE; +} -#define bvec_iter_len(bvec, iter) \ - min_t(unsigned, mp_bvec_iter_len((bvec), (iter)), \ - PAGE_SIZE - bvec_iter_offset((bvec), (iter))) +static __always_inline unsigned int +bvec_iter_len(const struct bio_vec *bvecs, const struct bvec_iter iter) +{ + return min(mp_bvec_iter_len(bvecs, iter), + PAGE_SIZE - bvec_iter_offset(bvecs, iter)); +} -#define bvec_iter_page(bvec, iter) \ - (mp_bvec_iter_page((bvec), (iter)) + \ - mp_bvec_iter_page_idx((bvec), (iter))) +static __always_inline struct page * +bvec_iter_page(const struct bio_vec *bvecs, const struct bvec_iter iter) +{ + return mp_bvec_iter_page(bvecs, iter) + + mp_bvec_iter_page_idx(bvecs, iter); +} -#define bvec_iter_bvec(bvec, iter) \ -((struct bio_vec) { \ - .bv_page = bvec_iter_page((bvec), (iter)), \ - .bv_len = bvec_iter_len((bvec), (iter)), \ - .bv_offset = bvec_iter_offset((bvec), (iter)), \ -}) +static __always_inline struct bio_vec +bvec_iter_bvec(const struct bio_vec *bvecs, const struct bvec_iter iter) +{ + return (struct bio_vec) { + .bv_page = bvec_iter_page(bvecs, iter), + .bv_len = bvec_iter_len(bvecs, iter), + .bv_offset = bvec_iter_offset(bvecs, iter), + }; +} static inline bool bvec_iter_advance(const struct bio_vec *bv, struct bvec_iter *iter, unsigned bytes) @@ -247,6 +289,7 @@ static inline void *bvec_kmap_local(struct bio_vec *bvec) /** * memcpy_from_bvec - copy data from a bvec + * @to: Kernel virtual address to copy to. * @bvec: bvec to copy from * * Must be called on single-page bvecs only. @@ -259,6 +302,7 @@ static inline void memcpy_from_bvec(char *to, struct bio_vec *bvec) /** * memcpy_to_bvec - copy data to a bvec * @bvec: bvec to copy to + * @from: Kernel virtual address to copy from. * * Must be called on single-page bvecs only. */ diff --git a/include/linux/drbd_genl.h b/include/linux/drbd_genl.h deleted file mode 100644 index f53c534aba0c..000000000000 --- a/include/linux/drbd_genl.h +++ /dev/null @@ -1,536 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -/* - * General overview: - * full generic netlink message: - * |nlmsghdr|genlmsghdr|<payload> - * - * payload: - * |optional fixed size family header|<sequence of netlink attributes> - * - * sequence of netlink attributes: - * I chose to have all "top level" attributes NLA_NESTED, - * corresponding to some real struct. - * So we have a sequence of |tla, len|<nested nla sequence> - * - * nested nla sequence: - * may be empty, or contain a sequence of netlink attributes - * representing the struct fields. - * - * The tag number of any field (regardless of containing struct) - * will be available as T_ ## field_name, - * so you cannot have the same field name in two differnt structs. - * - * The tag numbers themselves are per struct, though, - * so should always begin at 1 (not 0, that is the special "NLA_UNSPEC" type, - * which we won't use here). - * The tag numbers are used as index in the respective nla_policy array. - * - * GENL_struct(tag_name, tag_number, struct name, struct fields) - struct and policy - * genl_magic_struct.h - * generates the struct declaration, - * generates an entry in the tla enum, - * genl_magic_func.h - * generates an entry in the static tla policy - * with .type = NLA_NESTED - * generates the static <struct_name>_nl_policy definition, - * and static conversion functions - * - * genl_magic_func.h - * - * GENL_mc_group(group) - * genl_magic_struct.h - * does nothing - * genl_magic_func.h - * defines and registers the mcast group, - * and provides a send helper - * - * GENL_notification(op_name, op_num, mcast_group, tla list) - * These are notifications to userspace. - * - * genl_magic_struct.h - * generates an entry in the genl_ops enum, - * genl_magic_func.h - * does nothing - * - * mcast group: the name of the mcast group this notification should be - * expected on - * tla list: the list of expected top level attributes, - * for documentation and sanity checking. - * - * GENL_op(op_name, op_num, flags and handler, tla list) - "genl operations" - * These are requests from userspace. - * - * _op and _notification share the same "number space", - * op_nr will be assigned to "genlmsghdr->cmd" - * - * genl_magic_struct.h - * generates an entry in the genl_ops enum, - * genl_magic_func.h - * generates an entry in the static genl_ops array, - * and static register/unregister functions to - * genl_register_family(). - * - * flags and handler: - * GENL_op_init( .doit = x, .dumpit = y, .flags = something) - * GENL_doit(x) => .dumpit = NULL, .flags = GENL_ADMIN_PERM - * tla list: the list of expected top level attributes, - * for documentation and sanity checking. - */ - -/* - * STRUCTS - */ - -/* this is sent kernel -> userland on various error conditions, and contains - * informational textual info, which is supposedly human readable. - * The computer relevant return code is in the drbd_genlmsghdr. - */ -GENL_struct(DRBD_NLA_CFG_REPLY, 1, drbd_cfg_reply, - /* "arbitrary" size strings, nla_policy.len = 0 */ - __str_field(1, 0, info_text, 0) -) - -/* Configuration requests typically need a context to operate on. - * Possible keys are device minor (fits in the drbd_genlmsghdr), - * the replication link (aka connection) name, - * and/or the replication group (aka resource) name, - * and the volume id within the resource. */ -GENL_struct(DRBD_NLA_CFG_CONTEXT, 2, drbd_cfg_context, - __u32_field(1, 0, ctx_volume) - __str_field(2, 0, ctx_resource_name, 128) - __bin_field(3, 0, ctx_my_addr, 128) - __bin_field(4, 0, ctx_peer_addr, 128) -) - -GENL_struct(DRBD_NLA_DISK_CONF, 3, disk_conf, - __str_field(1, DRBD_F_REQUIRED | DRBD_F_INVARIANT, backing_dev, 128) - __str_field(2, DRBD_F_REQUIRED | DRBD_F_INVARIANT, meta_dev, 128) - __s32_field(3, DRBD_F_REQUIRED | DRBD_F_INVARIANT, meta_dev_idx) - - /* use the resize command to try and change the disk_size */ - __u64_field(4, DRBD_F_INVARIANT, disk_size) - /* we could change the max_bio_bvecs, - * but it won't propagate through the stack */ - __u32_field(5, DRBD_F_INVARIANT, max_bio_bvecs) - - __u32_field_def(6, 0, on_io_error, DRBD_ON_IO_ERROR_DEF) - __u32_field_def(7, 0, fencing, DRBD_FENCING_DEF) - - __u32_field_def(8, 0, resync_rate, DRBD_RESYNC_RATE_DEF) - __s32_field_def(9, 0, resync_after, DRBD_MINOR_NUMBER_DEF) - __u32_field_def(10, 0, al_extents, DRBD_AL_EXTENTS_DEF) - __u32_field_def(11, 0, c_plan_ahead, DRBD_C_PLAN_AHEAD_DEF) - __u32_field_def(12, 0, c_delay_target, DRBD_C_DELAY_TARGET_DEF) - __u32_field_def(13, 0, c_fill_target, DRBD_C_FILL_TARGET_DEF) - __u32_field_def(14, 0, c_max_rate, DRBD_C_MAX_RATE_DEF) - __u32_field_def(15, 0, c_min_rate, DRBD_C_MIN_RATE_DEF) - __u32_field_def(20, 0, disk_timeout, DRBD_DISK_TIMEOUT_DEF) - __u32_field_def(21, 0 /* OPTIONAL */, read_balancing, DRBD_READ_BALANCING_DEF) - __u32_field_def(25, 0 /* OPTIONAL */, rs_discard_granularity, DRBD_RS_DISCARD_GRANULARITY_DEF) - - __flg_field_def(16, 0, disk_barrier, DRBD_DISK_BARRIER_DEF) - __flg_field_def(17, 0, disk_flushes, DRBD_DISK_FLUSHES_DEF) - __flg_field_def(18, 0, disk_drain, DRBD_DISK_DRAIN_DEF) - __flg_field_def(19, 0, md_flushes, DRBD_MD_FLUSHES_DEF) - __flg_field_def(23, 0 /* OPTIONAL */, al_updates, DRBD_AL_UPDATES_DEF) - __flg_field_def(24, 0 /* OPTIONAL */, discard_zeroes_if_aligned, DRBD_DISCARD_ZEROES_IF_ALIGNED_DEF) - __flg_field_def(26, 0 /* OPTIONAL */, disable_write_same, DRBD_DISABLE_WRITE_SAME_DEF) -) - -GENL_struct(DRBD_NLA_RESOURCE_OPTS, 4, res_opts, - __str_field_def(1, 0, cpu_mask, DRBD_CPU_MASK_SIZE) - __u32_field_def(2, 0, on_no_data, DRBD_ON_NO_DATA_DEF) -) - -GENL_struct(DRBD_NLA_NET_CONF, 5, net_conf, - __str_field_def(1, DRBD_F_SENSITIVE, - shared_secret, SHARED_SECRET_MAX) - __str_field_def(2, 0, cram_hmac_alg, SHARED_SECRET_MAX) - __str_field_def(3, 0, integrity_alg, SHARED_SECRET_MAX) - __str_field_def(4, 0, verify_alg, SHARED_SECRET_MAX) - __str_field_def(5, 0, csums_alg, SHARED_SECRET_MAX) - __u32_field_def(6, 0, wire_protocol, DRBD_PROTOCOL_DEF) - __u32_field_def(7, 0, connect_int, DRBD_CONNECT_INT_DEF) - __u32_field_def(8, 0, timeout, DRBD_TIMEOUT_DEF) - __u32_field_def(9, 0, ping_int, DRBD_PING_INT_DEF) - __u32_field_def(10, 0, ping_timeo, DRBD_PING_TIMEO_DEF) - __u32_field_def(11, 0, sndbuf_size, DRBD_SNDBUF_SIZE_DEF) - __u32_field_def(12, 0, rcvbuf_size, DRBD_RCVBUF_SIZE_DEF) - __u32_field_def(13, 0, ko_count, DRBD_KO_COUNT_DEF) - __u32_field_def(14, 0, max_buffers, DRBD_MAX_BUFFERS_DEF) - __u32_field_def(15, 0, max_epoch_size, DRBD_MAX_EPOCH_SIZE_DEF) - __u32_field_def(16, 0, unplug_watermark, DRBD_UNPLUG_WATERMARK_DEF) - __u32_field_def(17, 0, after_sb_0p, DRBD_AFTER_SB_0P_DEF) - __u32_field_def(18, 0, after_sb_1p, DRBD_AFTER_SB_1P_DEF) - __u32_field_def(19, 0, after_sb_2p, DRBD_AFTER_SB_2P_DEF) - __u32_field_def(20, 0, rr_conflict, DRBD_RR_CONFLICT_DEF) - __u32_field_def(21, 0, on_congestion, DRBD_ON_CONGESTION_DEF) - __u32_field_def(22, 0, cong_fill, DRBD_CONG_FILL_DEF) - __u32_field_def(23, 0, cong_extents, DRBD_CONG_EXTENTS_DEF) - __flg_field_def(24, 0, two_primaries, DRBD_ALLOW_TWO_PRIMARIES_DEF) - __flg_field(25, DRBD_F_INVARIANT, discard_my_data) - __flg_field_def(26, 0, tcp_cork, DRBD_TCP_CORK_DEF) - __flg_field_def(27, 0, always_asbp, DRBD_ALWAYS_ASBP_DEF) - __flg_field(28, DRBD_F_INVARIANT, tentative) - __flg_field_def(29, 0, use_rle, DRBD_USE_RLE_DEF) - /* 9: __u32_field_def(30, 0, fencing_policy, DRBD_FENCING_DEF) */ - /* 9: __str_field_def(31, 0, name, SHARED_SECRET_MAX) */ - /* 9: __u32_field(32, DRBD_F_REQUIRED | DRBD_F_INVARIANT, peer_node_id) */ - __flg_field_def(33, 0 /* OPTIONAL */, csums_after_crash_only, DRBD_CSUMS_AFTER_CRASH_ONLY_DEF) - __u32_field_def(34, 0 /* OPTIONAL */, sock_check_timeo, DRBD_SOCKET_CHECK_TIMEO_DEF) -) - -GENL_struct(DRBD_NLA_SET_ROLE_PARMS, 6, set_role_parms, - __flg_field(1, 0, assume_uptodate) -) - -GENL_struct(DRBD_NLA_RESIZE_PARMS, 7, resize_parms, - __u64_field(1, 0, resize_size) - __flg_field(2, 0, resize_force) - __flg_field(3, 0, no_resync) - __u32_field_def(4, 0 /* OPTIONAL */, al_stripes, DRBD_AL_STRIPES_DEF) - __u32_field_def(5, 0 /* OPTIONAL */, al_stripe_size, DRBD_AL_STRIPE_SIZE_DEF) -) - -GENL_struct(DRBD_NLA_STATE_INFO, 8, state_info, - /* the reason of the broadcast, - * if this is an event triggered broadcast. */ - __u32_field(1, 0, sib_reason) - __u32_field(2, DRBD_F_REQUIRED, current_state) - __u64_field(3, 0, capacity) - __u64_field(4, 0, ed_uuid) - - /* These are for broadcast from after state change work. - * prev_state and new_state are from the moment the state change took - * place, new_state is not neccessarily the same as current_state, - * there may have been more state changes since. Which will be - * broadcasted soon, in their respective after state change work. */ - __u32_field(5, 0, prev_state) - __u32_field(6, 0, new_state) - - /* if we have a local disk: */ - __bin_field(7, 0, uuids, (UI_SIZE*sizeof(__u64))) - __u32_field(8, 0, disk_flags) - __u64_field(9, 0, bits_total) - __u64_field(10, 0, bits_oos) - /* and in case resync or online verify is active */ - __u64_field(11, 0, bits_rs_total) - __u64_field(12, 0, bits_rs_failed) - - /* for pre and post notifications of helper execution */ - __str_field(13, 0, helper, 32) - __u32_field(14, 0, helper_exit_code) - - __u64_field(15, 0, send_cnt) - __u64_field(16, 0, recv_cnt) - __u64_field(17, 0, read_cnt) - __u64_field(18, 0, writ_cnt) - __u64_field(19, 0, al_writ_cnt) - __u64_field(20, 0, bm_writ_cnt) - __u32_field(21, 0, ap_bio_cnt) - __u32_field(22, 0, ap_pending_cnt) - __u32_field(23, 0, rs_pending_cnt) -) - -GENL_struct(DRBD_NLA_START_OV_PARMS, 9, start_ov_parms, - __u64_field(1, 0, ov_start_sector) - __u64_field(2, 0, ov_stop_sector) -) - -GENL_struct(DRBD_NLA_NEW_C_UUID_PARMS, 10, new_c_uuid_parms, - __flg_field(1, 0, clear_bm) -) - -GENL_struct(DRBD_NLA_TIMEOUT_PARMS, 11, timeout_parms, - __u32_field(1, DRBD_F_REQUIRED, timeout_type) -) - -GENL_struct(DRBD_NLA_DISCONNECT_PARMS, 12, disconnect_parms, - __flg_field(1, 0, force_disconnect) -) - -GENL_struct(DRBD_NLA_DETACH_PARMS, 13, detach_parms, - __flg_field(1, 0, force_detach) -) - -GENL_struct(DRBD_NLA_RESOURCE_INFO, 15, resource_info, - __u32_field(1, 0, res_role) - __flg_field(2, 0, res_susp) - __flg_field(3, 0, res_susp_nod) - __flg_field(4, 0, res_susp_fen) - /* __flg_field(5, 0, res_weak) */ -) - -GENL_struct(DRBD_NLA_DEVICE_INFO, 16, device_info, - __u32_field(1, 0, dev_disk_state) -) - -GENL_struct(DRBD_NLA_CONNECTION_INFO, 17, connection_info, - __u32_field(1, 0, conn_connection_state) - __u32_field(2, 0, conn_role) -) - -GENL_struct(DRBD_NLA_PEER_DEVICE_INFO, 18, peer_device_info, - __u32_field(1, 0, peer_repl_state) - __u32_field(2, 0, peer_disk_state) - __u32_field(3, 0, peer_resync_susp_user) - __u32_field(4, 0, peer_resync_susp_peer) - __u32_field(5, 0, peer_resync_susp_dependency) -) - -GENL_struct(DRBD_NLA_RESOURCE_STATISTICS, 19, resource_statistics, - __u32_field(1, 0, res_stat_write_ordering) -) - -GENL_struct(DRBD_NLA_DEVICE_STATISTICS, 20, device_statistics, - __u64_field(1, 0, dev_size) /* (sectors) */ - __u64_field(2, 0, dev_read) /* (sectors) */ - __u64_field(3, 0, dev_write) /* (sectors) */ - __u64_field(4, 0, dev_al_writes) /* activity log writes (count) */ - __u64_field(5, 0, dev_bm_writes) /* bitmap writes (count) */ - __u32_field(6, 0, dev_upper_pending) /* application requests in progress */ - __u32_field(7, 0, dev_lower_pending) /* backing device requests in progress */ - __flg_field(8, 0, dev_upper_blocked) - __flg_field(9, 0, dev_lower_blocked) - __flg_field(10, 0, dev_al_suspended) /* activity log suspended */ - __u64_field(11, 0, dev_exposed_data_uuid) - __u64_field(12, 0, dev_current_uuid) - __u32_field(13, 0, dev_disk_flags) - __bin_field(14, 0, history_uuids, HISTORY_UUIDS * sizeof(__u64)) -) - -GENL_struct(DRBD_NLA_CONNECTION_STATISTICS, 21, connection_statistics, - __flg_field(1, 0, conn_congested) -) - -GENL_struct(DRBD_NLA_PEER_DEVICE_STATISTICS, 22, peer_device_statistics, - __u64_field(1, 0, peer_dev_received) /* sectors */ - __u64_field(2, 0, peer_dev_sent) /* sectors */ - __u32_field(3, 0, peer_dev_pending) /* number of requests */ - __u32_field(4, 0, peer_dev_unacked) /* number of requests */ - __u64_field(5, 0, peer_dev_out_of_sync) /* sectors */ - __u64_field(6, 0, peer_dev_resync_failed) /* sectors */ - __u64_field(7, 0, peer_dev_bitmap_uuid) - __u32_field(9, 0, peer_dev_flags) -) - -GENL_struct(DRBD_NLA_NOTIFICATION_HEADER, 23, drbd_notification_header, - __u32_field(1, 0, nh_type) -) - -GENL_struct(DRBD_NLA_HELPER, 24, drbd_helper_info, - __str_field(1, 0, helper_name, 32) - __u32_field(2, 0, helper_status) -) - -/* - * Notifications and commands (genlmsghdr->cmd) - */ -GENL_mc_group(events) - - /* kernel -> userspace announcement of changes */ -GENL_notification( - DRBD_EVENT, 1, events, - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_STATE_INFO, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_NET_CONF, 0) - GENL_tla_expected(DRBD_NLA_DISK_CONF, 0) - GENL_tla_expected(DRBD_NLA_SYNCER_CONF, 0) -) - - /* query kernel for specific or all info */ -GENL_op( - DRBD_ADM_GET_STATUS, 2, - GENL_op_init( - .doit = drbd_adm_get_status, - .dumpit = drbd_adm_get_status_all, - /* anyone may ask for the status, - * it is broadcasted anyways */ - ), - /* To select the object .doit. - * Or a subset of objects in .dumpit. */ - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, 0) -) - - /* add DRBD minor devices as volumes to resources */ -GENL_op(DRBD_ADM_NEW_MINOR, 5, GENL_doit(drbd_adm_new_minor), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)) -GENL_op(DRBD_ADM_DEL_MINOR, 6, GENL_doit(drbd_adm_del_minor), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)) - - /* add or delete resources */ -GENL_op(DRBD_ADM_NEW_RESOURCE, 7, GENL_doit(drbd_adm_new_resource), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)) -GENL_op(DRBD_ADM_DEL_RESOURCE, 8, GENL_doit(drbd_adm_del_resource), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)) - -GENL_op(DRBD_ADM_RESOURCE_OPTS, 9, - GENL_doit(drbd_adm_resource_opts), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_RESOURCE_OPTS, 0) -) - -GENL_op( - DRBD_ADM_CONNECT, 10, - GENL_doit(drbd_adm_connect), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_NET_CONF, DRBD_F_REQUIRED) -) - -GENL_op( - DRBD_ADM_CHG_NET_OPTS, 29, - GENL_doit(drbd_adm_net_opts), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_NET_CONF, DRBD_F_REQUIRED) -) - -GENL_op(DRBD_ADM_DISCONNECT, 11, GENL_doit(drbd_adm_disconnect), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)) - -GENL_op(DRBD_ADM_ATTACH, 12, - GENL_doit(drbd_adm_attach), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_DISK_CONF, DRBD_F_REQUIRED) -) - -GENL_op(DRBD_ADM_CHG_DISK_OPTS, 28, - GENL_doit(drbd_adm_disk_opts), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_DISK_OPTS, DRBD_F_REQUIRED) -) - -GENL_op( - DRBD_ADM_RESIZE, 13, - GENL_doit(drbd_adm_resize), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_RESIZE_PARMS, 0) -) - -GENL_op( - DRBD_ADM_PRIMARY, 14, - GENL_doit(drbd_adm_set_role), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_SET_ROLE_PARMS, DRBD_F_REQUIRED) -) - -GENL_op( - DRBD_ADM_SECONDARY, 15, - GENL_doit(drbd_adm_set_role), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_SET_ROLE_PARMS, DRBD_F_REQUIRED) -) - -GENL_op( - DRBD_ADM_NEW_C_UUID, 16, - GENL_doit(drbd_adm_new_c_uuid), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_NEW_C_UUID_PARMS, 0) -) - -GENL_op( - DRBD_ADM_START_OV, 17, - GENL_doit(drbd_adm_start_ov), - GENL_tla_expected(DRBD_NLA_START_OV_PARMS, 0) -) - -GENL_op(DRBD_ADM_DETACH, 18, GENL_doit(drbd_adm_detach), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_DETACH_PARMS, 0)) - -GENL_op(DRBD_ADM_INVALIDATE, 19, GENL_doit(drbd_adm_invalidate), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)) -GENL_op(DRBD_ADM_INVAL_PEER, 20, GENL_doit(drbd_adm_invalidate_peer), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)) -GENL_op(DRBD_ADM_PAUSE_SYNC, 21, GENL_doit(drbd_adm_pause_sync), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)) -GENL_op(DRBD_ADM_RESUME_SYNC, 22, GENL_doit(drbd_adm_resume_sync), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)) -GENL_op(DRBD_ADM_SUSPEND_IO, 23, GENL_doit(drbd_adm_suspend_io), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)) -GENL_op(DRBD_ADM_RESUME_IO, 24, GENL_doit(drbd_adm_resume_io), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)) -GENL_op(DRBD_ADM_OUTDATE, 25, GENL_doit(drbd_adm_outdate), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)) -GENL_op(DRBD_ADM_GET_TIMEOUT_TYPE, 26, GENL_doit(drbd_adm_get_timeout_type), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)) -GENL_op(DRBD_ADM_DOWN, 27, GENL_doit(drbd_adm_down), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED)) - -GENL_op(DRBD_ADM_GET_RESOURCES, 30, - GENL_op_init( - .dumpit = drbd_adm_dump_resources, - ), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, 0) - GENL_tla_expected(DRBD_NLA_RESOURCE_INFO, 0) - GENL_tla_expected(DRBD_NLA_RESOURCE_STATISTICS, 0)) - -GENL_op(DRBD_ADM_GET_DEVICES, 31, - GENL_op_init( - .dumpit = drbd_adm_dump_devices, - .done = drbd_adm_dump_devices_done, - ), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, 0) - GENL_tla_expected(DRBD_NLA_DEVICE_INFO, 0) - GENL_tla_expected(DRBD_NLA_DEVICE_STATISTICS, 0)) - -GENL_op(DRBD_ADM_GET_CONNECTIONS, 32, - GENL_op_init( - .dumpit = drbd_adm_dump_connections, - .done = drbd_adm_dump_connections_done, - ), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, 0) - GENL_tla_expected(DRBD_NLA_CONNECTION_INFO, 0) - GENL_tla_expected(DRBD_NLA_CONNECTION_STATISTICS, 0)) - -GENL_op(DRBD_ADM_GET_PEER_DEVICES, 33, - GENL_op_init( - .dumpit = drbd_adm_dump_peer_devices, - .done = drbd_adm_dump_peer_devices_done, - ), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, 0) - GENL_tla_expected(DRBD_NLA_PEER_DEVICE_INFO, 0) - GENL_tla_expected(DRBD_NLA_PEER_DEVICE_STATISTICS, 0)) - -GENL_notification( - DRBD_RESOURCE_STATE, 34, events, - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_NOTIFICATION_HEADER, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_RESOURCE_INFO, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_RESOURCE_STATISTICS, DRBD_F_REQUIRED)) - -GENL_notification( - DRBD_DEVICE_STATE, 35, events, - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_NOTIFICATION_HEADER, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_DEVICE_INFO, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_DEVICE_STATISTICS, DRBD_F_REQUIRED)) - -GENL_notification( - DRBD_CONNECTION_STATE, 36, events, - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_NOTIFICATION_HEADER, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_CONNECTION_INFO, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_CONNECTION_STATISTICS, DRBD_F_REQUIRED)) - -GENL_notification( - DRBD_PEER_DEVICE_STATE, 37, events, - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_NOTIFICATION_HEADER, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_PEER_DEVICE_INFO, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_PEER_DEVICE_STATISTICS, DRBD_F_REQUIRED)) - -GENL_op( - DRBD_ADM_GET_INITIAL_STATE, 38, - GENL_op_init( - .dumpit = drbd_adm_get_initial_state, - ), - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, 0)) - -GENL_notification( - DRBD_HELPER, 40, events, - GENL_tla_expected(DRBD_NLA_CFG_CONTEXT, DRBD_F_REQUIRED) - GENL_tla_expected(DRBD_NLA_HELPER, DRBD_F_REQUIRED)) - -GENL_notification( - DRBD_INITIAL_STATE_DONE, 41, events, - GENL_tla_expected(DRBD_NLA_NOTIFICATION_HEADER, DRBD_F_REQUIRED)) diff --git a/include/linux/drbd_genl_api.h b/include/linux/drbd_genl_api.h deleted file mode 100644 index 70682c058027..000000000000 --- a/include/linux/drbd_genl_api.h +++ /dev/null @@ -1,56 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef DRBD_GENL_STRUCT_H -#define DRBD_GENL_STRUCT_H - -/** - * struct drbd_genlmsghdr - DRBD specific header used in NETLINK_GENERIC requests - * @minor: - * For admin requests (user -> kernel): which minor device to operate on. - * For (unicast) replies or informational (broadcast) messages - * (kernel -> user): which minor device the information is about. - * If we do not operate on minors, but on connections or resources, - * the minor value shall be (~0), and the attribute DRBD_NLA_CFG_CONTEXT - * is used instead. - * @flags: possible operation modifiers (relevant only for user->kernel): - * DRBD_GENL_F_SET_DEFAULTS - * @volume: - * When creating a new minor (adding it to a resource), the resource needs - * to know which volume number within the resource this is supposed to be. - * The volume number corresponds to the same volume number on the remote side, - * whereas the minor number on the remote side may be different - * (union with flags). - * @ret_code: kernel->userland unicast cfg reply return code (union with flags); - */ -struct drbd_genlmsghdr { - __u32 minor; - union { - __u32 flags; - __s32 ret_code; - }; -}; - -/* To be used in drbd_genlmsghdr.flags */ -enum { - DRBD_GENL_F_SET_DEFAULTS = 1, -}; - -enum drbd_state_info_bcast_reason { - SIB_GET_STATUS_REPLY = 1, - SIB_STATE_CHANGE = 2, - SIB_HELPER_PRE = 3, - SIB_HELPER_POST = 4, - SIB_SYNC_PROGRESS = 5, -}; - -/* hack around predefined gcc/cpp "linux=1", - * we cannot possibly include <1/drbd_genl.h> */ -#undef linux - -#include <linux/drbd.h> -#define GENL_MAGIC_VERSION 1 -#define GENL_MAGIC_FAMILY drbd -#define GENL_MAGIC_FAMILY_HDRSZ sizeof(struct drbd_genlmsghdr) -#define GENL_MAGIC_INCLUDE_FILE <linux/drbd_genl.h> -#include <linux/genl_magic_struct.h> - -#endif diff --git a/include/linux/genl_magic_func.h b/include/linux/genl_magic_func.h deleted file mode 100644 index a7d36c9ea924..000000000000 --- a/include/linux/genl_magic_func.h +++ /dev/null @@ -1,413 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef GENL_MAGIC_FUNC_H -#define GENL_MAGIC_FUNC_H - -#include <linux/args.h> -#include <linux/build_bug.h> -#include <linux/genl_magic_struct.h> - -/* - * Magic: declare tla policy {{{1 - * Magic: declare nested policies - * {{{2 - */ -#undef GENL_mc_group -#define GENL_mc_group(group) - -#undef GENL_notification -#define GENL_notification(op_name, op_num, mcast_group, tla_list) - -#undef GENL_op -#define GENL_op(op_name, op_num, handler, tla_list) - -#undef GENL_struct -#define GENL_struct(tag_name, tag_number, s_name, s_fields) \ - [tag_name] = { .type = NLA_NESTED }, - -static struct nla_policy CONCATENATE(GENL_MAGIC_FAMILY, _tla_nl_policy)[] = { -#include GENL_MAGIC_INCLUDE_FILE -}; - -#undef GENL_struct -#define GENL_struct(tag_name, tag_number, s_name, s_fields) \ -static struct nla_policy s_name ## _nl_policy[] __read_mostly = \ -{ s_fields }; - -#undef __field -#define __field(attr_nr, attr_flag, name, nla_type, _type, __get, \ - __put, __is_signed) \ - [attr_nr] = { .type = nla_type }, - -#undef __array -#define __array(attr_nr, attr_flag, name, nla_type, _type, maxlen, \ - __get, __put, __is_signed) \ - [attr_nr] = { .type = nla_type, \ - .len = maxlen - (nla_type == NLA_NUL_STRING) }, - -#include GENL_MAGIC_INCLUDE_FILE - -#ifndef __KERNEL__ -#ifndef pr_info -#define pr_info(args...) fprintf(stderr, args); -#endif -#endif - -#ifdef GENL_MAGIC_DEBUG -static void dprint_field(const char *dir, int nla_type, - const char *name, void *valp) -{ - __u64 val = valp ? *(__u32 *)valp : 1; - switch (nla_type) { - case NLA_U8: val = (__u8)val; - case NLA_U16: val = (__u16)val; - case NLA_U32: val = (__u32)val; - pr_info("%s attr %s: %d 0x%08x\n", dir, - name, (int)val, (unsigned)val); - break; - case NLA_U64: - val = *(__u64*)valp; - pr_info("%s attr %s: %lld 0x%08llx\n", dir, - name, (long long)val, (unsigned long long)val); - break; - case NLA_FLAG: - if (val) - pr_info("%s attr %s: set\n", dir, name); - break; - } -} - -static void dprint_array(const char *dir, int nla_type, - const char *name, const char *val, unsigned len) -{ - switch (nla_type) { - case NLA_NUL_STRING: - if (len && val[len-1] == '\0') - len--; - pr_info("%s attr %s: [len:%u] '%s'\n", dir, name, len, val); - break; - default: - /* we can always show 4 byte, - * thats what nlattr are aligned to. */ - pr_info("%s attr %s: [len:%u] %02x%02x%02x%02x ...\n", - dir, name, len, val[0], val[1], val[2], val[3]); - } -} - -#define DPRINT_TLA(a, op, b) pr_info("%s %s %s\n", a, op, b); - -/* Name is a member field name of the struct s. - * If s is NULL (only parsing, no copy requested in *_from_attrs()), - * nla is supposed to point to the attribute containing the information - * corresponding to that struct member. */ -#define DPRINT_FIELD(dir, nla_type, name, s, nla) \ - do { \ - if (s) \ - dprint_field(dir, nla_type, #name, &s->name); \ - else if (nla) \ - dprint_field(dir, nla_type, #name, \ - (nla_type == NLA_FLAG) ? NULL \ - : nla_data(nla)); \ - } while (0) - -#define DPRINT_ARRAY(dir, nla_type, name, s, nla) \ - do { \ - if (s) \ - dprint_array(dir, nla_type, #name, \ - s->name, s->name ## _len); \ - else if (nla) \ - dprint_array(dir, nla_type, #name, \ - nla_data(nla), nla_len(nla)); \ - } while (0) -#else -#define DPRINT_TLA(a, op, b) do {} while (0) -#define DPRINT_FIELD(dir, nla_type, name, s, nla) do {} while (0) -#define DPRINT_ARRAY(dir, nla_type, name, s, nla) do {} while (0) -#endif - -/* - * Magic: provide conversion functions {{{1 - * populate struct from attribute table: - * {{{2 - */ - -/* processing of generic netlink messages is serialized. - * use one static buffer for parsing of nested attributes */ -static struct nlattr *nested_attr_tb[128]; - -#undef GENL_struct -#define GENL_struct(tag_name, tag_number, s_name, s_fields) \ -/* *_from_attrs functions are static, but potentially unused */ \ -static int __ ## s_name ## _from_attrs(struct s_name *s, \ - struct genl_info *info, bool exclude_invariants) \ -{ \ - const int maxtype = ARRAY_SIZE(s_name ## _nl_policy)-1; \ - struct nlattr *tla = info->attrs[tag_number]; \ - struct nlattr **ntb = nested_attr_tb; \ - struct nlattr *nla; \ - int err; \ - BUILD_BUG_ON(ARRAY_SIZE(s_name ## _nl_policy) > ARRAY_SIZE(nested_attr_tb)); \ - if (!tla) \ - return -ENOMSG; \ - DPRINT_TLA(#s_name, "<=-", #tag_name); \ - err = nla_parse_nested_deprecated(ntb, maxtype, tla, \ - s_name ## _nl_policy, NULL); \ - if (err) \ - return err; \ - \ - s_fields \ - return 0; \ -} __attribute__((unused)) \ -static int s_name ## _from_attrs(struct s_name *s, \ - struct genl_info *info) \ -{ \ - return __ ## s_name ## _from_attrs(s, info, false); \ -} __attribute__((unused)) \ -static int s_name ## _from_attrs_for_change(struct s_name *s, \ - struct genl_info *info) \ -{ \ - return __ ## s_name ## _from_attrs(s, info, true); \ -} __attribute__((unused)) \ - -#define __assign(attr_nr, attr_flag, name, nla_type, type, assignment...) \ - nla = ntb[attr_nr]; \ - if (nla) { \ - if (exclude_invariants && !!((attr_flag) & DRBD_F_INVARIANT)) { \ - pr_info("<< must not change invariant attr: %s\n", #name); \ - return -EEXIST; \ - } \ - assignment; \ - } else if (exclude_invariants && !!((attr_flag) & DRBD_F_INVARIANT)) { \ - /* attribute missing from payload, */ \ - /* which was expected */ \ - } else if ((attr_flag) & DRBD_F_REQUIRED) { \ - pr_info("<< missing attr: %s\n", #name); \ - return -ENOMSG; \ - } - -#undef __field -#define __field(attr_nr, attr_flag, name, nla_type, type, __get, __put, \ - __is_signed) \ - __assign(attr_nr, attr_flag, name, nla_type, type, \ - if (s) \ - s->name = __get(nla); \ - DPRINT_FIELD("<<", nla_type, name, s, nla)) - -/* validate_nla() already checked nla_len <= maxlen appropriately. */ -#undef __array -#define __array(attr_nr, attr_flag, name, nla_type, type, maxlen, \ - __get, __put, __is_signed) \ - __assign(attr_nr, attr_flag, name, nla_type, type, \ - if (s) \ - s->name ## _len = \ - __get(s->name, nla, maxlen); \ - DPRINT_ARRAY("<<", nla_type, name, s, nla)) - -#include GENL_MAGIC_INCLUDE_FILE - -#undef GENL_struct -#define GENL_struct(tag_name, tag_number, s_name, s_fields) - -/* - * Magic: define op number to op name mapping {{{1 - * {{{2 - */ -static const char *CONCATENATE(GENL_MAGIC_FAMILY, _genl_cmd_to_str)(__u8 cmd) -{ - switch (cmd) { -#undef GENL_op -#define GENL_op(op_name, op_num, handler, tla_list) \ - case op_num: return #op_name; -#include GENL_MAGIC_INCLUDE_FILE - default: - return "unknown"; - } -} - -#ifdef __KERNEL__ -#include <linux/stringify.h> -/* - * Magic: define genl_ops {{{1 - * {{{2 - */ - -#undef GENL_op -#define GENL_op(op_name, op_num, handler, tla_list) \ -{ \ - handler \ - .cmd = op_name, \ -}, - -#define ZZZ_genl_ops CONCATENATE(GENL_MAGIC_FAMILY, _genl_ops) -static struct genl_ops ZZZ_genl_ops[] __read_mostly = { -#include GENL_MAGIC_INCLUDE_FILE -}; - -#undef GENL_op -#define GENL_op(op_name, op_num, handler, tla_list) - -/* - * Define the genl_family, multicast groups, {{{1 - * and provide register/unregister functions. - * {{{2 - */ -#define ZZZ_genl_family CONCATENATE(GENL_MAGIC_FAMILY, _genl_family) -static struct genl_family ZZZ_genl_family; -/* - * Magic: define multicast groups - * Magic: define multicast group registration helper - */ -#define ZZZ_genl_mcgrps CONCATENATE(GENL_MAGIC_FAMILY, _genl_mcgrps) -static const struct genl_multicast_group ZZZ_genl_mcgrps[] = { -#undef GENL_mc_group -#define GENL_mc_group(group) { .name = #group, }, -#include GENL_MAGIC_INCLUDE_FILE -}; - -enum CONCATENATE(GENL_MAGIC_FAMILY, group_ids) { -#undef GENL_mc_group -#define GENL_mc_group(group) CONCATENATE(GENL_MAGIC_FAMILY, _group_ ## group), -#include GENL_MAGIC_INCLUDE_FILE -}; - -#undef GENL_mc_group -#define GENL_mc_group(group) \ -static int CONCATENATE(GENL_MAGIC_FAMILY, _genl_multicast_ ## group)( \ - struct sk_buff *skb, gfp_t flags) \ -{ \ - unsigned int group_id = \ - CONCATENATE(GENL_MAGIC_FAMILY, _group_ ## group); \ - return genlmsg_multicast(&ZZZ_genl_family, skb, 0, \ - group_id, flags); \ -} - -#include GENL_MAGIC_INCLUDE_FILE - -#undef GENL_mc_group -#define GENL_mc_group(group) - -static struct genl_family ZZZ_genl_family __ro_after_init = { - .name = __stringify(GENL_MAGIC_FAMILY), - .version = GENL_MAGIC_VERSION, -#ifdef GENL_MAGIC_FAMILY_HDRSZ - .hdrsize = NLA_ALIGN(GENL_MAGIC_FAMILY_HDRSZ), -#endif - .maxattr = ARRAY_SIZE(CONCATENATE(GENL_MAGIC_FAMILY, _tla_nl_policy))-1, - .policy = CONCATENATE(GENL_MAGIC_FAMILY, _tla_nl_policy), -#ifdef GENL_MAGIC_FAMILY_PRE_DOIT - .pre_doit = GENL_MAGIC_FAMILY_PRE_DOIT, - .post_doit = GENL_MAGIC_FAMILY_POST_DOIT, -#endif - .ops = ZZZ_genl_ops, - .n_ops = ARRAY_SIZE(ZZZ_genl_ops), - .mcgrps = ZZZ_genl_mcgrps, - .resv_start_op = 42, /* drbd is currently the only user */ - .n_mcgrps = ARRAY_SIZE(ZZZ_genl_mcgrps), - .module = THIS_MODULE, -}; - -int CONCATENATE(GENL_MAGIC_FAMILY, _genl_register)(void) -{ - return genl_register_family(&ZZZ_genl_family); -} - -void CONCATENATE(GENL_MAGIC_FAMILY, _genl_unregister)(void) -{ - genl_unregister_family(&ZZZ_genl_family); -} - -/* - * Magic: provide conversion functions {{{1 - * populate skb from struct. - * {{{2 - */ - -#undef GENL_op -#define GENL_op(op_name, op_num, handler, tla_list) - -#undef GENL_struct -#define GENL_struct(tag_name, tag_number, s_name, s_fields) \ -static int s_name ## _to_skb(struct sk_buff *skb, struct s_name *s, \ - const bool exclude_sensitive) \ -{ \ - struct nlattr *tla = nla_nest_start(skb, tag_number); \ - if (!tla) \ - goto nla_put_failure; \ - DPRINT_TLA(#s_name, "-=>", #tag_name); \ - s_fields \ - nla_nest_end(skb, tla); \ - return 0; \ - \ -nla_put_failure: \ - if (tla) \ - nla_nest_cancel(skb, tla); \ - return -EMSGSIZE; \ -} \ -static inline int s_name ## _to_priv_skb(struct sk_buff *skb, \ - struct s_name *s) \ -{ \ - return s_name ## _to_skb(skb, s, 0); \ -} \ -static inline int s_name ## _to_unpriv_skb(struct sk_buff *skb, \ - struct s_name *s) \ -{ \ - return s_name ## _to_skb(skb, s, 1); \ -} - - -#undef __field -#define __field(attr_nr, attr_flag, name, nla_type, type, __get, __put, \ - __is_signed) \ - if (!exclude_sensitive || !((attr_flag) & DRBD_F_SENSITIVE)) { \ - DPRINT_FIELD(">>", nla_type, name, s, NULL); \ - if (__put(skb, attr_nr, s->name)) \ - goto nla_put_failure; \ - } - -#undef __array -#define __array(attr_nr, attr_flag, name, nla_type, type, maxlen, \ - __get, __put, __is_signed) \ - if (!exclude_sensitive || !((attr_flag) & DRBD_F_SENSITIVE)) { \ - DPRINT_ARRAY(">>",nla_type, name, s, NULL); \ - if (__put(skb, attr_nr, min_t(int, maxlen, \ - s->name ## _len + (nla_type == NLA_NUL_STRING)),\ - s->name)) \ - goto nla_put_failure; \ - } - -#include GENL_MAGIC_INCLUDE_FILE - - -/* Functions for initializing structs to default values. */ - -#undef __field -#define __field(attr_nr, attr_flag, name, nla_type, type, __get, __put, \ - __is_signed) -#undef __array -#define __array(attr_nr, attr_flag, name, nla_type, type, maxlen, \ - __get, __put, __is_signed) -#undef __u32_field_def -#define __u32_field_def(attr_nr, attr_flag, name, default) \ - x->name = default; -#undef __s32_field_def -#define __s32_field_def(attr_nr, attr_flag, name, default) \ - x->name = default; -#undef __flg_field_def -#define __flg_field_def(attr_nr, attr_flag, name, default) \ - x->name = default; -#undef __str_field_def -#define __str_field_def(attr_nr, attr_flag, name, maxlen) \ - memset(x->name, 0, sizeof(x->name)); \ - x->name ## _len = 0; -#undef GENL_struct -#define GENL_struct(tag_name, tag_number, s_name, s_fields) \ -static void set_ ## s_name ## _defaults(struct s_name *x) __attribute__((unused)); \ -static void set_ ## s_name ## _defaults(struct s_name *x) { \ -s_fields \ -} - -#include GENL_MAGIC_INCLUDE_FILE - -#endif /* __KERNEL__ */ - -/* }}}1 */ -#endif /* GENL_MAGIC_FUNC_H */ diff --git a/include/linux/genl_magic_struct.h b/include/linux/genl_magic_struct.h deleted file mode 100644 index 2200cedd160a..000000000000 --- a/include/linux/genl_magic_struct.h +++ /dev/null @@ -1,272 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0 */ -#ifndef GENL_MAGIC_STRUCT_H -#define GENL_MAGIC_STRUCT_H - -#ifndef GENL_MAGIC_FAMILY -# error "you need to define GENL_MAGIC_FAMILY before inclusion" -#endif - -#ifndef GENL_MAGIC_VERSION -# error "you need to define GENL_MAGIC_VERSION before inclusion" -#endif - -#ifndef GENL_MAGIC_INCLUDE_FILE -# error "you need to define GENL_MAGIC_INCLUDE_FILE before inclusion" -#endif - -#include <linux/args.h> -#include <linux/types.h> -#include <net/genetlink.h> - -extern int CONCATENATE(GENL_MAGIC_FAMILY, _genl_register)(void); -extern void CONCATENATE(GENL_MAGIC_FAMILY, _genl_unregister)(void); - -/* - * Extension of genl attribute validation policies {{{2 - */ - -/* - * Flags specific to drbd and not visible at the netlink layer, used in - * <struct>_from_attrs and <struct>_to_skb: - * - * @DRBD_F_REQUIRED: Attribute is required; a request without this attribute is - * invalid. - * - * @DRBD_F_SENSITIVE: Attribute includes sensitive information and must not be - * included in unpriviledged get requests or broadcasts. - * - * @DRBD_F_INVARIANT: Attribute is set when an object is initially created, but - * cannot subsequently be changed. - */ -#define DRBD_F_REQUIRED (1 << 0) -#define DRBD_F_SENSITIVE (1 << 1) -#define DRBD_F_INVARIANT (1 << 2) - - -/* }}}1 - * MAGIC - * multi-include macro expansion magic starts here - */ - -/* MAGIC helpers {{{2 */ - -static inline int nla_put_u64_0pad(struct sk_buff *skb, int attrtype, u64 value) -{ - return nla_put_64bit(skb, attrtype, sizeof(u64), &value, 0); -} - -/* possible field types */ -#define __flg_field(attr_nr, attr_flag, name) \ - __field(attr_nr, attr_flag, name, NLA_U8, char, \ - nla_get_u8, nla_put_u8, false) -#define __u8_field(attr_nr, attr_flag, name) \ - __field(attr_nr, attr_flag, name, NLA_U8, unsigned char, \ - nla_get_u8, nla_put_u8, false) -#define __u16_field(attr_nr, attr_flag, name) \ - __field(attr_nr, attr_flag, name, NLA_U16, __u16, \ - nla_get_u16, nla_put_u16, false) -#define __u32_field(attr_nr, attr_flag, name) \ - __field(attr_nr, attr_flag, name, NLA_U32, __u32, \ - nla_get_u32, nla_put_u32, false) -#define __s32_field(attr_nr, attr_flag, name) \ - __field(attr_nr, attr_flag, name, NLA_U32, __s32, \ - nla_get_u32, nla_put_u32, true) -#define __u64_field(attr_nr, attr_flag, name) \ - __field(attr_nr, attr_flag, name, NLA_U64, __u64, \ - nla_get_u64, nla_put_u64_0pad, false) -#define __str_field(attr_nr, attr_flag, name, maxlen) \ - __array(attr_nr, attr_flag, name, NLA_NUL_STRING, char, maxlen, \ - nla_strscpy, nla_put, false) -#define __bin_field(attr_nr, attr_flag, name, maxlen) \ - __array(attr_nr, attr_flag, name, NLA_BINARY, char, maxlen, \ - nla_memcpy, nla_put, false) - -/* fields with default values */ -#define __flg_field_def(attr_nr, attr_flag, name, default) \ - __flg_field(attr_nr, attr_flag, name) -#define __u32_field_def(attr_nr, attr_flag, name, default) \ - __u32_field(attr_nr, attr_flag, name) -#define __s32_field_def(attr_nr, attr_flag, name, default) \ - __s32_field(attr_nr, attr_flag, name) -#define __str_field_def(attr_nr, attr_flag, name, maxlen) \ - __str_field(attr_nr, attr_flag, name, maxlen) - -#define GENL_op_init(args...) args -#define GENL_doit(handler) \ - .doit = handler, \ - .flags = GENL_ADMIN_PERM, -#define GENL_dumpit(handler) \ - .dumpit = handler, \ - .flags = GENL_ADMIN_PERM, - -/* }}}1 - * Magic: define the enum symbols for genl_ops - * Magic: define the enum symbols for top level attributes - * Magic: define the enum symbols for nested attributes - * {{{2 - */ - -#undef GENL_struct -#define GENL_struct(tag_name, tag_number, s_name, s_fields) - -#undef GENL_mc_group -#define GENL_mc_group(group) - -#undef GENL_notification -#define GENL_notification(op_name, op_num, mcast_group, tla_list) \ - op_name = op_num, - -#undef GENL_op -#define GENL_op(op_name, op_num, handler, tla_list) \ - op_name = op_num, - -enum { -#include GENL_MAGIC_INCLUDE_FILE -}; - -#undef GENL_notification -#define GENL_notification(op_name, op_num, mcast_group, tla_list) - -#undef GENL_op -#define GENL_op(op_name, op_num, handler, attr_list) - -#undef GENL_struct -#define GENL_struct(tag_name, tag_number, s_name, s_fields) \ - tag_name = tag_number, - -enum { -#include GENL_MAGIC_INCLUDE_FILE -}; - -#undef GENL_struct -#define GENL_struct(tag_name, tag_number, s_name, s_fields) \ -enum { \ - s_fields \ -}; - -#undef __field -#define __field(attr_nr, attr_flag, name, nla_type, type, \ - __get, __put, __is_signed) \ - T_ ## name = (__u16)(attr_nr), - -#undef __array -#define __array(attr_nr, attr_flag, name, nla_type, type, \ - maxlen, __get, __put, __is_signed) \ - T_ ## name = (__u16)(attr_nr), - -#include GENL_MAGIC_INCLUDE_FILE - -/* }}}1 - * Magic: compile time assert unique numbers for operations - * Magic: -"- unique numbers for top level attributes - * Magic: -"- unique numbers for nested attributes - * {{{2 - */ - -#undef GENL_struct -#define GENL_struct(tag_name, tag_number, s_name, s_fields) - -#undef GENL_op -#define GENL_op(op_name, op_num, handler, attr_list) \ - case op_name: - -#undef GENL_notification -#define GENL_notification(op_name, op_num, mcast_group, tla_list) \ - case op_name: - -static inline void ct_assert_unique_operations(void) -{ - switch (0) { -#include GENL_MAGIC_INCLUDE_FILE - case 0: - ; - } -} - -#undef GENL_op -#define GENL_op(op_name, op_num, handler, attr_list) - -#undef GENL_notification -#define GENL_notification(op_name, op_num, mcast_group, tla_list) - -#undef GENL_struct -#define GENL_struct(tag_name, tag_number, s_name, s_fields) \ - case tag_number: - -static inline void ct_assert_unique_top_level_attributes(void) -{ - switch (0) { -#include GENL_MAGIC_INCLUDE_FILE - case 0: - ; - } -} - -#undef GENL_struct -#define GENL_struct(tag_name, tag_number, s_name, s_fields) \ -static inline void ct_assert_unique_ ## s_name ## _attributes(void) \ -{ \ - switch (0) { \ - s_fields \ - case 0: \ - ; \ - } \ -} - -#undef __field -#define __field(attr_nr, attr_flag, name, nla_type, type, __get, __put, \ - __is_signed) \ - case attr_nr: - -#undef __array -#define __array(attr_nr, attr_flag, name, nla_type, type, maxlen, \ - __get, __put, __is_signed) \ - case attr_nr: - -#include GENL_MAGIC_INCLUDE_FILE - -/* }}}1 - * Magic: declare structs - * struct <name> { - * fields - * }; - * {{{2 - */ - -#undef GENL_struct -#define GENL_struct(tag_name, tag_number, s_name, s_fields) \ -struct s_name { s_fields }; - -#undef __field -#define __field(attr_nr, attr_flag, name, nla_type, type, __get, __put, \ - __is_signed) \ - type name; - -#undef __array -#define __array(attr_nr, attr_flag, name, nla_type, type, maxlen, \ - __get, __put, __is_signed) \ - type name[maxlen]; \ - __u32 name ## _len; - -#include GENL_MAGIC_INCLUDE_FILE - -#undef GENL_struct -#define GENL_struct(tag_name, tag_number, s_name, s_fields) \ -enum { \ - s_fields \ -}; - -#undef __field -#define __field(attr_nr, attr_flag, name, nla_type, type, __get, __put, \ - is_signed) \ - F_ ## name ## _IS_SIGNED = is_signed, - -#undef __array -#define __array(attr_nr, attr_flag, name, nla_type, type, maxlen, \ - __get, __put, is_signed) \ - F_ ## name ## _IS_SIGNED = is_signed, - -#include GENL_MAGIC_INCLUDE_FILE - -/* }}}1 */ -#endif /* GENL_MAGIC_STRUCT_H */ diff --git a/include/trace/events/block.h b/include/trace/events/block.h index 6aa79e2d799c..9c97a16850b9 100644 --- a/include/trace/events/block.h +++ b/include/trace/events/block.h @@ -227,6 +227,65 @@ DECLARE_EVENT_CLASS(block_rq, ); /** + * block_rq_tag_wait - triggered when a request is starved of a tag + * @q: request queue of the target device + * @hctx: hardware context of the request experiencing starvation + * @is_sched_tag: indicates whether the starved pool is the software scheduler + * @alloc_flags: allocation flags dictating the specific tag pool + * + * Called immediately before the submitting context is forced to block due + * to the exhaustion of available tags (i.e., physical hardware driver + * tags, software scheduler tags, or reserved tags). This trace point + * indicates that the context will be placed into an uninterruptible state + * via sbitmap_prepare_to_wait(). If a tag is not acquired in the final + * lockless retry, the context will yield the CPU via io_schedule() until + * an active request completes and relinquishes its assigned tag. + */ +TRACE_EVENT(block_rq_tag_wait, + + TP_PROTO(struct request_queue *q, struct blk_mq_hw_ctx *hctx, + bool is_sched_tag, unsigned int alloc_flags), + + TP_ARGS(q, hctx, is_sched_tag, alloc_flags), + + TP_STRUCT__entry( + __field( dev_t, dev ) + __field( u32, hctx_id ) + __field( u32, nr_tags ) + __field( bool, is_sched_tag ) + __field( bool, is_reserved ) + ), + + TP_fast_assign( + __entry->dev = q->disk ? disk_devt(q->disk) : 0; + __entry->hctx_id = hctx->queue_num; + __entry->is_sched_tag = is_sched_tag; + __entry->is_reserved = alloc_flags & BLK_MQ_REQ_RESERVED; + + if (__entry->is_reserved) { + __entry->nr_tags = is_sched_tag ? + hctx->sched_tags->nr_reserved_tags : + hctx->tags->nr_reserved_tags; + } else { + if (is_sched_tag) + __entry->nr_tags = hctx->sched_tags->nr_tags - + hctx->sched_tags->nr_reserved_tags; + else + __entry->nr_tags = hctx->tags->nr_tags - + hctx->tags->nr_reserved_tags; + } + + ), + + TP_printk("%d,%d hctx=%u starved on %s%s tags (depth=%u)", + MAJOR(__entry->dev), MINOR(__entry->dev), + __entry->hctx_id, + __entry->is_sched_tag ? "scheduler" : "hardware", + __entry->is_reserved ? " reserved" : "", + __entry->nr_tags) +); + +/** * block_rq_insert - insert block operation request into queue * @rq: block IO operation request * diff --git a/include/linux/drbd.h b/include/uapi/linux/drbd.h index 5468a2399d48..cf1ec3eb872f 100644 --- a/include/linux/drbd.h +++ b/include/uapi/linux/drbd.h @@ -1,4 +1,4 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* SPDX-License-Identifier: GPL-2.0-or-later WITH Linux-syscall-note */ /* drbd.h Kernel module for 2.6.x Kernels @@ -11,32 +11,10 @@ */ -#ifndef DRBD_H -#define DRBD_H -#include <asm/types.h> - -#ifdef __KERNEL__ +#ifndef _UAPI_LINUX_DRBD_H +#define _UAPI_LINUX_DRBD_H #include <linux/types.h> #include <asm/byteorder.h> -#else -#include <sys/types.h> -#include <sys/wait.h> -#include <limits.h> - -/* Although the Linux source code makes a difference between - generic endianness and the bitfields' endianness, there is no - architecture as of Linux-2.6.24-rc4 where the bitfields' endianness - does not match the generic endianness. */ - -#if __BYTE_ORDER == __LITTLE_ENDIAN -#define __LITTLE_ENDIAN_BITFIELD -#elif __BYTE_ORDER == __BIG_ENDIAN -#define __BIG_ENDIAN_BITFIELD -#else -# error "sorry, weird endianness on this box" -#endif - -#endif enum drbd_io_error_p { EP_PASS_ON, /* FIXME should the better be named "Ignore"? */ @@ -333,6 +311,9 @@ enum drbd_uuid_index { #define HISTORY_UUIDS MAX_PEERS +#define DRBD_NL_UUIDS_SIZE (UI_SIZE * sizeof(__u64)) +#define DRBD_NL_HISTORY_UUIDS_SIZE (HISTORY_UUIDS * sizeof(__u64)) + enum drbd_timeout_flag { UT_DEFAULT = 0, UT_DEGRADED = 1, @@ -389,4 +370,44 @@ enum write_ordering_e { #define DRBD_CPU_MASK_SIZE 32 -#endif +/** + * struct drbd_genlmsghdr - DRBD specific header used in NETLINK_GENERIC requests + * @minor: + * For admin requests (user -> kernel): which minor device to operate on. + * For (unicast) replies or informational (broadcast) messages + * (kernel -> user): which minor device the information is about. + * If we do not operate on minors, but on connections or resources, + * the minor value shall be (~0), and the attribute DRBD_NLA_CFG_CONTEXT + * is used instead. + * @flags: possible operation modifiers (relevant only for user->kernel): + * DRBD_GENL_F_SET_DEFAULTS + * @volume: + * When creating a new minor (adding it to a resource), the resource needs + * to know which volume number within the resource this is supposed to be. + * The volume number corresponds to the same volume number on the remote side, + * whereas the minor number on the remote side may be different + * (union with flags). + * @ret_code: kernel->userland unicast cfg reply return code (union with flags); + */ +struct drbd_genlmsghdr { + __u32 minor; + union { + __u32 flags; + __s32 ret_code; + }; +}; + +/* To be used in drbd_genlmsghdr.flags */ +enum { + DRBD_GENL_F_SET_DEFAULTS = 1, +}; + +enum drbd_state_info_bcast_reason { + SIB_GET_STATUS_REPLY = 1, + SIB_STATE_CHANGE = 2, + SIB_HELPER_PRE = 3, + SIB_HELPER_POST = 4, + SIB_SYNC_PROGRESS = 5, +}; + +#endif /* _UAPI_LINUX_DRBD_H */ diff --git a/include/uapi/linux/drbd_genl.h b/include/uapi/linux/drbd_genl.h new file mode 100644 index 000000000000..8b25f08d8a90 --- /dev/null +++ b/include/uapi/linux/drbd_genl.h @@ -0,0 +1,359 @@ +/* SPDX-License-Identifier: ((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) */ + +#ifndef _UAPI_LINUX_DRBD_GENL_H +#define _UAPI_LINUX_DRBD_GENL_H + +#define DRBD_FAMILY_NAME "drbd" +#define DRBD_FAMILY_VERSION 1 + +enum { + DRBD_NLA_CFG_REPLY = 1, + DRBD_NLA_CFG_CONTEXT, + DRBD_NLA_DISK_CONF, + DRBD_NLA_RESOURCE_OPTS, + DRBD_NLA_NET_CONF, + DRBD_NLA_SET_ROLE_PARMS, + DRBD_NLA_RESIZE_PARMS, + DRBD_NLA_STATE_INFO, + DRBD_NLA_START_OV_PARMS, + DRBD_NLA_NEW_C_UUID_PARMS, + DRBD_NLA_TIMEOUT_PARMS, + DRBD_NLA_DISCONNECT_PARMS, + DRBD_NLA_DETACH_PARMS, + DRBD_NLA_RESOURCE_INFO = 15, + DRBD_NLA_DEVICE_INFO, + DRBD_NLA_CONNECTION_INFO, + DRBD_NLA_PEER_DEVICE_INFO, + DRBD_NLA_RESOURCE_STATISTICS, + DRBD_NLA_DEVICE_STATISTICS, + DRBD_NLA_CONNECTION_STATISTICS, + DRBD_NLA_PEER_DEVICE_STATISTICS, + DRBD_NLA_NOTIFICATION_HEADER, + DRBD_NLA_HELPER, + + __DRBD_NLA_MAX, + DRBD_NLA_MAX = (__DRBD_NLA_MAX - 1) +}; + +enum { + DRBD_A_DRBD_CFG_REPLY_INFO_TEXT = 1, + + __DRBD_A_DRBD_CFG_REPLY_MAX, + DRBD_A_DRBD_CFG_REPLY_MAX = (__DRBD_A_DRBD_CFG_REPLY_MAX - 1) +}; + +enum { + DRBD_A_DRBD_CFG_CONTEXT_CTX_VOLUME = 1, + DRBD_A_DRBD_CFG_CONTEXT_CTX_RESOURCE_NAME, + DRBD_A_DRBD_CFG_CONTEXT_CTX_MY_ADDR, + DRBD_A_DRBD_CFG_CONTEXT_CTX_PEER_ADDR, + + __DRBD_A_DRBD_CFG_CONTEXT_MAX, + DRBD_A_DRBD_CFG_CONTEXT_MAX = (__DRBD_A_DRBD_CFG_CONTEXT_MAX - 1) +}; + +enum { + DRBD_A_DISK_CONF_BACKING_DEV = 1, + DRBD_A_DISK_CONF_META_DEV, + DRBD_A_DISK_CONF_META_DEV_IDX, + DRBD_A_DISK_CONF_DISK_SIZE, + DRBD_A_DISK_CONF_MAX_BIO_BVECS, + DRBD_A_DISK_CONF_ON_IO_ERROR, + DRBD_A_DISK_CONF_FENCING, + DRBD_A_DISK_CONF_RESYNC_RATE, + DRBD_A_DISK_CONF_RESYNC_AFTER, + DRBD_A_DISK_CONF_AL_EXTENTS, + DRBD_A_DISK_CONF_C_PLAN_AHEAD, + DRBD_A_DISK_CONF_C_DELAY_TARGET, + DRBD_A_DISK_CONF_C_FILL_TARGET, + DRBD_A_DISK_CONF_C_MAX_RATE, + DRBD_A_DISK_CONF_C_MIN_RATE, + DRBD_A_DISK_CONF_DISK_BARRIER, + DRBD_A_DISK_CONF_DISK_FLUSHES, + DRBD_A_DISK_CONF_DISK_DRAIN, + DRBD_A_DISK_CONF_MD_FLUSHES, + DRBD_A_DISK_CONF_DISK_TIMEOUT, + DRBD_A_DISK_CONF_READ_BALANCING, + DRBD_A_DISK_CONF_AL_UPDATES = 23, + DRBD_A_DISK_CONF_DISCARD_ZEROES_IF_ALIGNED, + DRBD_A_DISK_CONF_RS_DISCARD_GRANULARITY, + DRBD_A_DISK_CONF_DISABLE_WRITE_SAME, + + __DRBD_A_DISK_CONF_MAX, + DRBD_A_DISK_CONF_MAX = (__DRBD_A_DISK_CONF_MAX - 1) +}; + +enum { + DRBD_A_RES_OPTS_CPU_MASK = 1, + DRBD_A_RES_OPTS_ON_NO_DATA, + + __DRBD_A_RES_OPTS_MAX, + DRBD_A_RES_OPTS_MAX = (__DRBD_A_RES_OPTS_MAX - 1) +}; + +enum { + DRBD_A_NET_CONF_SHARED_SECRET = 1, + DRBD_A_NET_CONF_CRAM_HMAC_ALG, + DRBD_A_NET_CONF_INTEGRITY_ALG, + DRBD_A_NET_CONF_VERIFY_ALG, + DRBD_A_NET_CONF_CSUMS_ALG, + DRBD_A_NET_CONF_WIRE_PROTOCOL, + DRBD_A_NET_CONF_CONNECT_INT, + DRBD_A_NET_CONF_TIMEOUT, + DRBD_A_NET_CONF_PING_INT, + DRBD_A_NET_CONF_PING_TIMEO, + DRBD_A_NET_CONF_SNDBUF_SIZE, + DRBD_A_NET_CONF_RCVBUF_SIZE, + DRBD_A_NET_CONF_KO_COUNT, + DRBD_A_NET_CONF_MAX_BUFFERS, + DRBD_A_NET_CONF_MAX_EPOCH_SIZE, + DRBD_A_NET_CONF_UNPLUG_WATERMARK, + DRBD_A_NET_CONF_AFTER_SB_0P, + DRBD_A_NET_CONF_AFTER_SB_1P, + DRBD_A_NET_CONF_AFTER_SB_2P, + DRBD_A_NET_CONF_RR_CONFLICT, + DRBD_A_NET_CONF_ON_CONGESTION, + DRBD_A_NET_CONF_CONG_FILL, + DRBD_A_NET_CONF_CONG_EXTENTS, + DRBD_A_NET_CONF_TWO_PRIMARIES, + DRBD_A_NET_CONF_DISCARD_MY_DATA, + DRBD_A_NET_CONF_TCP_CORK, + DRBD_A_NET_CONF_ALWAYS_ASBP, + DRBD_A_NET_CONF_TENTATIVE, + DRBD_A_NET_CONF_USE_RLE, + DRBD_A_NET_CONF_CSUMS_AFTER_CRASH_ONLY = 33, + DRBD_A_NET_CONF_SOCK_CHECK_TIMEO, + + __DRBD_A_NET_CONF_MAX, + DRBD_A_NET_CONF_MAX = (__DRBD_A_NET_CONF_MAX - 1) +}; + +enum { + DRBD_A_SET_ROLE_PARMS_ASSUME_UPTODATE = 1, + + __DRBD_A_SET_ROLE_PARMS_MAX, + DRBD_A_SET_ROLE_PARMS_MAX = (__DRBD_A_SET_ROLE_PARMS_MAX - 1) +}; + +enum { + DRBD_A_RESIZE_PARMS_RESIZE_SIZE = 1, + DRBD_A_RESIZE_PARMS_RESIZE_FORCE, + DRBD_A_RESIZE_PARMS_NO_RESYNC, + DRBD_A_RESIZE_PARMS_AL_STRIPES, + DRBD_A_RESIZE_PARMS_AL_STRIPE_SIZE, + + __DRBD_A_RESIZE_PARMS_MAX, + DRBD_A_RESIZE_PARMS_MAX = (__DRBD_A_RESIZE_PARMS_MAX - 1) +}; + +enum { + DRBD_A_STATE_INFO_SIB_REASON = 1, + DRBD_A_STATE_INFO_CURRENT_STATE, + DRBD_A_STATE_INFO_CAPACITY, + DRBD_A_STATE_INFO_ED_UUID, + DRBD_A_STATE_INFO_PREV_STATE, + DRBD_A_STATE_INFO_NEW_STATE, + DRBD_A_STATE_INFO_UUIDS, + DRBD_A_STATE_INFO_DISK_FLAGS, + DRBD_A_STATE_INFO_BITS_TOTAL, + DRBD_A_STATE_INFO_BITS_OOS, + DRBD_A_STATE_INFO_BITS_RS_TOTAL, + DRBD_A_STATE_INFO_BITS_RS_FAILED, + DRBD_A_STATE_INFO_HELPER, + DRBD_A_STATE_INFO_HELPER_EXIT_CODE, + DRBD_A_STATE_INFO_SEND_CNT, + DRBD_A_STATE_INFO_RECV_CNT, + DRBD_A_STATE_INFO_READ_CNT, + DRBD_A_STATE_INFO_WRIT_CNT, + DRBD_A_STATE_INFO_AL_WRIT_CNT, + DRBD_A_STATE_INFO_BM_WRIT_CNT, + DRBD_A_STATE_INFO_AP_BIO_CNT, + DRBD_A_STATE_INFO_AP_PENDING_CNT, + DRBD_A_STATE_INFO_RS_PENDING_CNT, + + __DRBD_A_STATE_INFO_MAX, + DRBD_A_STATE_INFO_MAX = (__DRBD_A_STATE_INFO_MAX - 1) +}; + +enum { + DRBD_A_START_OV_PARMS_OV_START_SECTOR = 1, + DRBD_A_START_OV_PARMS_OV_STOP_SECTOR, + + __DRBD_A_START_OV_PARMS_MAX, + DRBD_A_START_OV_PARMS_MAX = (__DRBD_A_START_OV_PARMS_MAX - 1) +}; + +enum { + DRBD_A_NEW_C_UUID_PARMS_CLEAR_BM = 1, + + __DRBD_A_NEW_C_UUID_PARMS_MAX, + DRBD_A_NEW_C_UUID_PARMS_MAX = (__DRBD_A_NEW_C_UUID_PARMS_MAX - 1) +}; + +enum { + DRBD_A_TIMEOUT_PARMS_TIMEOUT_TYPE = 1, + + __DRBD_A_TIMEOUT_PARMS_MAX, + DRBD_A_TIMEOUT_PARMS_MAX = (__DRBD_A_TIMEOUT_PARMS_MAX - 1) +}; + +enum { + DRBD_A_DISCONNECT_PARMS_FORCE_DISCONNECT = 1, + + __DRBD_A_DISCONNECT_PARMS_MAX, + DRBD_A_DISCONNECT_PARMS_MAX = (__DRBD_A_DISCONNECT_PARMS_MAX - 1) +}; + +enum { + DRBD_A_DETACH_PARMS_FORCE_DETACH = 1, + + __DRBD_A_DETACH_PARMS_MAX, + DRBD_A_DETACH_PARMS_MAX = (__DRBD_A_DETACH_PARMS_MAX - 1) +}; + +enum { + DRBD_A_RESOURCE_INFO_RES_ROLE = 1, + DRBD_A_RESOURCE_INFO_RES_SUSP, + DRBD_A_RESOURCE_INFO_RES_SUSP_NOD, + DRBD_A_RESOURCE_INFO_RES_SUSP_FEN, + + __DRBD_A_RESOURCE_INFO_MAX, + DRBD_A_RESOURCE_INFO_MAX = (__DRBD_A_RESOURCE_INFO_MAX - 1) +}; + +enum { + DRBD_A_DEVICE_INFO_DEV_DISK_STATE = 1, + + __DRBD_A_DEVICE_INFO_MAX, + DRBD_A_DEVICE_INFO_MAX = (__DRBD_A_DEVICE_INFO_MAX - 1) +}; + +enum { + DRBD_A_CONNECTION_INFO_CONN_CONNECTION_STATE = 1, + DRBD_A_CONNECTION_INFO_CONN_ROLE, + + __DRBD_A_CONNECTION_INFO_MAX, + DRBD_A_CONNECTION_INFO_MAX = (__DRBD_A_CONNECTION_INFO_MAX - 1) +}; + +enum { + DRBD_A_PEER_DEVICE_INFO_PEER_REPL_STATE = 1, + DRBD_A_PEER_DEVICE_INFO_PEER_DISK_STATE, + DRBD_A_PEER_DEVICE_INFO_PEER_RESYNC_SUSP_USER, + DRBD_A_PEER_DEVICE_INFO_PEER_RESYNC_SUSP_PEER, + DRBD_A_PEER_DEVICE_INFO_PEER_RESYNC_SUSP_DEPENDENCY, + + __DRBD_A_PEER_DEVICE_INFO_MAX, + DRBD_A_PEER_DEVICE_INFO_MAX = (__DRBD_A_PEER_DEVICE_INFO_MAX - 1) +}; + +enum { + DRBD_A_RESOURCE_STATISTICS_RES_STAT_WRITE_ORDERING = 1, + + __DRBD_A_RESOURCE_STATISTICS_MAX, + DRBD_A_RESOURCE_STATISTICS_MAX = (__DRBD_A_RESOURCE_STATISTICS_MAX - 1) +}; + +enum { + DRBD_A_DEVICE_STATISTICS_DEV_SIZE = 1, + DRBD_A_DEVICE_STATISTICS_DEV_READ, + DRBD_A_DEVICE_STATISTICS_DEV_WRITE, + DRBD_A_DEVICE_STATISTICS_DEV_AL_WRITES, + DRBD_A_DEVICE_STATISTICS_DEV_BM_WRITES, + DRBD_A_DEVICE_STATISTICS_DEV_UPPER_PENDING, + DRBD_A_DEVICE_STATISTICS_DEV_LOWER_PENDING, + DRBD_A_DEVICE_STATISTICS_DEV_UPPER_BLOCKED, + DRBD_A_DEVICE_STATISTICS_DEV_LOWER_BLOCKED, + DRBD_A_DEVICE_STATISTICS_DEV_AL_SUSPENDED, + DRBD_A_DEVICE_STATISTICS_DEV_EXPOSED_DATA_UUID, + DRBD_A_DEVICE_STATISTICS_DEV_CURRENT_UUID, + DRBD_A_DEVICE_STATISTICS_DEV_DISK_FLAGS, + DRBD_A_DEVICE_STATISTICS_HISTORY_UUIDS, + + __DRBD_A_DEVICE_STATISTICS_MAX, + DRBD_A_DEVICE_STATISTICS_MAX = (__DRBD_A_DEVICE_STATISTICS_MAX - 1) +}; + +enum { + DRBD_A_CONNECTION_STATISTICS_CONN_CONGESTED = 1, + + __DRBD_A_CONNECTION_STATISTICS_MAX, + DRBD_A_CONNECTION_STATISTICS_MAX = (__DRBD_A_CONNECTION_STATISTICS_MAX - 1) +}; + +enum { + DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_RECEIVED = 1, + DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_SENT, + DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_PENDING, + DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_UNACKED, + DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_OUT_OF_SYNC, + DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_RESYNC_FAILED, + DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_BITMAP_UUID, + DRBD_A_PEER_DEVICE_STATISTICS_PEER_DEV_FLAGS = 9, + + __DRBD_A_PEER_DEVICE_STATISTICS_MAX, + DRBD_A_PEER_DEVICE_STATISTICS_MAX = (__DRBD_A_PEER_DEVICE_STATISTICS_MAX - 1) +}; + +enum { + DRBD_A_DRBD_NOTIFICATION_HEADER_NH_TYPE = 1, + + __DRBD_A_DRBD_NOTIFICATION_HEADER_MAX, + DRBD_A_DRBD_NOTIFICATION_HEADER_MAX = (__DRBD_A_DRBD_NOTIFICATION_HEADER_MAX - 1) +}; + +enum { + DRBD_A_DRBD_HELPER_INFO_HELPER_NAME = 1, + DRBD_A_DRBD_HELPER_INFO_HELPER_STATUS, + + __DRBD_A_DRBD_HELPER_INFO_MAX, + DRBD_A_DRBD_HELPER_INFO_MAX = (__DRBD_A_DRBD_HELPER_INFO_MAX - 1) +}; + +enum { + DRBD_ADM_EVENT = 1, + DRBD_ADM_GET_STATUS, + DRBD_ADM_NEW_MINOR = 5, + DRBD_ADM_DEL_MINOR, + DRBD_ADM_NEW_RESOURCE, + DRBD_ADM_DEL_RESOURCE, + DRBD_ADM_RESOURCE_OPTS, + DRBD_ADM_CONNECT, + DRBD_ADM_DISCONNECT, + DRBD_ADM_ATTACH, + DRBD_ADM_RESIZE, + DRBD_ADM_PRIMARY, + DRBD_ADM_SECONDARY, + DRBD_ADM_NEW_C_UUID, + DRBD_ADM_START_OV, + DRBD_ADM_DETACH, + DRBD_ADM_INVALIDATE, + DRBD_ADM_INVAL_PEER, + DRBD_ADM_PAUSE_SYNC, + DRBD_ADM_RESUME_SYNC, + DRBD_ADM_SUSPEND_IO, + DRBD_ADM_RESUME_IO, + DRBD_ADM_OUTDATE, + DRBD_ADM_GET_TIMEOUT_TYPE, + DRBD_ADM_DOWN, + DRBD_ADM_CHG_DISK_OPTS, + DRBD_ADM_CHG_NET_OPTS, + DRBD_ADM_GET_RESOURCES, + DRBD_ADM_GET_DEVICES, + DRBD_ADM_GET_CONNECTIONS, + DRBD_ADM_GET_PEER_DEVICES, + DRBD_ADM_RESOURCE_STATE, + DRBD_ADM_DEVICE_STATE, + DRBD_ADM_CONNECTION_STATE, + DRBD_ADM_PEER_DEVICE_STATE, + DRBD_ADM_GET_INITIAL_STATE, + DRBD_ADM_HELPER = 40, + DRBD_ADM_INITIAL_STATE_DONE, + + __DRBD_ADM_MAX, + DRBD_ADM_MAX = (__DRBD_ADM_MAX - 1) +}; + +#define DRBD_MCGRP_EVENTS "events" + +#endif /* _UAPI_LINUX_DRBD_GENL_H */ diff --git a/include/linux/drbd_limits.h b/include/uapi/linux/drbd_limits.h index 5b042fb427e9..acefe84bc602 100644 --- a/include/linux/drbd_limits.h +++ b/include/uapi/linux/drbd_limits.h @@ -1,4 +1,4 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ +/* SPDX-License-Identifier: GPL-2.0-only WITH Linux-syscall-note */ /* drbd_limits.h This file is part of DRBD by Philipp Reisner and Lars Ellenberg. @@ -11,10 +11,10 @@ * feedback about nonsense settings for certain configurable values. */ -#ifndef DRBD_LIMITS_H -#define DRBD_LIMITS_H 1 +#ifndef _UAPI_LINUX_DRBD_LIMITS_H +#define _UAPI_LINUX_DRBD_LIMITS_H -#define DEBUG_RANGE_CHECK 0 +#include <linux/drbd.h> #define DRBD_MINOR_COUNT_MIN 1U #define DRBD_MINOR_COUNT_MAX 255U @@ -248,4 +248,4 @@ #define DRBD_RS_DISCARD_GRANULARITY_DEF 0U /* disabled by default */ #define DRBD_RS_DISCARD_GRANULARITY_SCALE '1' /* bytes */ -#endif +#endif /* _UAPI_LINUX_DRBD_LIMITS_H */ diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 7f553d115e36..8d0f2ee24e0c 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -152,7 +152,7 @@ static void io_release_ubuf(void *priv) unsigned int i; for (i = 0; i < imu->nr_bvecs; i++) { - struct folio *folio = page_folio(imu->bvec[i].bv_page); + struct folio *folio = bvec_folio(&imu->bvec[i]); unpin_user_folio(folio, 1); } diff --git a/mm/page_io.c b/mm/page_io.c index 70cea9e24d2f..a59b73f8bdd9 100644 --- a/mm/page_io.c +++ b/mm/page_io.c @@ -490,7 +490,7 @@ static void sio_read_complete(struct kiocb *iocb, long ret) if (ret == sio->len) { for (p = 0; p < sio->pages; p++) { - struct folio *folio = page_folio(sio->bvec[p].bv_page); + struct folio *folio = bvec_folio(&sio->bvec[p]); count_mthp_stat(folio_order(folio), MTHP_STAT_SWPIN); count_memcg_folio_events(folio, PSWPIN, folio_nr_pages(folio)); @@ -500,7 +500,7 @@ static void sio_read_complete(struct kiocb *iocb, long ret) count_vm_events(PSWPIN, sio->len >> PAGE_SHIFT); } else { for (p = 0; p < sio->pages; p++) { - struct folio *folio = page_folio(sio->bvec[p].bv_page); + struct folio *folio = bvec_folio(&sio->bvec[p]); folio_unlock(folio); } diff --git a/rust/kernel/block/mq/gen_disk.rs b/rust/kernel/block/mq/gen_disk.rs index 912cb805caf5..fc97dd873974 100644 --- a/rust/kernel/block/mq/gen_disk.rs +++ b/rust/kernel/block/mq/gen_disk.rs @@ -150,6 +150,19 @@ impl GenDiskBuilder { // SAFETY: `gendisk` is a valid pointer as we initialized it above unsafe { (*gendisk).fops = &TABLE }; + let cleanup_failure = ScopeGuard::new_with_data((gendisk, data), |(gendisk, data)| { + // SAFETY: `gendisk` came from `__blk_mq_alloc_disk()` above and + // has not been added to the VFS on this cleanup path. + unsafe { bindings::put_disk(gendisk) }; + // SAFETY: `data` came from `into_foreign()` above and has not been + // converted back on this cleanup path. + drop(unsafe { T::QueueData::from_foreign(data) }); + }); + + // The failure guard now owns both pieces of cleanup; the early guard + // must not run on this path anymore. + recover_data.dismiss(); + let mut writer = NullTerminatedFormatter::new( // SAFETY: `gendisk` points to a valid and initialized instance. We // have exclusive access, since the disk is not added to the VFS @@ -172,7 +185,7 @@ impl GenDiskBuilder { }, )?; - recover_data.dismiss(); + cleanup_failure.dismiss(); // INVARIANT: `gendisk` was initialized above. // INVARIANT: `gendisk` was added to the VFS via `device_add_disk` above. @@ -215,6 +228,11 @@ impl<T: Operations> Drop for GenDisk<T> { // to the VFS. unsafe { bindings::del_gendisk(self.gendisk) }; + // SAFETY: By type invariant, `self.gendisk` was added to the VFS, so + // `put_disk()` must follow `del_gendisk()` to drop the final gendisk + // reference and trigger the remaining release path. + unsafe { bindings::put_disk(self.gendisk) }; + // SAFETY: `queue.queuedata` was created by `GenDiskBuilder::build` with // a call to `ForeignOwnable::into_foreign` to create `queuedata`. // `ForeignOwnable::from_foreign` is only called here. diff --git a/rust/kernel/block/mq/operations.rs b/rust/kernel/block/mq/operations.rs index 8ad46129a52c..861903e18fbf 100644 --- a/rust/kernel/block/mq/operations.rs +++ b/rust/kernel/block/mq/operations.rs @@ -218,7 +218,7 @@ impl<T: Operations> OperationsVTable<T> { _set: *mut bindings::blk_mq_tag_set, rq: *mut bindings::request, _hctx_idx: crate::ffi::c_uint, - _numa_node: crate::ffi::c_uint, + _numa_node: crate::ffi::c_int, ) -> crate::ffi::c_int { from_result(|| { // SAFETY: By the safety requirements of this function, `rq` points |