summaryrefslogtreecommitdiff
path: root/rust
diff options
context:
space:
mode:
authorLinus Torvalds <torvalds@linux-foundation.org>2026-08-19 10:42:18 -0700
committerLinus Torvalds <torvalds@linux-foundation.org>2026-08-19 10:42:18 -0700
commit59e6295fac26b8e85c1ea859cdd89fa1e47519d7 (patch)
tree7c071f8bab440eddbb569e84e9f31c80f8a4e529 /rust
parentc6cf4441a3a05bb7273ed022f3e56c4fc591da08 (diff)
parent3b5ea0f078e1b72276e99c237f3dfc2fd72938bb (diff)
Merge tag 'driver-core-7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/driver-core/driver-core
Pull driver core updates from Danilo Krummrich: "container_of: - Apply typeof_member(), remove the local __mptr variable to eliminate variable shadowing warnings on nested container_of() calls, and remove unnecessary parentheses core: - Add driver name to probe debug print for initcall_debug - Avoid repeatedly printing the same 'Fixed dependency cycle' log - Unwind device_add() on attribute creation failure in attribute_container_add_class_device() - Remove statistics group if encryption group creation fails in transport_add_class_device() debugfs: - Fix lockdown check for mmap_prepare() - Warn if file creation failed due to uninitialized debugfs device property: - Implement fw_devlink support for software nodes by adding software_node_add_links(), which creates fwnode links from DEV_PROP_REF properties to enable automatic probe ordering. Add kunit-managed fwnode helpers and test coverage - Fix infinite loop in fwnode_for_each_child_node() when the secondary fwnode has more than one child. Add test cases - Fix out-of-bounds access in software_node_get_reference_args() when called with index -1 (UINT_MAX) - Refactor to use RAII approach with __free() - Add Bartosz Golaszewski as software node reviewer firmware loader: - Fix race where a sysfs fallback request can complete before being queued as pending, leading to a use-after-free on the next fallback request - Reject 0-size built-in firmware and fail the build on empty firmware files in CONFIG_EXTRA_FIRMWARE kobject: - Provide __KOBJ_ATTR() and __KOBJ_ATTR_RO/WO() initialization macros and allow the constification of kobject attributes, enabling them to reside in read-only memory platform: - Provide platform_device_set_of_node(), platform_device_set_fwnode(), and platform_device_set_of_node_from_dev() helpers that encapsulate firmware node reference counting for dynamically allocated platform devices Convert all in-tree users that manually assigned dev.of_node or dev.fwnode, fixing a pre-existing refcount bug in powermac. Switch to counting references of all firmware node types, not only OF nodes - Unify the release path for dynamically allocated platform devices by removing platform_device_release_full(). Amend the fwnode setter API contract to warn if a primary software node is overwritten. Add KUnit tests for correct software node removal on device unregistration Rust: - Auxiliary: - Add registration_data_with() closure-based API for invariant ForLt types - Debugfs: - Migrate BinaryWriter and BinaryReaderMut trait requirements from kernel::transmute traits to zerocopy traits - Device: - Add BoundInternal device context and InternalBoundContext trait for bus abstractions that need internal access to a bound device. - Make the lifetime on Core and CoreInternal invariant to prevent coercion to shorter lifetimes - Devres: - Fix race between concurrent revokers where the losing revoker could return before the winning revoker finished dropping the inner data, causing use-after-free. - Ensure revocation is complete before the device finishes unbinding by making the synchronization bidirectional. - Add DevresLt<F: ForLt>, a wrapper around Devres that shortens 'static back to the caller's borrow scope. Implement ForLt and CovariantForLt for Bar, IoMem, and ExclusiveIoMem - Driver: - Switch from index-based to pointer-based device ID info lookup, storing static references in driver_data. Centralize device ID handling in device_id.rs, removing the open-coded ACPI/OF matching logic and duplicate ID table from driver.rs - I/O: - Make I/O regions typed (with a dynamically-sized Region type for the existing untyped case), create view types representing subregions of a mapped I/O region, and add io_project!() for safely creating subviews. - Split Io into a base trait (IoBase) and an extension trait (Io) with a blanket implementation, preventing implementers from overriding provided methods that unsafe code relies on. - Add a SysMem backend for shared system memory with volatile access, and make Coherent implement Io via an I/O view type. Add IoSysMap as sum type of Mmio and SysMem. Add copying methods (memcpy_{from,to}io()) and read_val()/write_val() for typed access. - Replace dma_read!()/dma_write!() with io_read!()/io_write!() for primitives and copying methods for aggregates; drop the old macros. Convert nova-core to use I/O projection. - Fix internal shortcut rule dispatch in the register!() macro, remove unused rule arguments, and use path fragments for alias destinations - IRQ: - Make irq::Registration compatible with lifetime-bound drivers by removing the 'static bound on Handler/ThreadedHandler and replacing Devres<RegistrationInner> with direct request_irq()/free_irq() calls. Handlers can now directly own lifetime-bound device resources - PCI: - Convert IrqVectorRegistration to a lifetime-annotated owning type, giving drivers explicit control over the allocation lifetime. IrqVector embeds a resolved IrqRequest, making the conversion infallible. Remove the redundant request_irq()/request_threaded_irq() wrappers from pci::Device. - Add pci_irq_type() C helper and expose it via irq_type() on IrqVectorRegistration and IrqVector, returning PCI_IRQ_MSIX, PCI_IRQ_MSI, or PCI_IRQ_INTX. - Mark pci::Device refcount methods inline - Serdev: - Add Rust abstractions for the serial device bus, including serdev::Driver trait, serdev::Device wrapping struct serdev_device, and serdev::Adapter implementing RegistrationOps. Includes a sample driver. Markus Probst takes over as serdev maintainer for both C and Rust code - Misc: - Split ForLt into a base trait (providing the Of<'a> GAT) and an unsafe CovariantForLt subtrait guaranteeing covariance, enabling invariant types (e.g. those containing Mutex<&'bound T>) to participate in the ForLt abstraction. - Fix Coherent read past EOF returning -ERANGE instead of zero. - Fix firmware example UB by avoiding null-pointer ARef misc: - Avoid iattr allocation in kernfs listxattr by using kernfs_iattrs_noalloc(). - Unregister SoC bus on early device registration failure. - Remove unused DMA_FENCE_TRACE Kconfig symbol. - Fix /sys/module path in comment. - Refactor ISA bus init to remove nested blocks. - Remove redundant nodemask clears in numa_init(). - Add kernel-doc for fwnode_operations and sys_soc.h, mark internal property data as private for kernel-doc, and add property.h/fwnode.h to driver-api infrastructure docs. - Add MAINTAINERS entry for sys_soc.h" * tag 'driver-core-7.3-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/driver-core/driver-core: (129 commits) rust: pci: expose the allocated interrupt type PCI: Add pci_irq_type() to query the allocated interrupt type rust: pci: remove request_irq() and request_threaded_irq() from Device rust: pci: resolve IRQ in index() and embed IrqRequest in IrqVector rust: pci: convert IrqVectorRegistration to a lifetime-managed owning type kernfs: avoid iattr allocation in listxattr rust: serdev: use ThisModule::as_ptr() instead of field access ACPI/IORT: use platform_device_set_fwnode() ACPI/APMT: use platform_device_set_fwnode() firmware_loader: do not queue completed sysfs fallback requests rust: pci: Mark Device refcount methods inline rust: irq: make Registration compatible with lifetime-bound drivers rust: net/phy: remove expansion from doc rust: dma: return zero for Coherent reads past EOF rust: io: register: use path fragment for alias destination rust: io: register: remove unused rule arguments rust: io: register: dispatch shortcut rules internally MAINTAINERS: add sys_soc.h to DRIVER CORE rust: debugfs: remove unsafe blocks from traits impl for Vec rust: debugfs: migrate debugfs traits requirements to zerocopy ...
Diffstat (limited to 'rust')
-rw-r--r--rust/bindings/bindings_helper.h1
-rw-r--r--rust/helpers/acpi.c16
-rw-r--r--rust/helpers/helpers.c2
-rw-r--r--rust/helpers/io.c13
-rw-r--r--rust/helpers/pci.c5
-rw-r--r--rust/helpers/serdev.c22
-rw-r--r--rust/kernel/acpi.rs14
-rw-r--r--rust/kernel/auxiliary.rs96
-rw-r--r--rust/kernel/debugfs/traits.rs41
-rw-r--r--rust/kernel/device.rs71
-rw-r--r--rust/kernel/device_id.rs196
-rw-r--r--rust/kernel/devres.rs151
-rw-r--r--rust/kernel/dma.rs287
-rw-r--r--rust/kernel/driver.rs135
-rw-r--r--rust/kernel/firmware.rs8
-rw-r--r--rust/kernel/i2c.rs26
-rw-r--r--rust/kernel/io.rs1519
-rw-r--r--rust/kernel/io/mem.rs94
-rw-r--r--rust/kernel/io/poll.rs6
-rw-r--r--rust/kernel/io/register.rs86
-rw-r--r--rust/kernel/irq.rs2
-rw-r--r--rust/kernel/irq/request.rs430
-rw-r--r--rust/kernel/lib.rs5
-rw-r--r--rust/kernel/net/phy.rs66
-rw-r--r--rust/kernel/of.rs14
-rw-r--r--rust/kernel/pci.rs32
-rw-r--r--rust/kernel/pci/io.rs203
-rw-r--r--rust/kernel/pci/irq.rs210
-rw-r--r--rust/kernel/platform.rs53
-rw-r--r--rust/kernel/ptr.rs12
-rw-r--r--rust/kernel/serdev.rs604
-rw-r--r--rust/kernel/types.rs5
-rw-r--r--rust/kernel/types/for_lt.rs103
-rw-r--r--rust/kernel/usb.rs24
-rw-r--r--rust/macros/for_lt.rs41
-rw-r--r--rust/macros/lib.rs18
36 files changed, 3140 insertions, 1471 deletions
diff --git a/rust/bindings/bindings_helper.h b/rust/bindings/bindings_helper.h
index 1124785e210b..fe7c505da236 100644
--- a/rust/bindings/bindings_helper.h
+++ b/rust/bindings/bindings_helper.h
@@ -85,6 +85,7 @@
#include <linux/regulator/consumer.h>
#include <linux/sched.h>
#include <linux/security.h>
+#include <linux/serdev.h>
#include <linux/slab.h>
#include <linux/sys_soc.h>
#include <linux/task_work.h>
diff --git a/rust/helpers/acpi.c b/rust/helpers/acpi.c
deleted file mode 100644
index e75c9807bbad..000000000000
--- a/rust/helpers/acpi.c
+++ /dev/null
@@ -1,16 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-
-#include <linux/acpi.h>
-#include <acpi/acpi_bus.h>
-
-__rust_helper bool rust_helper_acpi_of_match_device(const struct acpi_device *adev,
- const struct of_device_id *of_match_table,
- const struct of_device_id **of_id)
-{
- return acpi_of_match_device(adev, of_match_table, of_id);
-}
-
-__rust_helper struct acpi_device *rust_helper_to_acpi_device_node(struct fwnode_handle *fwnode)
-{
- return to_acpi_device_node(fwnode);
-}
diff --git a/rust/helpers/helpers.c b/rust/helpers/helpers.c
index d60af162a078..198c5f807eba 100644
--- a/rust/helpers/helpers.c
+++ b/rust/helpers/helpers.c
@@ -38,7 +38,6 @@
#define __rust_helper __always_inline
#endif
-#include "acpi.c"
#include "atomic.c"
#include "atomic_ext.c"
#include "auxiliary.c"
@@ -90,6 +89,7 @@
#include "regulator.c"
#include "scatterlist.c"
#include "security.c"
+#include "serdev.c"
#include "signal.c"
#include "slab.c"
#include "spinlock.c"
diff --git a/rust/helpers/io.c b/rust/helpers/io.c
index 1edbc274951c..308950aae19c 100644
--- a/rust/helpers/io.c
+++ b/rust/helpers/io.c
@@ -21,6 +21,19 @@ __rust_helper void rust_helper_iounmap(void __iomem *addr)
}
#endif /* CONFIG_HAS_IOMEM */
+__rust_helper void rust_helper_memcpy_fromio(void *dst,
+ const volatile void __iomem *src,
+ size_t count)
+{
+ memcpy_fromio(dst, src, count);
+}
+
+__rust_helper void rust_helper_memcpy_toio(volatile void __iomem *dst,
+ const void *src, size_t count)
+{
+ memcpy_toio(dst, src, count);
+}
+
__rust_helper u8 rust_helper_readb(const void __iomem *addr)
{
return readb(addr);
diff --git a/rust/helpers/pci.c b/rust/helpers/pci.c
index e44905317d75..23b06becb448 100644
--- a/rust/helpers/pci.c
+++ b/rust/helpers/pci.c
@@ -24,6 +24,11 @@ __rust_helper bool rust_helper_dev_is_pci(const struct device *dev)
return dev_is_pci(dev);
}
+__rust_helper unsigned int rust_helper_pci_irq_type(struct pci_dev *pdev)
+{
+ return pci_irq_type(pdev);
+}
+
#ifndef CONFIG_PCI_MSI
__rust_helper int rust_helper_pci_alloc_irq_vectors(struct pci_dev *dev,
unsigned int min_vecs,
diff --git a/rust/helpers/serdev.c b/rust/helpers/serdev.c
new file mode 100644
index 000000000000..c52b78ca3fc7
--- /dev/null
+++ b/rust/helpers/serdev.c
@@ -0,0 +1,22 @@
+// SPDX-License-Identifier: GPL-2.0
+
+#include <linux/serdev.h>
+
+__rust_helper
+void rust_helper_serdev_device_driver_unregister(struct serdev_device_driver *sdrv)
+{
+ serdev_device_driver_unregister(sdrv);
+}
+
+__rust_helper
+void rust_helper_serdev_device_put(struct serdev_device *serdev)
+{
+ serdev_device_put(serdev);
+}
+
+__rust_helper
+void rust_helper_serdev_device_set_client_ops(struct serdev_device *serdev,
+ const struct serdev_device_ops *ops)
+{
+ serdev_device_set_client_ops(serdev, ops);
+}
diff --git a/rust/kernel/acpi.rs b/rust/kernel/acpi.rs
index 9b8efa623130..ea2ce61ee393 100644
--- a/rust/kernel/acpi.rs
+++ b/rust/kernel/acpi.rs
@@ -25,10 +25,6 @@ unsafe impl RawDeviceId for DeviceId {
// SAFETY: `DRIVER_DATA_OFFSET` is the offset to the `driver_data` field.
unsafe impl RawDeviceIdIndex for DeviceId {
const DRIVER_DATA_OFFSET: usize = core::mem::offset_of!(bindings::acpi_device_id, driver_data);
-
- fn index(&self) -> usize {
- self.0.driver_data
- }
}
impl DeviceId {
@@ -53,13 +49,7 @@ impl DeviceId {
/// Create an ACPI `IdTable` with an "alias" for modpost.
#[macro_export]
macro_rules! acpi_device_table {
- ($table_name:ident, $module_table_name:ident, $id_info_type: ty, $table_data: expr) => {
- const $table_name: $crate::device_id::IdArray<
- $crate::acpi::DeviceId,
- $id_info_type,
- { $table_data.len() },
- > = $crate::device_id::IdArray::new($table_data);
-
- $crate::module_device_table!("acpi", $module_table_name, $table_name);
+ ($($tt:tt)*) => {
+ $crate::module_device_table!("acpi", $crate::acpi::DeviceId, $($tt)*);
};
}
diff --git a/rust/kernel/auxiliary.rs b/rust/kernel/auxiliary.rs
index cc9745fbf179..60dfbec8f330 100644
--- a/rust/kernel/auxiliary.rs
+++ b/rust/kernel/auxiliary.rs
@@ -20,6 +20,7 @@ use crate::{
},
prelude::*,
types::{
+ CovariantForLt,
ForLt,
ForeignOwnable,
Opaque, //
@@ -93,7 +94,9 @@ impl<T: Driver> Adapter<T> {
// SAFETY: `DeviceId` is a `#[repr(transparent)`] wrapper of `struct auxiliary_device_id`
// and does not add additional invariants, so it's safe to transmute.
let id = unsafe { &*id.cast::<DeviceId>() };
- let info = T::ID_TABLE.info(id.index());
+
+ // SAFETY: `id` comes from `T::ID_TABLE` which is of type `IdArray<_, T::IdInfo>`.
+ let info = unsafe { id.info_unchecked::<T::IdInfo>() };
from_result(|| {
let data = T::probe(adev, info);
@@ -169,10 +172,6 @@ unsafe impl RawDeviceId for DeviceId {
unsafe impl RawDeviceIdIndex for DeviceId {
const DRIVER_DATA_OFFSET: usize =
core::mem::offset_of!(bindings::auxiliary_device_id, driver_data);
-
- fn index(&self) -> usize {
- self.0.driver_data
- }
}
/// IdTable type for auxiliary drivers.
@@ -181,14 +180,8 @@ pub type IdTable<T> = &'static dyn kernel::device_id::IdTable<DeviceId, T>;
/// Create a auxiliary `IdTable` with its alias for modpost.
#[macro_export]
macro_rules! auxiliary_device_table {
- ($table_name:ident, $module_table_name:ident, $id_info_type: ty, $table_data: expr) => {
- const $table_name: $crate::device_id::IdArray<
- $crate::auxiliary::DeviceId,
- $id_info_type,
- { $table_data.len() },
- > = $crate::device_id::IdArray::new($table_data);
-
- $crate::module_device_table!("auxiliary", $module_table_name, $table_name);
+ ($($tt:tt)*) => {
+ $crate::module_device_table!("auxiliary", $crate::auxiliary::DeviceId, $($tt)*);
};
}
@@ -270,18 +263,15 @@ impl Device<device::Bound> {
unsafe { parent.as_bound() }
}
- /// Returns a pinned reference to the registration data set by the registering (parent) driver.
+ /// Returns the stored registration data as a pinned reference.
///
- /// `F` is the [`ForLt`](trait@ForLt) encoding of the data type. The returned
- /// reference has its lifetime shortened from `'static` to `&self`'s borrow lifetime via
- /// [`ForLt::cast_ref`].
+ /// Performs null and [`TypeId`] checks, then borrows the stored [`KBox`].
///
- /// Returns [`EINVAL`] if `F` does not match the type used by the parent driver when calling
- /// [`Registration::new()`].
+ /// # Safety
///
- /// Returns [`ENOENT`] if no registration data has been set, e.g. when the device was
- /// registered by a C driver.
- pub fn registration_data<F: ForLt + 'static>(&self) -> Result<Pin<&F::Of<'_>>> {
+ /// Callers must ensure that the lifetime shortening from the original `'static` storage to
+ /// `'_` is sound, e.g. via an HRTB closure or [`CovariantForLt`] guarantee.
+ unsafe fn registration_data_pinned<F: ForLt + 'static>(&self) -> Result<Pin<&F::Of<'_>>> {
// SAFETY: By the type invariant, `self.as_raw()` is a valid `struct auxiliary_device`.
let ptr = unsafe { (*self.as_raw()).registration_data_rust };
if ptr.is_null() {
@@ -300,17 +290,59 @@ impl Device<device::Bound> {
return Err(EINVAL);
}
- // SAFETY: The `TypeId` check above confirms that the stored type matches
- // `F::Of<'static>`; `ptr` remains valid until `Registration::drop()` calls
- // `from_foreign()`.
- let wrapper = unsafe { Pin::<KBox<RegistrationData<F::Of<'static>>>>::borrow(ptr) };
+ // SAFETY: The `TypeId` check above confirms that the stored type matches `F`'s
+ // encoding; lifetimes are erased at runtime, so borrowing as `F::Of<'_>` is
+ // layout-compatible with the stored `F::Of<'static>`. `ptr` remains valid until
+ // `Registration::drop()` calls `from_foreign()`.
+ let wrapper = unsafe { Pin::<KBox<RegistrationData<F::Of<'_>>>>::borrow(ptr) };
// SAFETY: `data` is a structurally pinned field of `RegistrationData`.
- let pinned: Pin<&F::Of<'_>> = unsafe { wrapper.map_unchecked(|w| &w.data) };
+ Ok(unsafe { wrapper.map_unchecked(|w| &w.data) })
+ }
+
+ /// Access the registration data set by the registering (parent) driver through a closure.
+ ///
+ /// `F` is the [`ForLt`](trait@ForLt) encoding of the data type. The closure receives a pinned
+ /// reference to the registration data.
+ ///
+ /// For covariant types that implement [`trait@CovariantForLt`], prefer
+ /// [`registration_data`](Self::registration_data) which returns a direct reference.
+ ///
+ /// Returns [`EINVAL`] if `F` does not match the type used by the parent driver when calling
+ /// [`Registration::new()`].
+ ///
+ /// Returns [`ENOENT`] if no registration data has been set, e.g. when the device was
+ /// registered by a C driver.
+ #[inline]
+ pub fn registration_data_with<F: ForLt + 'static, R>(
+ &self,
+ f: impl for<'a> FnOnce(Pin<&'a F::Of<'a>>) -> R,
+ ) -> Result<R> {
+ // SAFETY: The HRTB closure prevents the caller from smuggling in references with a
+ // concrete short lifetime, making the round-trip from `'static` sound regardless of
+ // variance.
+ let pinned = unsafe { self.registration_data_pinned::<F>()? };
- // SAFETY: The data was pinned when stored; `cast_ref` only shortens
- // the lifetime, so the pinning guarantee is preserved.
- Ok(unsafe { Pin::new_unchecked(F::cast_ref(pinned.get_ref())) })
+ Ok(f(pinned))
+ }
+
+ /// Returns a pinned reference to the registration data set by the registering (parent) driver.
+ ///
+ /// This method is only available when `F` implements [`trait@CovariantForLt`], which guarantees
+ /// that the lifetime shortening is sound.
+ ///
+ /// For non-covariant types, use the closure-based [`Self::registration_data_with`].
+ ///
+ /// Returns [`EINVAL`] if `F` does not match the type used by the parent driver when calling
+ /// [`Registration::new()`].
+ ///
+ /// Returns [`ENOENT`] if no registration data has been set, e.g. when the device was
+ /// registered by a C driver.
+ #[inline]
+ pub fn registration_data<F: CovariantForLt + 'static>(&self) -> Result<Pin<&F::Of<'_>>> {
+ // SAFETY: `CovariantForLt` guarantees covariance, which makes the lifetime shortening
+ // from `'static` to `'_` performed by `registration_data_pinned` sound.
+ unsafe { self.registration_data_pinned::<F>() }
}
}
@@ -401,7 +433,9 @@ struct RegistrationData<T> {
///
/// The type parameter `F` is a [`ForLt`](trait@ForLt) encoding of the registration
/// data type. For non-lifetime-parameterized types, use [`ForLt!(T)`](macro@ForLt).
-/// The data can be accessed by the auxiliary driver through [`Device::registration_data()`].
+///
+/// The data can be accessed by the auxiliary driver through [`Device::registration_data()`] and
+/// [`Device::registration_data_with()`].
///
/// # Invariants
///
diff --git a/rust/kernel/debugfs/traits.rs b/rust/kernel/debugfs/traits.rs
index 8c39524b6a99..b295f8420abd 100644
--- a/rust/kernel/debugfs/traits.rs
+++ b/rust/kernel/debugfs/traits.rs
@@ -18,10 +18,6 @@ use crate::{
Arc,
Mutex, //
},
- transmute::{
- AsBytes,
- FromBytes, //
- },
uaccess::{
UserSliceReader,
UserSliceWriter, //
@@ -36,6 +32,8 @@ use core::{
str::FromStr,
};
+use zerocopy::Immutable;
+
/// A trait for types that can be written into a string.
///
/// This works very similarly to `Debug`, and is automatically implemented if `Debug` is
@@ -76,8 +74,8 @@ pub trait BinaryWriter {
) -> Result<usize>;
}
-// Base implementation for any `T: AsBytes`.
-impl<T: AsBytes> BinaryWriter for T {
+// Base implementation for any `T: Immutable + IntoBytes`.
+impl<T: Immutable + IntoBytes> BinaryWriter for T {
fn write_to_slice(
&self,
writer: &mut UserSliceWriter,
@@ -147,7 +145,7 @@ where
// Delegate for `Vec<T, A>`.
impl<T, A> BinaryWriter for Vec<T, A>
where
- T: AsBytes,
+ T: Immutable + IntoBytes,
A: Allocator,
{
fn write_to_slice(
@@ -155,14 +153,7 @@ where
writer: &mut UserSliceWriter,
offset: &mut file::Offset,
) -> Result<usize> {
- let slice = self.as_slice();
-
- // SAFETY: `T: AsBytes` allows us to treat `&[T]` as `&[u8]`.
- let buffer = unsafe {
- core::slice::from_raw_parts(slice.as_ptr().cast(), core::mem::size_of_val(slice))
- };
-
- writer.write_slice_file(buffer, offset)
+ writer.write_slice_file(self.as_bytes(), offset)
}
}
@@ -230,14 +221,14 @@ pub trait BinaryReaderMut {
) -> Result<usize>;
}
-// Base implementation for any `T: AsBytes + FromBytes`.
-impl<T: AsBytes + FromBytes> BinaryReaderMut for T {
+// Base implementation for any `T: FromBytes + IntoBytes`.
+impl<T: FromBytes + IntoBytes> BinaryReaderMut for T {
fn read_from_slice_mut(
&mut self,
reader: &mut UserSliceReader,
offset: &mut file::Offset,
) -> Result<usize> {
- reader.read_slice_file(self.as_bytes_mut(), offset)
+ reader.read_slice_file(self.as_mut_bytes(), offset)
}
}
@@ -255,7 +246,7 @@ impl<T: ?Sized + BinaryReaderMut, A: Allocator> BinaryReaderMut for Box<T, A> {
// Delegate for `Vec<T, A>`: Support a `Vec<T, A>` with an outer lock.
impl<T, A> BinaryReaderMut for Vec<T, A>
where
- T: AsBytes + FromBytes,
+ T: FromBytes + IntoBytes,
A: Allocator,
{
fn read_from_slice_mut(
@@ -263,17 +254,7 @@ where
reader: &mut UserSliceReader,
offset: &mut file::Offset,
) -> Result<usize> {
- let slice = self.as_mut_slice();
-
- // SAFETY: `T: AsBytes + FromBytes` allows us to treat `&mut [T]` as `&mut [u8]`.
- let buffer = unsafe {
- core::slice::from_raw_parts_mut(
- slice.as_mut_ptr().cast(),
- core::mem::size_of_val(slice),
- )
- };
-
- reader.read_slice_file(buffer, offset)
+ reader.read_slice_file(self.as_mut_bytes(), offset)
}
}
diff --git a/rust/kernel/device.rs b/rust/kernel/device.rs
index 1a38b3bbdfb7..2291d85b6849 100644
--- a/rust/kernel/device.rs
+++ b/rust/kernel/device.rs
@@ -54,7 +54,8 @@ pub mod property;
/// reference is valid in. For instance, the [`Bound`] context guarantees that the [`Device`] is
/// bound to a driver for the entire duration of the existence of a [`Device<Bound>`] reference.
///
-/// Other [`DeviceContext`] types besides [`Bound`] are [`Normal`], [`Core`] and [`CoreInternal`].
+/// Other [`DeviceContext`] types besides [`Bound`] are [`Normal`], [`Core`], [`CoreInternal`] and
+/// [`BoundInternal`].
///
/// Unless selected otherwise [`Device`] defaults to the [`Normal`] [`DeviceContext`], which by
/// itself has no additional requirements.
@@ -235,7 +236,9 @@ impl<'a> Device<CoreInternal<'a>> {
// in `into_foreign()`.
Some(unsafe { Pin::<KBox<T>>::from_foreign(ptr.cast()) })
}
+}
+impl<Ctx: InternalBoundContext> Device<Ctx> {
/// Borrow the driver's private data bound to this [`Device`].
///
/// # Safety
@@ -245,22 +248,6 @@ impl<'a> Device<CoreInternal<'a>> {
/// - The type `T` must match the type of the `ForeignOwnable` previously stored by
/// [`Device::set_drvdata`].
pub unsafe fn drvdata_borrow<T>(&self) -> Pin<&T> {
- // SAFETY: `drvdata_unchecked()` has the exact same safety requirements as the ones
- // required by this method.
- unsafe { self.drvdata_unchecked() }
- }
-}
-
-impl Device<Bound> {
- /// Borrow the driver's private data bound to this [`Device`].
- ///
- /// # Safety
- ///
- /// - Must only be called after a preceding call to [`Device::set_drvdata`] and before
- /// the device is fully unbound.
- /// - The type `T` must match the type of the `ForeignOwnable` previously stored by
- /// [`Device::set_drvdata`].
- unsafe fn drvdata_unchecked<T>(&self) -> Pin<&T> {
// SAFETY: By the type invariants, `self.as_raw()` is a valid pointer to a `struct device`.
let ptr = unsafe { bindings::dev_get_drvdata(self.as_raw()) };
@@ -476,7 +463,8 @@ unsafe impl Sync for Device<Bound> {}
/// [`DeviceContext`] is a marker trait for types representing the context of a bus specific
/// [`Device`].
///
-/// The specific device context types are: [`CoreInternal`], [`Core`], [`Bound`] and [`Normal`].
+/// The specific device context types are: [`CoreInternal`], [`Core`], [`BoundInternal`], [`Bound`]
+/// and [`Normal`].
///
/// [`DeviceContext`] types are hierarchical, which means that there is a strict hierarchy that
/// defines which [`DeviceContext`] type can be derived from another. For instance, any
@@ -485,6 +473,11 @@ unsafe impl Sync for Device<Bound> {}
/// The following enumeration illustrates the dereference hierarchy of [`DeviceContext`] types.
///
/// - [`CoreInternal`] => [`Core`] => [`Bound`] => [`Normal`]
+/// - [`BoundInternal`] => [`Bound`] => [`Normal`]
+///
+/// Both [`CoreInternal`] and [`BoundInternal`] implement the [`InternalBoundContext`] trait,
+/// which provides access to internal bus abstraction methods on [`Device`] that are not available
+/// to drivers.
///
/// Bus devices can automatically implement the dereference hierarchy by using
/// [`impl_device_context_deref`].
@@ -511,7 +504,11 @@ pub struct Normal;
/// callback it appears in. It is intended to be used for synchronization purposes. Bus device
/// implementations can implement methods for [`Device<Core>`], such that they can only be called
/// from bus callbacks.
-pub struct Core<'a>(PhantomData<&'a ()>);
+///
+/// The lifetime `'a` is for "lifetime branding" purpose. Callbacks need to polymorphic over this
+/// lifetime so the `&'bound Device<Core<'_>>` provided to them cannot outlive the scope of the
+/// function. For this reason, it needs to be invariant.
+pub struct Core<'a>(PhantomData<fn(&'a ()) -> &'a ()>);
/// Semantically the same as [`Core`], but reserved for internal usage of the corresponding bus
/// abstraction.
@@ -522,7 +519,21 @@ pub struct Core<'a>(PhantomData<&'a ()>);
///
/// This context mainly exists to share generic [`Device`] infrastructure that should only be called
/// from bus callbacks with bus abstractions, but without making them accessible for drivers.
-pub struct CoreInternal<'a>(PhantomData<&'a ()>);
+///
+/// Lifetime `'a` is invariant for the same reason as [`Core`].
+pub struct CoreInternal<'a>(PhantomData<fn(&'a ()) -> &'a ()>);
+
+/// Semantically the same as [`Bound`], but reserved for internal usage of the corresponding bus
+/// abstraction.
+///
+/// The internal bound context is intended to be used in exactly the same way as the [`Bound`]
+/// context, with the difference that this [`DeviceContext`] is internal to the corresponding bus
+/// abstraction.
+///
+/// This context exists for cases where the bus abstraction needs access to internal device
+/// infrastructure (such as [`Device::drvdata_borrow`]), where [`CoreInternal`] would not be
+/// justified.
+pub struct BoundInternal;
/// The [`Bound`] context is the [`DeviceContext`] of a bus specific device when it is guaranteed to
/// be bound to a driver.
@@ -546,16 +557,28 @@ mod private {
pub trait Sealed {}
impl Sealed for super::Bound {}
+ impl Sealed for super::BoundInternal {}
impl<'a> Sealed for super::Core<'a> {}
impl<'a> Sealed for super::CoreInternal<'a> {}
impl Sealed for super::Normal {}
}
impl DeviceContext for Bound {}
+impl DeviceContext for BoundInternal {}
impl<'a> DeviceContext for Core<'a> {}
impl<'a> DeviceContext for CoreInternal<'a> {}
impl DeviceContext for Normal {}
+/// Marker trait for [`DeviceContext`] types that have internal bound-level access.
+///
+/// This trait is implemented by [`CoreInternal`] and [`BoundInternal`], allowing methods that
+/// require internal bus abstraction access to a bound device to be generic over both contexts.
+///
+/// Methods bounded by this trait are available to bus abstractions but not to drivers.
+pub trait InternalBoundContext: DeviceContext {}
+impl<'a> InternalBoundContext for CoreInternal<'a> {}
+impl InternalBoundContext for BoundInternal {}
+
impl<Ctx: DeviceContext> AsRef<Device<Ctx>> for Device<Ctx> {
#[inline]
fn as_ref(&self) -> &Device<Ctx> {
@@ -665,6 +688,13 @@ macro_rules! impl_device_context_deref {
// `__impl_device_context_deref!`.
::kernel::__impl_device_context_deref!(unsafe {
$device,
+ $crate::device::BoundInternal => $crate::device::Bound
+ });
+
+ // SAFETY: This macro has the exact same safety requirement as
+ // `__impl_device_context_deref!`.
+ ::kernel::__impl_device_context_deref!(unsafe {
+ $device,
$crate::device::Bound => $crate::device::Normal
});
};
@@ -700,6 +730,7 @@ macro_rules! impl_device_context_into_aref {
::kernel::__impl_device_context_into_aref!(
<'a> $crate::device::Core<'a>, $device
);
+ ::kernel::__impl_device_context_into_aref!($crate::device::BoundInternal, $device);
::kernel::__impl_device_context_into_aref!($crate::device::Bound, $device);
};
}
diff --git a/rust/kernel/device_id.rs b/rust/kernel/device_id.rs
index 8e9721446014..c81fca5b4986 100644
--- a/rust/kernel/device_id.rs
+++ b/rust/kernel/device_id.rs
@@ -5,7 +5,10 @@
//! Each bus / subsystem that matches device and driver through a bus / subsystem specific ID is
//! expected to implement [`RawDeviceId`].
-use core::mem::MaybeUninit;
+use core::{
+ marker::PhantomData,
+ mem::MaybeUninit, //
+};
/// Marker trait to indicate a Rust device ID type represents a corresponding C device ID type.
///
@@ -47,112 +50,110 @@ pub unsafe trait RawDeviceIdIndex: RawDeviceId {
/// The offset (in bytes) to the context/data field in the raw device ID.
const DRIVER_DATA_OFFSET: usize;
- /// The index stored at `DRIVER_DATA_OFFSET` of the implementor of the [`RawDeviceIdIndex`]
- /// trait.
- fn index(&self) -> usize;
-}
-
-/// A zero-terminated device id array.
-#[repr(C)]
-pub struct RawIdArray<T: RawDeviceId, const N: usize> {
- ids: [T::RawType; N],
- sentinel: MaybeUninit<T::RawType>,
-}
+ /// Obtain the data pointer stored inside the device ID.
+ ///
+ /// # Safety
+ ///
+ /// `&Self` must be stored inside a `IdArray<Self, U>`.
+ unsafe fn info_unchecked<U>(&self) -> &'static U {
+ // SAFETY: By safety requirement of the trait, this is `self.driver_data as *const U` and by
+ // the safety requirement of the function, this is stored in `IdArray<Self, U>` so is
+ // convertible to `&'static U`.
+ unsafe {
+ core::ptr::from_ref(self)
+ .byte_add(Self::DRIVER_DATA_OFFSET)
+ .cast::<&U>()
+ .read()
+ }
+ }
-impl<T: RawDeviceId, const N: usize> RawIdArray<T, N> {
- #[doc(hidden)]
- pub const fn size(&self) -> usize {
- core::mem::size_of::<Self>()
+ /// Obtain the data pointer stored inside the device ID.
+ ///
+ /// # Safety
+ ///
+ /// `&Self` must be stored inside a `IdArray<Self, U>`, or has NULL (or 0) as driver data.
+ unsafe fn info_unchecked_opt<U>(&self) -> Option<&'static U> {
+ // SAFETY: By safety requirement of the trait, this is `self.driver_data as *const U` and by
+ // the safety requirement of the function, if this is stored in `IdArray<Self, U>`, this is
+ // convertible to `Option<&'static U>`. Otherwise it is NULL which is `None` as
+ // `Option<&U>`.
+ unsafe {
+ core::ptr::from_ref(self)
+ .byte_add(Self::DRIVER_DATA_OFFSET)
+ .cast::<Option<&U>>()
+ .read()
+ }
}
}
/// A zero-terminated device id array, followed by context data.
#[repr(C)]
-pub struct IdArray<T: RawDeviceId, U, const N: usize> {
- raw_ids: RawIdArray<T, N>,
- id_infos: [U; N],
+pub struct IdArray<T: RawDeviceId, U: 'static, const N: usize> {
+ // This is `MaybeUninit<T::RawType>` so any bytes inside it can carry provenance in CTFE.
+ // If this were `T::RawType`, integer fields would not be able to contain pointers.
+ ids: [MaybeUninit<T::RawType>; N],
+ sentinel: MaybeUninit<T::RawType>,
+ phantom: PhantomData<&'static U>,
}
-impl<T: RawDeviceId, U, const N: usize> IdArray<T, U, N> {
+// SAFETY: device ID is plain data plus a `&'static U` and can thus be sent between threads safely
+// if `&U` can.
+unsafe impl<T: RawDeviceId, U: Sync + 'static, const N: usize> Send for IdArray<T, U, N> {}
+
+// SAFETY: device ID is plain data plus a `&'static U` and can thus be shared between threads safely
+// if `&U` can.
+unsafe impl<T: RawDeviceId, U: Sync + 'static, const N: usize> Sync for IdArray<T, U, N> {}
+
+impl<T: RawDeviceId + RawDeviceIdIndex, U: 'static, const N: usize> IdArray<T, U, N> {
/// Creates a new instance of the array.
///
/// The contents are derived from the given identifiers and context information.
- ///
- /// # Safety
- ///
- /// `data_offset` as `None` is always safe.
- /// If `data_offset` is `Some(data_offset)`, then:
- /// - `data_offset` must be the correct offset (in bytes) to the context/data field
- /// (e.g., the `driver_data` field) within the raw device ID structure.
- /// - The field at `data_offset` must be correctly sized to hold a `usize`.
- const unsafe fn build(ids: [(T, U); N], data_offset: Option<usize>) -> Self {
+ pub const fn new(ids: [(T, &'static U); N]) -> Self {
let mut raw_ids = [const { MaybeUninit::<T::RawType>::uninit() }; N];
- let mut infos = [const { MaybeUninit::uninit() }; N];
let mut i = 0usize;
while i < N {
// SAFETY: by the safety requirement of `RawDeviceId`, we're guaranteed that `T` is
// layout-wise compatible with `RawType`.
raw_ids[i] = unsafe { core::mem::transmute_copy(&ids[i].0) };
- if let Some(data_offset) = data_offset {
- // SAFETY: by the safety requirement of this function, this would be effectively
- // `raw_ids[i].driver_data = i;`.
- unsafe {
- raw_ids[i]
- .as_mut_ptr()
- .byte_add(data_offset)
- .cast::<usize>()
- .write(i);
- }
+ // SAFETY: by the safety requirement of `RawDeviceIdIndex`, this would be effectively
+ // `raw_ids[i].driver_data = ids[i].1;`.
+ unsafe {
+ raw_ids[i]
+ .as_mut_ptr()
+ .byte_add(T::DRIVER_DATA_OFFSET)
+ .cast::<&U>()
+ .write(ids[i].1);
}
- // SAFETY: this is effectively a move: `infos[i] = ids[i].1`. We make a copy here but
- // later forget `ids`.
- infos[i] = MaybeUninit::new(unsafe { core::ptr::read(&ids[i].1) });
i += 1;
}
core::mem::forget(ids);
Self {
- raw_ids: RawIdArray {
- // SAFETY: this is effectively `array_assume_init`, which is unstable, so we use
- // `transmute_copy` instead. We have initialized all elements of `raw_ids` so this
- // `array_assume_init` is safe.
- ids: unsafe { core::mem::transmute_copy(&raw_ids) },
- sentinel: MaybeUninit::zeroed(),
- },
- // SAFETY: We have initialized all elements of `infos` so this `array_assume_init` is
- // safe.
- id_infos: unsafe { core::mem::transmute_copy(&infos) },
+ ids: raw_ids,
+ sentinel: MaybeUninit::zeroed(),
+ phantom: PhantomData,
}
}
+}
+impl<T: RawDeviceId, const N: usize> IdArray<T, (), N> {
/// Creates a new instance of the array without writing index values.
///
/// The contents are derived from the given identifiers and context information.
/// If the device implements [`RawDeviceIdIndex`], consider using [`IdArray::new`] instead.
- pub const fn new_without_index(ids: [(T, U); N]) -> Self {
- // SAFETY: Calling `Self::build` with `offset = None` is always safe,
- // because no raw memory writes are performed in this case.
- unsafe { Self::build(ids, None) }
- }
-
- /// Reference to the contained [`RawIdArray`].
- pub const fn raw_ids(&self) -> &RawIdArray<T, N> {
- &self.raw_ids
- }
-}
+ pub const fn new_without_index(ids: [T; N]) -> Self {
+ // SAFETY: `T` is layout-wise compatible with `T::RawType`, so is the array of them.
+ let raw_ids: [MaybeUninit<T::RawType>; N] = unsafe { core::mem::transmute_copy(&ids) };
+ core::mem::forget(ids);
-impl<T: RawDeviceId + RawDeviceIdIndex, U, const N: usize> IdArray<T, U, N> {
- /// Creates a new instance of the array.
- ///
- /// The contents are derived from the given identifiers and context information.
- pub const fn new(ids: [(T, U); N]) -> Self {
- // SAFETY: by the safety requirement of `RawDeviceIdIndex`,
- // `T::DRIVER_DATA_OFFSET` is guaranteed to be the correct offset (in bytes) to
- // a field within `T::RawType`.
- unsafe { Self::build(ids, Some(T::DRIVER_DATA_OFFSET)) }
+ Self {
+ ids: raw_ids,
+ sentinel: MaybeUninit::zeroed(),
+ phantom: PhantomData,
+ }
}
}
@@ -165,12 +166,6 @@ impl<T: RawDeviceId + RawDeviceIdIndex, U, const N: usize> IdArray<T, U, N> {
pub trait IdTable<T: RawDeviceId, U> {
/// Obtain the pointer to the ID table.
fn as_ptr(&self) -> *const T::RawType;
-
- /// Obtain the pointer to the bus specific device ID from an index.
- fn id(&self, index: usize) -> &T::RawType;
-
- /// Obtain the pointer to the driver-specific information from an index.
- fn info(&self, index: usize) -> &U;
}
impl<T: RawDeviceId, U, const N: usize> IdTable<T, U> for IdArray<T, U, N> {
@@ -179,28 +174,45 @@ impl<T: RawDeviceId, U, const N: usize> IdTable<T, U> for IdArray<T, U, N> {
// to access the sentinel.
core::ptr::from_ref(self).cast()
}
-
- fn id(&self, index: usize) -> &T::RawType {
- &self.raw_ids.ids[index]
- }
-
- fn info(&self, index: usize) -> &U {
- &self.id_infos[index]
- }
}
/// Create device table alias for modpost.
#[macro_export]
macro_rules! module_device_table {
- ($table_type: literal, $module_table_name:ident, $table_name:ident) => {
- #[rustfmt::skip]
+ (
+ $table_type: literal, $device_id_ty: ty,
+ $table_name: ident, $id_info_type: ty,
+ [$(($id: expr, $info:expr $(,)?)),* $(,)?]
+ ) => {
+ #[export_name =
+ concat!("__mod_device_table__", ::core::line!(),
+ "__kmod_", module_path!(),
+ "__", $table_type,
+ "__", stringify!($table_name))
+ ]
+ static $table_name: $crate::device_id::IdArray<
+ $device_id_ty,
+ $id_info_type,
+ { <[$device_id_ty]>::len(&[$($id,)*]) },
+ > = $crate::device_id::IdArray::new([$(($id, &$info),)*]);
+ };
+
+ // Case for no ID info.
+ (
+ $table_type: literal, $device_id_ty: ty,
+ $table_name: ident, @none,
+ [$($id: expr),* $(,)?]
+ ) => {
#[export_name =
- concat!("__mod_device_table__", line!(),
+ concat!("__mod_device_table__", ::core::line!(),
"__kmod_", module_path!(),
"__", $table_type,
"__", stringify!($table_name))
]
- static $module_table_name: [::core::mem::MaybeUninit<u8>; $table_name.raw_ids().size()] =
- unsafe { ::core::mem::transmute_copy($table_name.raw_ids()) };
+ static $table_name: $crate::device_id::IdArray<
+ $device_id_ty,
+ (),
+ { <[$device_id_ty]>::len(&[$($id,)*]) },
+ > = $crate::device_id::IdArray::new_without_index([$($id),*]);
};
}
diff --git a/rust/kernel/devres.rs b/rust/kernel/devres.rs
index 8ff8aedf251a..d2924aaae008 100644
--- a/rust/kernel/devres.rs
+++ b/rust/kernel/devres.rs
@@ -21,9 +21,12 @@ use crate::{
sync::{
aref::ARef,
rcu,
- Arc, //
+ Arc,
+ Completion, //
},
types::{
+ CovariantForLt,
+ ForLt,
ForeignOwnable,
Opaque, //
},
@@ -37,6 +40,8 @@ struct Inner<T> {
node: Opaque<bindings::devres_node>,
#[pin]
data: Revocable<T>,
+ #[pin]
+ revocation: Completion,
}
/// This abstraction is meant to be used by subsystems to containerize [`Device`] bound resources to
@@ -53,6 +58,10 @@ struct Inner<T> {
/// After the [`Devres`] has been unbound it is not possible to access the encapsulated resource
/// anymore.
///
+/// When a [`Devres`] is dropped, it is guaranteed that `T` has been fully dropped by the time
+/// [`Devres::drop`] returns, even if a concurrent revocation through the release callback is in
+/// progress.
+///
/// [`Devres`] users should make sure to simply free the corresponding backing resource in `T`'s
/// [`Drop`] implementation.
///
@@ -69,17 +78,19 @@ struct Inner<T> {
/// devres::Devres,
/// io::{
/// Io,
-/// IoKnownSize,
+/// IoBase,
/// Mmio,
/// MmioRaw,
-/// PhysAddr, //
+/// MmioBackend,
+/// PhysAddr,
+/// Region, //
/// },
/// prelude::*,
/// };
/// use core::ops::Deref;
///
/// // See also [`pci::Bar`] for a real example.
-/// struct IoMem<const SIZE: usize>(MmioRaw<SIZE>);
+/// struct IoMem<const SIZE: usize>(MmioRaw<Region<SIZE>>);
///
/// impl<const SIZE: usize> IoMem<SIZE> {
/// /// # Safety
@@ -94,7 +105,7 @@ struct Inner<T> {
/// return Err(ENOMEM);
/// }
///
-/// Ok(IoMem(MmioRaw::new(addr as usize, SIZE)?))
+/// Ok(IoMem(MmioRaw::new_region(addr as usize, SIZE)?))
/// }
/// }
///
@@ -105,12 +116,13 @@ struct Inner<T> {
/// }
/// }
///
-/// impl<const SIZE: usize> Deref for IoMem<SIZE> {
-/// type Target = Mmio<SIZE>;
+/// impl<'a, const SIZE: usize> IoBase<'a> for &'a IoMem<SIZE> {
+/// type Backend = MmioBackend;
+/// type Target = Region<SIZE>;
///
-/// fn deref(&self) -> &Self::Target {
+/// fn as_view(self) -> Mmio<'a, Region<SIZE>> {
/// // SAFETY: The memory range stored in `self` has been properly mapped in `Self::new`.
-/// unsafe { Mmio::from_raw(&self.0) }
+/// unsafe { Mmio::from_raw(self.0) }
/// }
/// }
/// # fn no_run(dev: &Device<Bound>) -> Result<(), Error> {
@@ -218,6 +230,7 @@ impl<T: Send + 'static> Devres<T> {
};
}),
data <- Revocable::new(data),
+ revocation <- Completion::new(),
}),
GFP_KERNEL,
)?;
@@ -255,7 +268,14 @@ impl<T: Send + 'static> Devres<T> {
// SAFETY: `inner` is a valid `Inner<T>` pointer.
let inner = unsafe { &*inner };
- inner.data.revoke();
+ if inner.data.revoke() {
+ inner.revocation.complete_all();
+ } else {
+ // Devres::drop() is concurrently revoking; wait for it to finish `drop_in_place()`
+ // before returning to `devres_release_all()`, ensuring `T` is fully torn down before
+ // the device finishes unbinding.
+ inner.revocation.wait_for_completion();
+ }
}
#[allow(clippy::missing_safety_doc)]
@@ -298,10 +318,7 @@ impl<T: Send + 'static> Devres<T> {
/// use kernel::{
/// device::Core,
/// devres::Devres,
- /// io::{
- /// Io,
- /// IoKnownSize, //
- /// },
+ /// io::Io,
/// pci, //
/// };
///
@@ -355,6 +372,8 @@ impl<T: Send + 'static> Drop for Devres<T> {
// SAFETY: When `drop` runs, it is guaranteed that nobody is accessing the revocable data
// anymore, hence it is safe not to wait for the grace period to finish.
if unsafe { self.data().revoke_nosync() } {
+ self.inner.revocation.complete_all();
+
// We revoked `self.data` before devres did, hence try to remove it.
if self.remove_node() {
// SAFETY: In `Self::new` we have taken an additional reference count of `self.data`
@@ -362,10 +381,114 @@ impl<T: Send + 'static> Drop for Devres<T> {
// this additional reference count.
drop(unsafe { Arc::from_raw(Arc::as_ptr(&self.inner)) });
}
+ } else {
+ // The release callback is concurrently revoking; wait for it to finish
+ // `drop_in_place()` of the wrapped object before returning.
+ self.inner.revocation.wait_for_completion();
}
}
}
+/// Guard returned by [`DevresLt::try_access`].
+///
+/// Dereferences to `F::Of<'a>`, shortening the lifetime of the stored data to the guard's borrow
+/// lifetime.
+pub struct DevresGuard<'a, F: CovariantForLt>(RevocableGuard<'a, F::Of<'static>>);
+
+impl<'a, F: CovariantForLt> core::ops::Deref for DevresGuard<'a, F> {
+ type Target = F::Of<'a>;
+
+ #[inline]
+ fn deref(&self) -> &Self::Target {
+ F::cast_ref(&*self.0)
+ }
+}
+
+/// Device-managed resource with [`ForLt`](trait@ForLt)-aware access.
+///
+/// `DevresLt` wraps [`Devres`] and shortens the stored `'static` lifetime to the caller's borrow
+/// lifetime in all access methods.
+///
+/// Types that implement [`trait@CovariantForLt`] get direct-reference accessors ([`Self::access`],
+/// [`Self::try_access`]). Plain [`ForLt`](trait@ForLt) types use closure-based accessors
+/// ([`Self::access_with`], [`Self::try_access_with`]).
+pub struct DevresLt<F: ForLt>(Devres<F::Of<'static>>)
+where
+ for<'a> F::Of<'a>: Send;
+
+impl<F: ForLt> DevresLt<F>
+where
+ for<'a> F::Of<'a>: Send,
+{
+ /// Creates a new [`DevresLt`] instance of the given `data`.
+ ///
+ /// # Safety
+ ///
+ /// The data must remain valid for the device's full bound scope. [`DevresLt`] allows
+ /// access until the device is unbound, which may outlast `'a`.
+ pub unsafe fn new<'a, E>(
+ dev: &'a Device<Bound>,
+ data: impl PinInit<F::Of<'a>, E>,
+ ) -> Result<Self>
+ where
+ Error: From<E>,
+ {
+ // SAFETY: The caller guarantees the data is valid for the device's full bound scope.
+ // Lifetimes do not affect layout, so F::Of<'a> and F::Of<'static> have identical
+ // representation; casting the slot pointer is sound.
+ let data = unsafe { pin_init::cast_pin_init(data) };
+
+ Ok(Self(Devres::new(dev, data)?))
+ }
+
+ /// Return a reference of the [`Device`] this [`DevresLt`] instance has been created with.
+ #[inline]
+ pub fn device(&self) -> &Device {
+ self.0.device()
+ }
+
+ /// Obtain `&F::Of<'_>`, bypassing the [`Revocable`], through a closure.
+ ///
+ /// This method works like [`DevresLt::access`](DevresLt::access) but accepts any
+ /// [`trait@ForLt`] type, not just [`trait@CovariantForLt`].
+ #[inline]
+ pub fn access_with<R, G>(&self, dev: &Device<Bound>, f: G) -> Result<R>
+ where
+ G: for<'a> FnOnce(&F::Of<'a>) -> R,
+ {
+ self.0.access(dev).map(f)
+ }
+
+ /// [`DevresLt`] accessor for [`Revocable::try_access_with`].
+ #[inline]
+ pub fn try_access_with<R, G>(&self, f: G) -> Option<R>
+ where
+ G: for<'a> FnOnce(&F::Of<'a>) -> R,
+ {
+ self.0.data().try_access_with(f)
+ }
+}
+
+impl<F: CovariantForLt> DevresLt<F>
+where
+ for<'a> F::Of<'a>: Send,
+{
+ /// Obtain `&'a F::Of<'a>`, bypassing the [`Revocable`].
+ ///
+ /// This method works like [`Devres::access`], but shortens the returned reference's lifetime
+ /// from `'static` to `'a` via [`CovariantForLt::cast_ref`].
+ #[inline]
+ pub fn access<'a>(&'a self, dev: &'a Device<Bound>) -> Result<&'a F::Of<'a>> {
+ self.0.access(dev).map(F::cast_ref)
+ }
+
+ /// [`DevresLt`] accessor for [`Revocable::try_access`].
+ #[inline]
+ pub fn try_access(&self) -> Option<DevresGuard<'_, F>> {
+ self.0.data().try_access().map(DevresGuard)
+ }
+}
+
/// Consume `data` and [`Drop::drop`] `data` once `dev` is unbound.
fn register_foreign<P>(dev: &Device<Bound>, data: P) -> Result
where
diff --git a/rust/kernel/dma.rs b/rust/kernel/dma.rs
index 8e36a4e7f514..9a0bf745614c 100644
--- a/rust/kernel/dma.rs
+++ b/rust/kernel/dma.rs
@@ -14,14 +14,22 @@ use crate::{
},
error::to_result,
fs::file,
+ io::{
+ IoBackend,
+ IoBase,
+ IoCapable,
+ IoCopyable,
+ SysMem,
+ SysMemBackend, //
+ },
prelude::*,
ptr::KnownSize,
sync::aref::ARef,
transmute::{
AsBytes,
FromBytes, //
- }, //
- uaccess::UserSliceWriter,
+ },
+ uaccess::UserSliceWriter, //
};
use core::{
ops::{
@@ -654,52 +662,6 @@ impl<T: KnownSize + ?Sized> Coherent<T> {
// SAFETY: per safety requirement.
unsafe { &mut *self.as_mut_ptr() }
}
-
- /// Reads the value of `field` and ensures that its type is [`FromBytes`].
- ///
- /// # Safety
- ///
- /// This must be called from the [`dma_read`] macro which ensures that the `field` pointer is
- /// validated beforehand.
- ///
- /// Public but hidden since it should only be used from [`dma_read`] macro.
- #[doc(hidden)]
- pub unsafe fn field_read<F: FromBytes>(&self, field: *const F) -> F {
- // SAFETY:
- // - By the safety requirements field is valid.
- // - Using read_volatile() here is not sound as per the usual rules, the usage here is
- // a special exception with the following notes in place. When dealing with a potential
- // race from a hardware or code outside kernel (e.g. user-space program), we need that
- // read on a valid memory is not UB. Currently read_volatile() is used for this, and the
- // rationale behind is that it should generate the same code as READ_ONCE() which the
- // kernel already relies on to avoid UB on data races. Note that the usage of
- // read_volatile() is limited to this particular case, it cannot be used to prevent
- // the UB caused by racing between two kernel functions nor do they provide atomicity.
- unsafe { field.read_volatile() }
- }
-
- /// Writes a value to `field` and ensures that its type is [`AsBytes`].
- ///
- /// # Safety
- ///
- /// This must be called from the [`dma_write`] macro which ensures that the `field` pointer is
- /// validated beforehand.
- ///
- /// Public but hidden since it should only be used from [`dma_write`] macro.
- #[doc(hidden)]
- pub unsafe fn field_write<F: AsBytes>(&self, field: *mut F, val: F) {
- // SAFETY:
- // - By the safety requirements field is valid.
- // - Using write_volatile() here is not sound as per the usual rules, the usage here is
- // a special exception with the following notes in place. When dealing with a potential
- // race from a hardware or code outside kernel (e.g. user-space program), we need that
- // write on a valid memory is not UB. Currently write_volatile() is used for this, and the
- // rationale behind is that it should generate the same code as WRITE_ONCE() which the
- // kernel already relies on to avoid UB on data races. Note that the usage of
- // write_volatile() is limited to this particular case, it cannot be used to prevent
- // the UB caused by racing between two kernel functions nor do they provide atomicity.
- unsafe { field.write_volatile(val) }
- }
}
impl<T: AsBytes + FromBytes> Coherent<T> {
@@ -1005,7 +967,11 @@ impl<T: KnownSize + AsBytes + ?Sized> debugfs::BinaryWriter for Coherent<T> {
return Ok(0);
};
- let count = self.size().saturating_sub(offset_val).min(writer.len());
+ if offset_val >= self.size() {
+ return Ok(0);
+ }
+
+ let count = (self.size() - offset_val).min(writer.len());
writer.write_dma(self, offset_val, count)?;
@@ -1133,84 +1099,153 @@ unsafe impl Send for CoherentHandle {}
// plain `Copy` values.
unsafe impl Sync for CoherentHandle {}
-/// Reads a field of an item from an allocated region of structs.
-///
-/// The syntax is of the form `kernel::dma_read!(dma, proj)` where `dma` is an expression evaluating
-/// to a [`Coherent`] and `proj` is a [projection specification](kernel::ptr::project!).
-///
-/// # Examples
-///
-/// ```
-/// use kernel::device::Device;
-/// use kernel::dma::{attrs::*, Coherent};
+/// View type for `Coherent`.
///
-/// struct MyStruct { field: u32, }
-///
-/// // SAFETY: All bit patterns are acceptable values for `MyStruct`.
-/// unsafe impl kernel::transmute::FromBytes for MyStruct{};
-/// // SAFETY: Instances of `MyStruct` have no uninitialized portions.
-/// unsafe impl kernel::transmute::AsBytes for MyStruct{};
-///
-/// # fn test(alloc: &kernel::dma::Coherent<[MyStruct]>) -> Result {
-/// let whole = kernel::dma_read!(alloc, [try: 2]);
-/// let field = kernel::dma_read!(alloc, [panic: 1].field);
-/// # Ok::<(), Error>(()) }
-/// ```
-#[macro_export]
-macro_rules! dma_read {
- ($dma:expr, $($proj:tt)*) => {{
- let dma = &$dma;
- let ptr = $crate::ptr::project!(
- $crate::dma::Coherent::as_ptr(dma), $($proj)*
- );
- // SAFETY: The pointer created by the projection is within the DMA region.
- unsafe { $crate::dma::Coherent::field_read(dma, ptr) }
- }};
+/// This is same as [`SysMem`] but with additional information that allows handing out a DMA handle.
+pub struct CoherentView<'a, T: ?Sized> {
+ cpu_addr: SysMem<'a, T>,
+ dma_handle: DmaAddress,
}
-/// Writes to a field of an item from an allocated region of structs.
-///
-/// The syntax is of the form `kernel::dma_write!(dma, proj, val)` where `dma` is an expression
-/// evaluating to a [`Coherent`], `proj` is a
-/// [projection specification](kernel::ptr::project!), and `val` is the value to be written to the
-/// projected location.
-///
-/// # Examples
-///
-/// ```
-/// use kernel::device::Device;
-/// use kernel::dma::{attrs::*, Coherent};
-///
-/// struct MyStruct { member: u32, }
-///
-/// // SAFETY: All bit patterns are acceptable values for `MyStruct`.
-/// unsafe impl kernel::transmute::FromBytes for MyStruct{};
-/// // SAFETY: Instances of `MyStruct` have no uninitialized portions.
-/// unsafe impl kernel::transmute::AsBytes for MyStruct{};
-///
-/// # fn test(alloc: &kernel::dma::Coherent<[MyStruct]>) -> Result {
-/// kernel::dma_write!(alloc, [try: 2].member, 0xf);
-/// kernel::dma_write!(alloc, [panic: 1], MyStruct { member: 0xf });
-/// # Ok::<(), Error>(()) }
-/// ```
-#[macro_export]
-macro_rules! dma_write {
- (@parse [$dma:expr] [$($proj:tt)*] [, $val:expr]) => {{
- let dma = &$dma;
- let ptr = $crate::ptr::project!(
- mut $crate::dma::Coherent::as_mut_ptr(dma), $($proj)*
- );
- let val = $val;
- // SAFETY: The pointer created by the projection is within the DMA region.
- unsafe { $crate::dma::Coherent::field_write(dma, ptr, val) }
- }};
- (@parse [$dma:expr] [$($proj:tt)*] [.$field:tt $($rest:tt)*]) => {
- $crate::dma_write!(@parse [$dma] [$($proj)* .$field] [$($rest)*])
- };
- (@parse [$dma:expr] [$($proj:tt)*] [[$flavor:ident: $index:expr] $($rest:tt)*]) => {
- $crate::dma_write!(@parse [$dma] [$($proj)* [$flavor: $index]] [$($rest)*])
- };
- ($dma:expr, $($rest:tt)*) => {
- $crate::dma_write!(@parse [$dma] [] [$($rest)*])
- };
+impl<T: ?Sized> Copy for CoherentView<'_, T> {}
+impl<T: ?Sized> Clone for CoherentView<'_, T> {
+ #[inline]
+ fn clone(&self) -> Self {
+ *self
+ }
+}
+
+impl<'a, T: ?Sized> CoherentView<'a, T> {
+ /// Erase the DMA handle information and obtain a [`SysMem`] view of the same memory region.
+ #[inline]
+ pub fn as_sys_mem(self) -> SysMem<'a, T> {
+ self.cpu_addr
+ }
+
+ /// Returns a DMA handle which may be given to the device as the DMA address base of the region.
+ #[inline]
+ pub fn dma_handle(self) -> DmaAddress {
+ self.dma_handle
+ }
+
+ /// Returns a reference to the data in the region.
+ ///
+ /// # Safety
+ ///
+ /// * Callers must ensure that the device does not read/write to/from memory while the returned
+ /// reference is live.
+ /// * Callers must ensure that this call does not race with a write (including call to `as_mut`)
+ /// to the same region while the returned reference is live.
+ #[inline]
+ pub unsafe fn as_ref(self) -> &'a T {
+ // SAFETY: pointer is aligned and valid per type invariant. Aliasing rule is satisfied per
+ // safety requirement.
+ unsafe { &*self.cpu_addr.as_ptr() }
+ }
+
+ /// Returns a mutable reference to the data in the region.
+ ///
+ /// # Safety
+ ///
+ /// * Callers must ensure that the device does not read/write to/from memory while the returned
+ /// reference is live.
+ /// * Callers must ensure that this call does not race with a read (including call to `as_ref`)
+ /// or write (including call to `as_mut`) to the same region while the returned reference is
+ /// live.
+ #[inline]
+ pub unsafe fn as_mut(self) -> &'a mut T {
+ // SAFETY: pointer is aligned and valid per type invariant. Aliasing rule is satisfied per
+ // safety requirement.
+ unsafe { &mut *self.cpu_addr.as_ptr() }
+ }
+}
+
+/// `IoBackend` implementation for `Coherent`.
+pub struct CoherentIoBackend;
+
+impl IoBackend for CoherentIoBackend {
+ type View<'a, T: ?Sized + KnownSize> = CoherentView<'a, T>;
+
+ #[inline]
+ fn as_ptr<'a, T: ?Sized + KnownSize>(view: Self::View<'a, T>) -> *mut T {
+ SysMemBackend::as_ptr(view.cpu_addr)
+ }
+
+ #[inline]
+ unsafe fn project_view<'a, T: ?Sized + KnownSize, U: ?Sized + KnownSize>(
+ view: Self::View<'a, T>,
+ ptr: *mut U,
+ ) -> Self::View<'a, U> {
+ let offset = ptr.addr() - view.cpu_addr.as_ptr().addr();
+ // CAST: The offset DMA address can never overflow.
+ let dma_handle = view.dma_handle + offset as DmaAddress;
+ CoherentView {
+ dma_handle,
+ // SAFETY: Per safety requirement.
+ cpu_addr: unsafe { SysMemBackend::project_view(view.cpu_addr, ptr) },
+ }
+ }
+}
+
+impl<T> IoCapable<T> for CoherentIoBackend
+where
+ SysMemBackend: IoCapable<T>,
+{
+ #[inline]
+ fn io_read<'a>(view: Self::View<'a, T>) -> T {
+ SysMemBackend::io_read(view.cpu_addr)
+ }
+
+ #[inline]
+ fn io_write<'a>(view: Self::View<'a, T>, value: T) {
+ SysMemBackend::io_write(view.cpu_addr, value)
+ }
+}
+
+impl IoCopyable for CoherentIoBackend {
+ #[inline]
+ unsafe fn copy_from_io(view: Self::View<'_, [u8]>, buffer: *mut u8) {
+ // SAFETY: Per safety requirement.
+ unsafe { SysMemBackend::copy_from_io(view.cpu_addr, buffer) }
+ }
+
+ #[inline]
+ unsafe fn copy_to_io(view: Self::View<'_, [u8]>, buffer: *const u8) {
+ // SAFETY: Per safety requirement.
+ unsafe { SysMemBackend::copy_to_io(view.cpu_addr, buffer) }
+ }
+
+ #[inline]
+ fn copy_read<T: zerocopy::FromBytes>(view: Self::View<'_, T>) -> T {
+ SysMemBackend::copy_read(view.cpu_addr)
+ }
+
+ #[inline]
+ fn copy_write<T: zerocopy::IntoBytes>(view: Self::View<'_, T>, value: T) {
+ SysMemBackend::copy_write(view.cpu_addr, value)
+ }
+}
+
+impl<'a, T: ?Sized + KnownSize> IoBase<'a> for CoherentView<'a, T> {
+ type Backend = CoherentIoBackend;
+ type Target = T;
+
+ #[inline]
+ fn as_view(self) -> CoherentView<'a, Self::Target> {
+ self
+ }
+}
+
+impl<'a, T: ?Sized + KnownSize> IoBase<'a> for &'a Coherent<T> {
+ type Backend = CoherentIoBackend;
+ type Target = T;
+
+ #[inline]
+ fn as_view(self) -> CoherentView<'a, Self::Target> {
+ CoherentView {
+ // SAFETY: `cpu_addr` is valid and aligned kernel accessible memory.
+ cpu_addr: unsafe { SysMem::new(self.cpu_addr.as_ptr()) },
+ dma_handle: self.dma_handle,
+ }
+ }
}
diff --git a/rust/kernel/driver.rs b/rust/kernel/driver.rs
index bf5ba0d27553..c9c74c4dde8f 100644
--- a/rust/kernel/driver.rs
+++ b/rust/kernel/driver.rs
@@ -291,26 +291,6 @@ macro_rules! module_driver {
}
}
-// Calling the FFI function directly from the `Adapter` impl may result in it being called
-// directly from driver modules. This happens since the Rust compiler will use monomorphisation, so
-// it might happen that functions are instantiated within the calling driver module. For now, work
-// around this with `#[inline(never)]` helpers.
-//
-// TODO: Remove once a more generic solution has been implemented. For instance, we may be able to
-// leverage `bindgen` to take care of this depending on whether a symbol is (already) exported.
-#[inline(never)]
-#[allow(clippy::missing_safety_doc)]
-#[allow(dead_code)]
-#[must_use]
-unsafe fn acpi_of_match_device(
- adev: *const bindings::acpi_device,
- of_match_table: *const bindings::of_device_id,
- of_id: *mut *const bindings::of_device_id,
-) -> bool {
- // SAFETY: Safety requirements are the same as `bindings::acpi_of_match_device`.
- unsafe { bindings::acpi_of_match_device(adev, of_match_table, of_id) }
-}
-
/// The bus independent adapter to match a drivers and a devices.
///
/// This trait should be implemented by the bus specific adapter, which represents the connection
@@ -324,118 +304,23 @@ pub trait Adapter {
/// The [`acpi::IdTable`] of the corresponding driver
fn acpi_id_table() -> Option<acpi::IdTable<Self::IdInfo>>;
- /// Returns the driver's private data from the matching entry in the [`acpi::IdTable`], if any.
- ///
- /// If this returns `None`, it means there is no match with an entry in the [`acpi::IdTable`].
- fn acpi_id_info(dev: &device::Device) -> Option<&'static Self::IdInfo> {
- #[cfg(not(CONFIG_ACPI))]
- {
- let _ = dev;
- None
- }
-
- #[cfg(CONFIG_ACPI)]
- {
- let table = Self::acpi_id_table()?;
-
- // SAFETY:
- // - `table` has static lifetime, hence it's valid for read,
- // - `dev` is guaranteed to be valid while it's alive, and so is `dev.as_raw()`.
- let raw_id = unsafe { bindings::acpi_match_device(table.as_ptr(), dev.as_raw()) };
-
- if raw_id.is_null() {
- None
- } else {
- // SAFETY: `DeviceId` is a `#[repr(transparent)]` wrapper of `struct acpi_device_id`
- // and does not add additional invariants, so it's safe to transmute.
- let id = unsafe { &*raw_id.cast::<acpi::DeviceId>() };
-
- Some(table.info(<acpi::DeviceId as crate::device_id::RawDeviceIdIndex>::index(id)))
- }
- }
- }
-
/// The [`of::IdTable`] of the corresponding driver.
fn of_id_table() -> Option<of::IdTable<Self::IdInfo>>;
- /// Returns the driver's private data from the matching entry in the [`of::IdTable`], if any.
- ///
- /// If this returns `None`, it means there is no match with an entry in the [`of::IdTable`].
- fn of_id_info(dev: &device::Device) -> Option<&'static Self::IdInfo> {
- let table = Self::of_id_table()?;
-
- #[cfg(not(any(CONFIG_OF, CONFIG_ACPI)))]
- {
- let _ = (dev, table);
- }
-
- #[cfg(CONFIG_OF)]
- {
- // SAFETY:
- // - `table` has static lifetime, hence it's valid for read,
- // - `dev` is guaranteed to be valid while it's alive, and so is `dev.as_raw()`.
- let raw_id = unsafe { bindings::of_match_device(table.as_ptr(), dev.as_raw()) };
-
- if !raw_id.is_null() {
- // SAFETY: `DeviceId` is a `#[repr(transparent)]` wrapper of `struct of_device_id`
- // and does not add additional invariants, so it's safe to transmute.
- let id = unsafe { &*raw_id.cast::<of::DeviceId>() };
-
- return Some(table.info(
- <of::DeviceId as crate::device_id::RawDeviceIdIndex>::index(id),
- ));
- }
- }
-
- #[cfg(CONFIG_ACPI)]
- {
- use core::ptr;
- use device::property::FwNode;
-
- let mut raw_id = ptr::null();
-
- let fwnode = dev.fwnode().map_or(ptr::null_mut(), FwNode::as_raw);
-
- // SAFETY: `fwnode` is a pointer to a valid `fwnode_handle`. A null pointer will be
- // passed through the function.
- let adev = unsafe { bindings::to_acpi_device_node(fwnode) };
-
- // SAFETY:
- // - `adev` is a valid pointer to `acpi_device` or is null. It is guaranteed to be
- // valid as long as `dev` is alive.
- // - `table` has static lifetime, hence it's valid for read.
- if unsafe { acpi_of_match_device(adev, table.as_ptr(), &raw mut raw_id) } {
- // SAFETY:
- // - the function returns true, therefore `raw_id` has been set to a pointer to a
- // valid `of_device_id`.
- // - `DeviceId` is a `#[repr(transparent)]` wrapper of `struct of_device_id`
- // and does not add additional invariants, so it's safe to transmute.
- let id = unsafe { &*raw_id.cast::<of::DeviceId>() };
-
- return Some(table.info(
- <of::DeviceId as crate::device_id::RawDeviceIdIndex>::index(id),
- ));
- }
- }
-
- None
- }
-
/// Returns the driver's private data from the matching entry of any of the ID tables, if any.
///
/// If this returns `None`, it means that there is no match in any of the ID tables directly
/// associated with a [`device::Device`].
- fn id_info(dev: &device::Device) -> Option<&'static Self::IdInfo> {
- let id = Self::acpi_id_info(dev);
- if id.is_some() {
- return id;
- }
-
- let id = Self::of_id_info(dev);
- if id.is_some() {
- return id;
- }
+ ///
+ /// # Safety
+ ///
+ /// The caller must ensure that the `dev` matched data is of type `Self::IdInfo`.
+ #[inline]
+ unsafe fn id_info(dev: &device::Device) -> Option<&'static Self::IdInfo> {
+ // SAFETY: `dev` is guaranteed to be valid while it's alive, and so is `dev.as_raw()`.
+ let data = unsafe { bindings::device_get_match_data(dev.as_raw()) };
- None
+ // SAFETY: Per safety requirement, `data` is of type `Self::IdInfo`.
+ unsafe { data.cast::<Self::IdInfo>().as_ref() }
}
}
diff --git a/rust/kernel/firmware.rs b/rust/kernel/firmware.rs
index 71168d8004e2..1e3dd2e3422d 100644
--- a/rust/kernel/firmware.rs
+++ b/rust/kernel/firmware.rs
@@ -51,12 +51,8 @@ impl FwFunc {
/// # Examples
///
/// ```no_run
-/// # use kernel::{device::Device, firmware::Firmware};
-///
-/// # fn no_run() -> Result<(), Error> {
-/// # // SAFETY: *NOT* safe, just for the example to get an `ARef<Device>` instance
-/// # let dev = unsafe { Device::get_device(core::ptr::null_mut()) };
-///
+/// # use kernel::{device::Device, firmware::Firmware, sync::aref::ARef};
+/// # fn no_run(dev: ARef<Device>) -> Result<(), Error> {
/// let fw = Firmware::request(c"path/to/firmware.bin", &dev)?;
/// let blob = fw.data();
///
diff --git a/rust/kernel/i2c.rs b/rust/kernel/i2c.rs
index dd9271af5eb8..fbc5fb06ca16 100644
--- a/rust/kernel/i2c.rs
+++ b/rust/kernel/i2c.rs
@@ -65,10 +65,6 @@ unsafe impl RawDeviceId for DeviceId {
// SAFETY: `DRIVER_DATA_OFFSET` is the offset to the `driver_data` field.
unsafe impl RawDeviceIdIndex for DeviceId {
const DRIVER_DATA_OFFSET: usize = core::mem::offset_of!(bindings::i2c_device_id, driver_data);
-
- fn index(&self) -> usize {
- self.0.driver_data
- }
}
/// IdTable type for I2C
@@ -77,14 +73,8 @@ pub type IdTable<T> = &'static dyn kernel::device_id::IdTable<DeviceId, T>;
/// Create a I2C `IdTable` with its alias for modpost.
#[macro_export]
macro_rules! i2c_device_table {
- ($table_name:ident, $module_table_name:ident, $id_info_type: ty, $table_data: expr) => {
- const $table_name: $crate::device_id::IdArray<
- $crate::i2c::DeviceId,
- $id_info_type,
- { $table_data.len() },
- > = $crate::device_id::IdArray::new($table_data);
-
- $crate::module_device_table!("i2c", $module_table_name, $table_name);
+ ($($tt:tt)*) => {
+ $crate::module_device_table!("i2c", $crate::i2c::DeviceId, $($tt)*);
};
}
@@ -159,8 +149,10 @@ impl<T: Driver> Adapter<T> {
// INVARIANT: `idev` is valid for the duration of `probe_callback()`.
let idev = unsafe { &*idev.cast::<I2cClient<device::CoreInternal<'_>>>() };
- let info =
- Self::i2c_id_info(idev).or_else(|| <Self as driver::Adapter>::id_info(idev.as_ref()));
+ let info = Self::i2c_id_info(idev).or_else(|| {
+ // SAFETY: `idev` matched data is of type `Self::IdInfo`.
+ unsafe { <Self as driver::Adapter>::id_info(idev.as_ref()) }
+ });
from_result(|| {
let data = T::probe(idev, info);
@@ -218,7 +210,8 @@ impl<T: Driver> Adapter<T> {
// does not add additional invariants, so it's safe to transmute.
let id = unsafe { &*raw_id.cast::<DeviceId>() };
- Some(table.info(<DeviceId as RawDeviceIdIndex>::index(id)))
+ // SAFETY: `id` comes from `table` which is of type `IdArray<_, Self::IdInfo>`.
+ Some(unsafe { id.info_unchecked::<T::IdInfo>() })
}
}
@@ -267,7 +260,6 @@ macro_rules! module_i2c_driver {
///
/// kernel::acpi_device_table!(
/// ACPI_TABLE,
-/// MODULE_ACPI_TABLE,
/// <MyDriver as i2c::Driver>::IdInfo,
/// [
/// (acpi::DeviceId::new(c"LNUXBEEF"), ())
@@ -276,7 +268,6 @@ macro_rules! module_i2c_driver {
///
/// kernel::i2c_device_table!(
/// I2C_TABLE,
-/// MODULE_I2C_TABLE,
/// <MyDriver as i2c::Driver>::IdInfo,
/// [
/// (i2c::DeviceId::new(c"rust_driver_i2c"), ())
@@ -285,7 +276,6 @@ macro_rules! module_i2c_driver {
///
/// kernel::of_device_table!(
/// OF_TABLE,
-/// MODULE_OF_TABLE,
/// <MyDriver as i2c::Driver>::IdInfo,
/// [
/// (of::DeviceId::new(c"test,device"), ())
diff --git a/rust/kernel/io.rs b/rust/kernel/io.rs
index d4063ee41200..5ce9fd129068 100644
--- a/rust/kernel/io.rs
+++ b/rust/kernel/io.rs
@@ -4,9 +4,18 @@
//!
//! C header: [`include/asm-generic/io.h`](srctree/include/asm-generic/io.h)
+use core::{
+ marker::PhantomData,
+ mem::MaybeUninit, //
+};
+
use crate::{
bindings,
- prelude::*, //
+ prelude::*,
+ ptr::{
+ Alignment,
+ KnownSize, //
+ }, //
};
#[cfg(CONFIG_HAS_IOMEM)]
@@ -32,129 +41,226 @@ pub type PhysAddr = bindings::phys_addr_t;
/// `CONFIG_PHYS_ADDR_T_64BIT`, and it can be a u64 even on 32-bit architectures.
pub type ResourceSize = bindings::resource_size_t;
+/// Untyped I/O region.
+///
+/// This type can be used when an I/O region without known type information has a compile-time known
+/// minimum size (and a runtime known actual size).
+///
+/// # Invariants
+///
+/// - Size of the region is at least as large as the `SIZE` generic parameter.
+/// - Size of the region is multiple of 4.
+#[repr(C, align(4))]
+#[derive(FromBytes)]
+pub struct Region<const SIZE: usize = 0> {
+ inner: [u8],
+}
+
+impl<const SIZE: usize> Region<SIZE> {
+ /// Create a raw mutable pointer from given base address and size.
+ ///
+ /// `size` should be at least as large as the minimum size `SIZE`, and `base` and `size` should
+ /// be 4-byte aligned to uphold the type invariant.
+ ///
+ /// Just like other methods on raw pointers, it is not unsafe to create a raw pointer
+ /// that does not uphold the type invariants. However such pointers are not valid.
+ #[inline]
+ pub fn ptr_from_raw_parts_mut(base: *mut u8, size: usize) -> *mut Self {
+ core::ptr::slice_from_raw_parts_mut(base, size) as *mut Region<SIZE>
+ }
+
+ /// Create a raw mutable pointer from given base address and size.
+ ///
+ /// The alignment of `base` is checked, and `size` is checked against the minimum size specified
+ /// via const generics.
+ #[inline]
+ pub fn ptr_try_from_raw_parts_mut(base: *mut u8, size: usize) -> Result<*mut Self> {
+ if size < SIZE || base.align_offset(4) != 0 || !size.is_multiple_of(4) {
+ return Err(EINVAL);
+ }
+
+ Ok(Self::ptr_from_raw_parts_mut(base, size))
+ }
+}
+
+impl<const SIZE: usize> KnownSize for Region<SIZE> {
+ const MIN_SIZE: usize = SIZE;
+ // Alignment of 4 is the most common; different base types can be added once required.
+ const MIN_ALIGN: Alignment = Alignment::new::<4>();
+
+ #[inline(always)]
+ fn size(p: *const Self) -> usize {
+ (p as *const [u8]).len()
+ }
+}
+
+// SAFETY:
+// - Values read from I/O are always treated as initialized.
+// - Per type invariant the size is multiple of 4 and the type is 4-byte aligned, so it is padding
+// free.
+//
+// This cannot be derived as `derive(IntoBytes)` as the padding free property comes from type
+// invariant which the macro does not know.
+unsafe impl<const SIZE: usize> IntoBytes for Region<SIZE> {
+ #[inline]
+ #[allow(unused)] // Rust 1.87+ stops requiring this and will emit unused warnings.
+ fn only_derive_is_allowed_to_implement_this_trait() {}
+}
+
/// Raw representation of an MMIO region.
///
+/// `MmioRaw<T>` is equivalent to `T __iomem *` in C.
+///
/// By itself, the existence of an instance of this structure does not provide any guarantees that
/// the represented MMIO region does exist or is properly mapped.
///
/// Instead, the bus specific MMIO implementation must convert this raw representation into an
/// `Mmio` instance providing the actual memory accessors. Only by the conversion into an `Mmio`
/// structure any guarantees are given.
-pub struct MmioRaw<const SIZE: usize = 0> {
- addr: usize,
- maxsize: usize,
+pub struct MmioRaw<T: ?Sized> {
+ /// Pointer is in I/O address space.
+ ///
+ /// The provenance does not matter, only the address and metadata do.
+ ptr: *mut T,
}
-impl<const SIZE: usize> MmioRaw<SIZE> {
- /// Returns a new `MmioRaw` instance on success, an error otherwise.
- pub fn new(addr: usize, maxsize: usize) -> Result<Self> {
- if maxsize < SIZE {
- return Err(EINVAL);
+impl<T: ?Sized> Copy for MmioRaw<T> {}
+impl<T: ?Sized> Clone for MmioRaw<T> {
+ #[inline]
+ fn clone(&self) -> Self {
+ *self
+ }
+}
+
+// SAFETY: `MmioRaw` is just an address, so is thread-safe.
+unsafe impl<T: ?Sized> Send for MmioRaw<T> {}
+// SAFETY: `MmioRaw` is just an address, so is thread-safe.
+unsafe impl<T: ?Sized> Sync for MmioRaw<T> {}
+
+impl<T> MmioRaw<T> {
+ /// Create a `MmioRaw` from address.
+ #[inline]
+ pub fn new(addr: usize) -> Self {
+ Self {
+ ptr: core::ptr::without_provenance_mut(addr),
}
+ }
+}
- Ok(Self { addr, maxsize })
+impl<const SIZE: usize> MmioRaw<Region<SIZE>> {
+ /// Create a `MmioRaw` representing a I/O region with given size.
+ ///
+ /// The size is checked against the minimum size specified via const generics.
+ #[inline]
+ pub fn new_region(addr: usize, size: usize) -> Result<Self> {
+ Ok(Self {
+ ptr: Region::ptr_try_from_raw_parts_mut(core::ptr::without_provenance_mut(addr), size)?,
+ })
}
+}
+impl<T: ?Sized + KnownSize> MmioRaw<T> {
/// Returns the base address of the MMIO region.
#[inline]
pub fn addr(&self) -> usize {
- self.addr
+ self.ptr.addr()
}
- /// Returns the maximum size of the MMIO region.
+ /// Returns the size of the MMIO region.
#[inline]
- pub fn maxsize(&self) -> usize {
- self.maxsize
+ pub fn size(&self) -> usize {
+ KnownSize::size(self.ptr)
}
}
-/// IO-mapped memory region.
-///
-/// The creator (usually a subsystem / bus such as PCI) is responsible for creating the
-/// mapping, performing an additional region request etc.
-///
-/// # Invariant
-///
-/// `addr` is the start and `maxsize` the length of valid I/O mapped memory region of size
-/// `maxsize`.
-///
-/// # Examples
-///
-/// ```no_run
-/// # #![cfg(CONFIG_HAS_IOMEM)]
-/// use kernel::{
-/// bindings,
-/// ffi::c_void,
-/// io::{
-/// Io,
-/// IoKnownSize,
-/// Mmio,
-/// MmioRaw,
-/// PhysAddr,
-/// },
-/// };
-/// use core::ops::Deref;
-///
-/// // See also `pci::Bar` for a real example.
-/// struct IoMem<const SIZE: usize>(MmioRaw<SIZE>);
-///
-/// impl<const SIZE: usize> IoMem<SIZE> {
-/// /// # Safety
-/// ///
-/// /// [`paddr`, `paddr` + `SIZE`) must be a valid MMIO region that is mappable into the CPUs
-/// /// virtual address space.
-/// unsafe fn new(paddr: usize) -> Result<Self>{
-/// // SAFETY: By the safety requirements of this function [`paddr`, `paddr` + `SIZE`) is
-/// // valid for `ioremap`.
-/// let addr = unsafe { bindings::ioremap(paddr as PhysAddr, SIZE) };
-/// if addr.is_null() {
-/// return Err(ENOMEM);
-/// }
-///
-/// Ok(IoMem(MmioRaw::new(addr as usize, SIZE)?))
-/// }
-/// }
-///
-/// impl<const SIZE: usize> Drop for IoMem<SIZE> {
-/// fn drop(&mut self) {
-/// // SAFETY: `self.0.addr()` is guaranteed to be properly mapped by `Self::new`.
-/// unsafe { bindings::iounmap(self.0.addr() as *mut c_void); };
-/// }
-/// }
-///
-/// impl<const SIZE: usize> Deref for IoMem<SIZE> {
-/// type Target = Mmio<SIZE>;
-///
-/// fn deref(&self) -> &Self::Target {
-/// // SAFETY: The memory range stored in `self` has been properly mapped in `Self::new`.
-/// unsafe { Mmio::from_raw(&self.0) }
-/// }
-/// }
-///
-///# fn no_run() -> Result<(), Error> {
-/// // SAFETY: Invalid usage for example purposes.
-/// let iomem = unsafe { IoMem::<{ core::mem::size_of::<u32>() }>::new(0xBAAAAAAD)? };
-/// iomem.write32(0x42, 0x0);
-/// assert!(iomem.try_write32(0x42, 0x0).is_ok());
-/// assert!(iomem.try_write32(0x42, 0x4).is_err());
-/// # Ok(())
-/// # }
-/// ```
-#[repr(transparent)]
-pub struct Mmio<const SIZE: usize = 0>(MmioRaw<SIZE>);
-
-/// Checks whether an access of type `U` at the given `offset`
+/// Checks whether an access of type `U` at the given `base` and the given `offset`
/// is valid within this region.
+///
+/// The `base` is used for alignment checking only. This can be set to 0 to skip the check.
#[inline]
-const fn offset_valid<U>(offset: usize, size: usize) -> bool {
- let type_size = core::mem::size_of::<U>();
- if let Some(end) = offset.checked_add(type_size) {
- end <= size && offset % type_size == 0
+const fn offset_valid<U>(base: usize, offset: usize, size: usize) -> bool {
+ if let Some(end) = offset.checked_add(size_of::<U>()) {
+ end <= size && (base.wrapping_add(offset) % align_of::<U>() == 0)
} else {
false
}
}
+/// Returns a view for a given `offset`, performing compile-time bound checks.
+// Always inline to optimize out error path of `build_assert`.
+#[inline(always)]
+fn io_view_assert<'a, IO: Io<'a>, U>(
+ this: IO,
+ offset: usize,
+) -> <IO::Backend as IoBackend>::View<'a, U> {
+ // We cannot check alignment with `offset_valid` using `ptr.addr()`. So set 0 for it and
+ // ensure alignment by checking that the alignment of `U` is smaller or equal to the
+ // alignment of `IO::Target`.
+ const_assert!(Alignment::of::<U>().as_usize() <= IO::Target::MIN_ALIGN.as_usize());
+ build_assert!(offset_valid::<U>(0, offset, IO::Target::MIN_SIZE));
+
+ let view = this.as_view();
+ let ptr = IO::Backend::as_ptr(view);
+ let projected_ptr = ptr.cast::<U>().wrapping_byte_add(offset);
+ // SAFETY: `offset_valid` checks for size and alignment and therefore `projected_ptr` is a
+ // valid projection.
+ unsafe { IO::Backend::project_view(view, projected_ptr) }
+}
+
+/// Returns a view for a given `offset`, performing runtime bound checks.
+#[inline]
+fn io_view<'a, IO: Io<'a>, U>(
+ this: IO,
+ offset: usize,
+) -> Result<<IO::Backend as IoBackend>::View<'a, U>> {
+ let view = this.as_view();
+ let ptr = IO::Backend::as_ptr(view);
+
+ if !offset_valid::<U>(ptr.addr(), offset, KnownSize::size(ptr)) {
+ return Err(EINVAL);
+ }
+
+ let projected_ptr = ptr.cast::<U>().wrapping_byte_add(offset);
+ // SAFETY: `offset_valid` checks for size and alignment and therefore `projected_ptr` is a
+ // valid projection.
+ Ok(unsafe { IO::Backend::project_view(view, projected_ptr) })
+}
+
+/// I/O backends.
+///
+/// This is an abstract representation to be implemented by arbitrary I/O
+/// backends (e.g. MMIO, PCI config space, etc.).
+///
+/// The base trait only defines the projection operations; which I/O methods are available depends
+/// on which [`IoCapable<T>`] traits are implemented for the type. For example, for MMIO regions,
+/// all widths (u8, u16, u32, and u64 on 64-bit systems) are typically supported. For PCI
+/// configuration space, u8, u16, and u32 are supported but u64 is not.
+///
+/// This trait is separate from the `Io` trait as multiple different I/O types may share the same
+/// operation.
+pub trait IoBackend {
+ /// View type for this I/O backend.
+ type View<'a, T: ?Sized + KnownSize>: IoBase<'a, Backend = Self, Target = T>;
+
+ /// Convert a `view` to a raw pointer for projection.
+ ///
+ /// The returned pointer is private implementation detail of the backend; it is likely not
+ /// valid. It should not be dereferenced.
+ fn as_ptr<'a, T: ?Sized + KnownSize>(view: Self::View<'a, T>) -> *mut T;
+
+ /// Project `view` to its subregion indicated by `ptr`.
+ ///
+ /// If input `view` is valid, returned view must also be valid.
+ ///
+ /// # Safety
+ ///
+ /// `ptr` must be a projection of `Self::as_ptr(view)`.
+ unsafe fn project_view<'a, T: ?Sized + KnownSize, U: ?Sized + KnownSize>(
+ view: Self::View<'a, T>,
+ ptr: *mut U,
+ ) -> Self::View<'a, U>;
+}
+
/// Trait indicating that an I/O backend supports operations of a certain type and providing an
/// implementation for these operations.
///
@@ -163,20 +269,75 @@ const fn offset_valid<U>(offset: usize, size: usize) -> bool {
/// For example, a PCI configuration space may implement `IoCapable<u8>`, `IoCapable<u16>`,
/// and `IoCapable<u32>`, but not `IoCapable<u64>`, while an MMIO region on a 64-bit
/// system might implement all four.
-pub trait IoCapable<T> {
- /// Performs an I/O read of type `T` at `address` and returns the result.
+pub trait IoCapable<T>: IoBackend {
+ /// Performs an I/O read of type `T` at `view` and returns the result.
+ fn io_read<'a>(view: Self::View<'a, T>) -> T;
+
+ /// Performs an I/O write of `value` at `view`.
+ fn io_write<'a>(view: Self::View<'a, T>, value: T);
+}
+
+/// Trait indicating that an I/O backend supports memory copy operations.
+pub trait IoCopyable: IoBackend {
+ /// Copy contents of `view` to `buffer`.
///
/// # Safety
///
- /// The range `[address..address + size_of::<T>()]` must be within the bounds of `Self`.
- unsafe fn io_read(&self, address: usize) -> T;
+ /// - `buffer` is valid for volatile write for `view.size()` bytes.
+ /// - `buffer` should not overlap with `view`.
+ unsafe fn copy_from_io(view: Self::View<'_, [u8]>, buffer: *mut u8);
- /// Performs an I/O write of `value` at `address`.
+ /// Copy contents from `buffer` to `view`.
///
/// # Safety
///
- /// The range `[address..address + size_of::<T>()]` must be within the bounds of `Self`.
- unsafe fn io_write(&self, value: T, address: usize);
+ /// - `buffer` is valid for volatile read for `view.size()` bytes.
+ /// - `buffer` should not overlap with `view`.
+ unsafe fn copy_to_io(view: Self::View<'_, [u8]>, buffer: *const u8);
+
+ /// Copy from `view` and return the value.
+ #[inline]
+ fn copy_read<T: FromBytes>(view: Self::View<'_, T>) -> T {
+ // Project `self` to `[u8]`.
+ let ptr = Self::as_ptr(view);
+ // SAFETY: This is a identity projection.
+ let slice_view = unsafe {
+ Self::project_view(
+ view,
+ core::ptr::slice_from_raw_parts_mut::<u8>(ptr.cast(), size_of::<T>()),
+ )
+ };
+
+ let mut buf = MaybeUninit::<T>::uninit();
+ // SAFETY:
+ // - `buf.as_mut_ptr()` is valid for write for `size_of::<T>()` bytes.
+ // - `buf` is local so `buf.as_mut_ptr()` cannot overlap with `slice_view`.
+ unsafe { Self::copy_from_io(slice_view, buf.as_mut_ptr().cast()) };
+ // SAFETY: `T: FromBytes` guarantee that all bit patterns are valid.
+ unsafe { buf.assume_init() }
+ }
+
+ /// Copy `value` to `view`.
+ ///
+ /// Destructor of `value` will not be executed, consistent with [`zerocopy::transmute`].
+ #[inline]
+ fn copy_write<T: IntoBytes>(view: Self::View<'_, T>, value: T) {
+ // Project `self` to `[u8]`.
+ let ptr = Self::as_ptr(view);
+ // SAFETY: This is a identity projection.
+ let slice_view = unsafe {
+ Self::project_view(
+ view,
+ core::ptr::slice_from_raw_parts_mut::<u8>(ptr.cast(), size_of::<T>()),
+ )
+ };
+
+ // SAFETY:
+ // - `&raw const value` is valid for read for `size_of::<T>()` bytes.
+ // - `value` is local so `&raw const value` cannot overlap with `slice_view`.
+ unsafe { Self::copy_to_io(slice_view, (&raw const value).cast()) };
+ core::mem::forget(value);
+ }
}
/// Describes a given I/O location: its offset, width, and type to convert the raw value from and
@@ -188,15 +349,16 @@ pub trait IoCapable<T> {
/// (for primitive types like [`u32`]) and typed ones (like those generated by the [`register!`]
/// macro).
///
-/// An `IoLoc<T>` carries three pieces of information:
+/// An `IoLoc<Base, T>` carries the following pieces of information:
///
+/// - The valid `Base` to operate on. For most registers, this should be [`Region`].
/// - The offset to access (returned by [`IoLoc::offset`]),
/// - The width of the access (determined by [`IoLoc::IoType`]),
/// - The type `T` in which the raw data is returned or provided.
///
/// `T` and `IoLoc::IoType` may differ: for instance, a typed register has `T` = the register type
/// with its bitfields, and `IoType` = its backing primitive (e.g. `u32`).
-pub trait IoLoc<T> {
+pub trait IoLoc<Base: ?Sized, T> {
/// Size ([`u8`], [`u16`], etc) of the I/O performed on the returned [`offset`](IoLoc::offset).
type IoType: Into<T> + From<T>;
@@ -204,12 +366,12 @@ pub trait IoLoc<T> {
fn offset(self) -> usize;
}
-/// Implements [`IoLoc<$ty>`] for [`usize`], allowing [`usize`] to be used as a parameter of
-/// [`Io::read`] and [`Io::write`].
+/// Implements [`IoLoc<Region<SIZE>, $ty>`] for [`usize`], allowing [`usize`] to be used as a
+/// parameter of [`Io::read`] and [`Io::write`].
macro_rules! impl_usize_ioloc {
($($ty:ty),*) => {
$(
- impl IoLoc<$ty> for usize {
+ impl<const SIZE: usize> IoLoc<Region<SIZE>, $ty> for usize {
type IoType = $ty;
#[inline(always)]
@@ -227,181 +389,430 @@ impl_usize_ioloc!(u8, u16, u32, u64);
/// Types implementing this trait (e.g. MMIO BARs or PCI config regions)
/// can perform I/O operations on regions of memory.
///
-/// This is an abstract representation to be implemented by arbitrary I/O
-/// backends (e.g. MMIO, PCI config space, etc.).
+/// This trait defines which backend shall be used for I/O operations and provides a method to
+/// convert into [`IoBackend::View`]. Users should use the [`Io`] trait which provides the actual
+/// methods to perform I/O operations.
+///
+/// This should be implemented on cheaply copyable handles, such as references or view types.
+pub trait IoBase<'a>: Copy {
+ /// Type that defines all I/O operations.
+ type Backend: IoBackend;
+
+ /// Type of this I/O region. For untyped regions, [`Region`] can be used.
+ type Target: ?Sized + KnownSize;
+
+ /// Return a view that covers the full region.
+ fn as_view(self) -> <Self::Backend as IoBackend>::View<'a, Self::Target>;
+}
+
+/// Extension trait to provide I/O operation methods to types that implement [`IoBase`].
///
-/// The [`Io`] trait provides:
-/// - Base address and size information
+/// This trait provides:
/// - Helper methods for offset validation and address calculation
/// - Fallible (runtime checked) accessors for different data widths
///
-/// Which I/O methods are available depends on which [`IoCapable<T>`] traits
-/// are implemented for the type.
-///
-/// # Examples
-///
-/// For MMIO regions, all widths (u8, u16, u32, and u64 on 64-bit systems) are typically
-/// supported. For PCI configuration space, u8, u16, and u32 are supported but u64 is not.
-pub trait Io {
- /// Returns the base address of this mapping.
- fn addr(&self) -> usize;
+/// Which I/O methods are available depends on the associated [`IoBackend`] implementation.
+pub trait Io<'a>: IoBase<'a> {
+ /// Returns the size of this I/O region.
+ #[inline]
+ fn size(self) -> usize {
+ KnownSize::size(Self::Backend::as_ptr(self.as_view()))
+ }
+
+ /// Returns the length of the slice in number of elements.
+ #[inline]
+ fn len<T>(self) -> usize
+ where
+ Self: Io<'a, Target = [T]>,
+ {
+ Self::Backend::as_ptr(self.as_view()).len()
+ }
- /// Returns the maximum size of this mapping.
- fn maxsize(&self) -> usize;
+ /// Returns `true` if the slice has a length of 0.
+ #[inline]
+ fn is_empty<T>(self) -> bool
+ where
+ Self: Io<'a, Target = [T]>,
+ {
+ self.len() == 0
+ }
- /// Returns the absolute I/O address for a given `offset`,
- /// performing runtime bound checks.
+ /// Try to convert into a different typed I/O view.
+ ///
+ /// A runtime check is performed to ensure that the target type is of same or smaller size to
+ /// current type, and the current view is properly aligned for the target type. Returns
+ /// `Err(EINVAL)` if the runtime check fails.
+ ///
+ /// # Examples
+ ///
+ /// ```no_run
+ /// use kernel::io::{
+ /// io_project,
+ /// Mmio,
+ /// Io,
+ /// Region,
+ /// };
+ /// #[derive(FromBytes, IntoBytes)]
+ /// #[repr(C)]
+ /// struct MyStruct { field: u32, }
+ ///
+ /// # fn test(mmio: &Mmio<'_, Region>) -> Result {
+ /// // let mmio: Mmio<'_, Region>;
+ /// let whole: Mmio<'_, MyStruct> = mmio.try_cast()?;
+ /// # Ok::<(), Error>(()) }
+ /// ```
#[inline]
- fn io_addr<U>(&self, offset: usize) -> Result<usize> {
- if !offset_valid::<U>(offset, self.maxsize()) {
+ fn try_cast<U>(self) -> Result<<Self::Backend as IoBackend>::View<'a, U>>
+ where
+ Self::Target: FromBytes + IntoBytes,
+ U: FromBytes + IntoBytes,
+ {
+ let view = self.as_view();
+ let ptr = Self::Backend::as_ptr(view);
+
+ if size_of::<U>() > KnownSize::size(ptr) {
+ return Err(EINVAL);
+ }
+
+ if ptr.addr() % align_of::<U>() != 0 {
return Err(EINVAL);
}
- // Probably no need to check, since the safety requirements of `Self::new` guarantee that
- // this can't overflow.
- self.addr().checked_add(offset).ok_or(EINVAL)
+ // SAFETY: We have checked bounds and alignment, so this is a valid projection.
+ Ok(unsafe { Self::Backend::project_view(view, ptr.cast()) })
+ }
+
+ /// Read a value from I/O.
+ ///
+ /// This only works for primitives supported by the I/O backend.
+ ///
+ /// # Examples
+ ///
+ /// ```no_run
+ /// # use kernel::io::*;
+ /// # fn test_read_val(mmio: Mmio<'_, u32>) {
+ /// // let mmio: Mmio<'_, u32>;
+ /// let val: u32 = mmio.read_val();
+ /// # }
+ /// ```
+ #[inline]
+ fn read_val(self) -> Self::Target
+ where
+ Self::Backend: IoCapable<Self::Target>,
+ Self::Target: Sized,
+ {
+ Self::Backend::io_read(self.as_view())
+ }
+
+ /// Write a value to I/O.
+ ///
+ /// This only works for primitives supported by the I/O backend.
+ ///
+ /// # Examples
+ ///
+ /// ```no_run
+ /// # use kernel::io::*;
+ /// # fn test_write_val(mmio: Mmio<'_, u32>) {
+ /// // let mmio: Mmio<'_, u32>;
+ /// mmio.write_val(1u32);
+ /// # }
+ /// ```
+ #[inline]
+ fn write_val(self, value: Self::Target)
+ where
+ Self::Backend: IoCapable<Self::Target>,
+ Self::Target: Sized,
+ {
+ Self::Backend::io_write(self.as_view(), value)
+ }
+
+ /// Copy-read from I/O memory.
+ ///
+ /// This is equivalent to reading from the I/O memory with byte-wise copy, although the actual
+ /// implementation might be more efficient. There is no atomicity guarantee. Note that for some
+ /// backends (e.g. `Mmio`), this can read different value compared to [`read_val`] as
+ /// byte-swapping is not performed.
+ ///
+ /// [`read_val`]: Io::read_val
+ ///
+ /// # Examples
+ ///
+ /// ```no_run
+ /// # use kernel::io::*;
+ /// # fn test_copy_read(mmio: Mmio<'_, [u8; 6]>) {
+ /// // let mmio: Mmio<'_, [u8; 6]>;
+ /// let val: [u8; 6] = mmio.copy_read();
+ /// # }
+ /// ```
+ #[inline]
+ fn copy_read(self) -> Self::Target
+ where
+ Self::Backend: IoCopyable,
+ Self::Target: Sized + FromBytes,
+ {
+ Self::Backend::copy_read(self.as_view())
+ }
+
+ /// Copy-write to I/O memory.
+ ///
+ /// This is equivalent to writing to the I/O memory with byte-wise copy, although the actual
+ /// implementation might be more efficient. There is no atomicity guarantee. Note that for some
+ /// backends (e.g. `Mmio`), this can write different value compared to [`write_val`] as
+ /// byte-swapping is not performed.
+ ///
+ /// [`write_val`]: Io::write_val
+ ///
+ /// # Examples
+ ///
+ /// ```no_run
+ /// # use kernel::io::*;
+ /// # fn test_copy_write(mmio: Mmio<'_, [u8; 6]>) {
+ /// // let mmio: Mmio<'_, [u8; 6]>;
+ /// mmio.copy_write([0xAA, 0xBB, 0xCC, 0xDD, 0xEE, 0xFF]);
+ /// # }
+ /// ```
+ #[inline]
+ fn copy_write(self, value: Self::Target)
+ where
+ Self::Backend: IoCopyable,
+ Self::Target: Sized + IntoBytes,
+ {
+ Self::Backend::copy_write(self.as_view(), value);
+ }
+
+ /// Copy bytes from `data` to I/O memory.
+ ///
+ /// # Panics
+ ///
+ /// This function will panic if the length of `self` differs from the length of `data`, similar
+ /// to [`[u8]::copy_from_slice`].
+ ///
+ /// # Examples
+ ///
+ /// ```no_run
+ /// # use kernel::io::*;
+ /// # fn test_copy_write(mmio: Mmio<'_, [u8]>) {
+ /// // let mmio: Mmio<'_, [u8]>;
+ /// mmio.copy_from_slice(&[0xAA, 0xBB, 0xCC, 0xDD, 0xEE, 0xFF]);
+ /// # }
+ /// ```
+ #[inline]
+ fn copy_from_slice(self, data: &[u8])
+ where
+ Self::Backend: IoCopyable,
+ Self: Io<'a, Target = [u8]>,
+ {
+ assert_eq!(self.len(), data.len());
+
+ // SAFETY: `data.as_ptr()` is valid for read for `self.size()` bytes.
+ unsafe {
+ Self::Backend::copy_to_io(self.as_view(), data.as_ptr());
+ }
+ }
+
+ /// Copy bytes from I/O memory to `data`.
+ ///
+ /// # Panics
+ ///
+ /// This function will panic if the length of `self` differs from the length of `data`, similar
+ /// to [`[u8]::copy_from_slice`].
+ ///
+ /// # Examples
+ ///
+ /// ```no_run
+ /// # use kernel::io::*;
+ /// # fn test_copy_write(mmio: Mmio<'_, [u8]>) {
+ /// // let mmio: Mmio<'_, [u8]>;
+ /// let mut buf = [0; 6];
+ /// mmio.copy_to_slice(&mut buf);
+ /// # }
+ /// ```
+ #[inline]
+ fn copy_to_slice(self, data: &mut [u8])
+ where
+ Self::Backend: IoCopyable,
+ Self: Io<'a, Target = [u8]>,
+ {
+ assert_eq!(self.len(), data.len());
+
+ // SAFETY: `data.as_mut_ptr()` is valid for write for `self.size()` bytes.
+ unsafe {
+ Self::Backend::copy_from_io(self.as_view(), data.as_mut_ptr());
+ }
}
/// Fallible 8-bit read with runtime bounds check.
#[inline(always)]
- fn try_read8(&self, offset: usize) -> Result<u8>
+ fn try_read8(self, offset: usize) -> Result<u8>
where
- Self: IoCapable<u8>,
+ usize: IoLoc<Self::Target, u8, IoType = u8>,
+ Self::Backend: IoCapable<u8>,
{
self.try_read(offset)
}
/// Fallible 16-bit read with runtime bounds check.
#[inline(always)]
- fn try_read16(&self, offset: usize) -> Result<u16>
+ fn try_read16(self, offset: usize) -> Result<u16>
where
- Self: IoCapable<u16>,
+ usize: IoLoc<Self::Target, u16, IoType = u16>,
+ Self::Backend: IoCapable<u16>,
{
self.try_read(offset)
}
/// Fallible 32-bit read with runtime bounds check.
#[inline(always)]
- fn try_read32(&self, offset: usize) -> Result<u32>
+ fn try_read32(self, offset: usize) -> Result<u32>
where
- Self: IoCapable<u32>,
+ usize: IoLoc<Self::Target, u32, IoType = u32>,
+ Self::Backend: IoCapable<u32>,
{
self.try_read(offset)
}
/// Fallible 64-bit read with runtime bounds check.
#[inline(always)]
- fn try_read64(&self, offset: usize) -> Result<u64>
+ fn try_read64(self, offset: usize) -> Result<u64>
where
- Self: IoCapable<u64>,
+ usize: IoLoc<Self::Target, u64, IoType = u64>,
+ Self::Backend: IoCapable<u64>,
{
self.try_read(offset)
}
/// Fallible 8-bit write with runtime bounds check.
#[inline(always)]
- fn try_write8(&self, value: u8, offset: usize) -> Result
+ fn try_write8(self, value: u8, offset: usize) -> Result
where
- Self: IoCapable<u8>,
+ usize: IoLoc<Self::Target, u8, IoType = u8>,
+ Self::Backend: IoCapable<u8>,
{
self.try_write(offset, value)
}
/// Fallible 16-bit write with runtime bounds check.
#[inline(always)]
- fn try_write16(&self, value: u16, offset: usize) -> Result
+ fn try_write16(self, value: u16, offset: usize) -> Result
where
- Self: IoCapable<u16>,
+ usize: IoLoc<Self::Target, u16, IoType = u16>,
+ Self::Backend: IoCapable<u16>,
{
self.try_write(offset, value)
}
/// Fallible 32-bit write with runtime bounds check.
#[inline(always)]
- fn try_write32(&self, value: u32, offset: usize) -> Result
+ fn try_write32(self, value: u32, offset: usize) -> Result
where
- Self: IoCapable<u32>,
+ usize: IoLoc<Self::Target, u32, IoType = u32>,
+ Self::Backend: IoCapable<u32>,
{
self.try_write(offset, value)
}
/// Fallible 64-bit write with runtime bounds check.
#[inline(always)]
- fn try_write64(&self, value: u64, offset: usize) -> Result
+ fn try_write64(self, value: u64, offset: usize) -> Result
where
- Self: IoCapable<u64>,
+ usize: IoLoc<Self::Target, u64, IoType = u64>,
+ Self::Backend: IoCapable<u64>,
{
self.try_write(offset, value)
}
/// Infallible 8-bit read with compile-time bounds check.
+ ///
+ /// `offset` should be constant.
#[inline(always)]
- fn read8(&self, offset: usize) -> u8
+ fn read8(self, offset: usize) -> u8
where
- Self: IoKnownSize + IoCapable<u8>,
+ usize: IoLoc<Self::Target, u8, IoType = u8>,
+ Self::Backend: IoCapable<u8>,
{
self.read(offset)
}
/// Infallible 16-bit read with compile-time bounds check.
+ ///
+ /// `offset` should be constant.
#[inline(always)]
- fn read16(&self, offset: usize) -> u16
+ fn read16(self, offset: usize) -> u16
where
- Self: IoKnownSize + IoCapable<u16>,
+ usize: IoLoc<Self::Target, u16, IoType = u16>,
+ Self::Backend: IoCapable<u16>,
{
self.read(offset)
}
/// Infallible 32-bit read with compile-time bounds check.
+ ///
+ /// `offset` should be constant.
#[inline(always)]
- fn read32(&self, offset: usize) -> u32
+ fn read32(self, offset: usize) -> u32
where
- Self: IoKnownSize + IoCapable<u32>,
+ usize: IoLoc<Self::Target, u32, IoType = u32>,
+ Self::Backend: IoCapable<u32>,
{
self.read(offset)
}
/// Infallible 64-bit read with compile-time bounds check.
+ ///
+ /// `offset` should be constant.
#[inline(always)]
- fn read64(&self, offset: usize) -> u64
+ fn read64(self, offset: usize) -> u64
where
- Self: IoKnownSize + IoCapable<u64>,
+ usize: IoLoc<Self::Target, u64, IoType = u64>,
+ Self::Backend: IoCapable<u64>,
{
self.read(offset)
}
/// Infallible 8-bit write with compile-time bounds check.
+ ///
+ /// `offset` should be constant.
#[inline(always)]
- fn write8(&self, value: u8, offset: usize)
+ fn write8(self, value: u8, offset: usize)
where
- Self: IoKnownSize + IoCapable<u8>,
+ usize: IoLoc<Self::Target, u8, IoType = u8>,
+ Self::Backend: IoCapable<u8>,
{
self.write(offset, value)
}
/// Infallible 16-bit write with compile-time bounds check.
+ ///
+ /// `offset` should be constant.
#[inline(always)]
- fn write16(&self, value: u16, offset: usize)
+ fn write16(self, value: u16, offset: usize)
where
- Self: IoKnownSize + IoCapable<u16>,
+ usize: IoLoc<Self::Target, u16, IoType = u16>,
+ Self::Backend: IoCapable<u16>,
{
self.write(offset, value)
}
/// Infallible 32-bit write with compile-time bounds check.
+ ///
+ /// `offset` should be constant.
#[inline(always)]
- fn write32(&self, value: u32, offset: usize)
+ fn write32(self, value: u32, offset: usize)
where
- Self: IoKnownSize + IoCapable<u32>,
+ usize: IoLoc<Self::Target, u32, IoType = u32>,
+ Self::Backend: IoCapable<u32>,
{
self.write(offset, value)
}
/// Infallible 64-bit write with compile-time bounds check.
+ ///
+ /// `offset` should be constant.
#[inline(always)]
- fn write64(&self, value: u64, offset: usize)
+ fn write64(self, value: u64, offset: usize)
where
- Self: IoKnownSize + IoCapable<u64>,
+ usize: IoLoc<Self::Target, u64, IoType = u64>,
+ Self::Backend: IoCapable<u64>,
{
self.write(offset, value)
}
@@ -416,9 +827,10 @@ pub trait Io {
/// use kernel::io::{
/// Io,
/// Mmio,
+ /// Region,
/// };
///
- /// fn do_reads(io: &Mmio) -> Result {
+ /// fn do_reads(io: Mmio<'_, Region>) -> Result {
/// // 32-bit read from address `0x10`.
/// let v: u32 = io.try_read(0x10)?;
///
@@ -429,15 +841,13 @@ pub trait Io {
/// }
/// ```
#[inline(always)]
- fn try_read<T, L>(&self, location: L) -> Result<T>
+ fn try_read<T, L>(self, location: L) -> Result<T>
where
- L: IoLoc<T>,
- Self: IoCapable<L::IoType>,
+ L: IoLoc<Self::Target, T>,
+ Self::Backend: IoCapable<L::IoType>,
{
- let address = self.io_addr::<L::IoType>(location.offset())?;
-
- // SAFETY: `address` has been validated by `io_addr`.
- Ok(unsafe { self.io_read(address) }.into())
+ let view = io_view::<Self, L::IoType>(self, location.offset())?;
+ Ok(Self::Backend::io_read(view).into())
}
/// Generic fallible write with runtime bounds check.
@@ -450,9 +860,10 @@ pub trait Io {
/// use kernel::io::{
/// Io,
/// Mmio,
+ /// Region,
/// };
///
- /// fn do_writes(io: &Mmio) -> Result {
+ /// fn do_writes(io: Mmio<'_, Region>) -> Result {
/// // 32-bit write of value `1` at address `0x10`.
/// io.try_write(0x10, 1u32)?;
///
@@ -463,17 +874,14 @@ pub trait Io {
/// }
/// ```
#[inline(always)]
- fn try_write<T, L>(&self, location: L, value: T) -> Result
+ fn try_write<T, L>(self, location: L, value: T) -> Result
where
- L: IoLoc<T>,
- Self: IoCapable<L::IoType>,
+ L: IoLoc<Self::Target, T>,
+ Self::Backend: IoCapable<L::IoType>,
{
- let address = self.io_addr::<L::IoType>(location.offset())?;
+ let view = io_view::<Self, L::IoType>(self, location.offset())?;
let io_value = value.into();
-
- // SAFETY: `address` has been validated by `io_addr`.
- unsafe { self.io_write(io_value, address) }
-
+ Self::Backend::io_write(view, io_value);
Ok(())
}
@@ -488,6 +896,7 @@ pub trait Io {
/// register,
/// Io,
/// Mmio,
+ /// Region,
/// };
///
/// register! {
@@ -503,17 +912,17 @@ pub trait Io {
/// }
/// }
///
- /// fn do_write_reg(io: &Mmio) -> Result {
+ /// fn do_write_reg(io: Mmio<'_, Region>) -> Result {
///
/// io.try_write_reg(VERSION::new(1, 0))
/// }
/// ```
#[inline(always)]
- fn try_write_reg<T, L, V>(&self, value: V) -> Result
+ fn try_write_reg<T, L, V>(self, value: V) -> Result
where
- L: IoLoc<T>,
- V: LocatedRegister<Location = L, Value = T>,
- Self: IoCapable<L::IoType>,
+ L: IoLoc<Self::Target, T>,
+ V: LocatedRegister<Self::Target, Location = L, Value = T>,
+ Self::Backend: IoCapable<L::IoType>,
{
let (location, value) = value.into_io_op();
@@ -533,29 +942,27 @@ pub trait Io {
/// use kernel::io::{
/// Io,
/// Mmio,
+ /// Region,
/// };
///
- /// fn do_update(io: &Mmio<0x1000>) -> Result {
+ /// fn do_update(io: Mmio<'_, Region<0x1000>>) -> Result {
/// io.try_update(0x10, |v: u32| {
/// v + 1
/// })
/// }
/// ```
#[inline(always)]
- fn try_update<T, L, F>(&self, location: L, f: F) -> Result
+ fn try_update<T, L, F>(self, location: L, f: F) -> Result
where
- L: IoLoc<T>,
- Self: IoCapable<L::IoType>,
+ L: IoLoc<Self::Target, T>,
+ Self::Backend: IoCapable<L::IoType>,
F: FnOnce(T) -> T,
{
- let address = self.io_addr::<L::IoType>(location.offset())?;
+ let view = io_view::<Self, L::IoType>(self, location.offset())?;
- // SAFETY: `address` has been validated by `io_addr`.
- let value: T = unsafe { self.io_read(address) }.into();
+ let value: T = Self::Backend::io_read(view).into();
let io_value = f(value).into();
-
- // SAFETY: `address` has been validated by `io_addr`.
- unsafe { self.io_write(io_value, address) }
+ Self::Backend::io_write(view, io_value);
Ok(())
}
@@ -570,9 +977,10 @@ pub trait Io {
/// use kernel::io::{
/// Io,
/// Mmio,
+ /// Region,
/// };
///
- /// fn do_reads(io: &Mmio<0x1000>) {
+ /// fn do_reads(io: Mmio<'_, Region<0x1000>>) {
/// // 32-bit read from address `0x10`.
/// let v: u32 = io.read(0x10);
///
@@ -581,15 +989,13 @@ pub trait Io {
/// }
/// ```
#[inline(always)]
- fn read<T, L>(&self, location: L) -> T
+ fn read<T, L>(self, location: L) -> T
where
- L: IoLoc<T>,
- Self: IoKnownSize + IoCapable<L::IoType>,
+ L: IoLoc<Self::Target, T>,
+ Self::Backend: IoCapable<L::IoType>,
{
- let address = self.io_addr_assert::<L::IoType>(location.offset());
-
- // SAFETY: `address` has been validated by `io_addr_assert`.
- unsafe { self.io_read(address) }.into()
+ let view = io_view_assert::<Self, L::IoType>(self, location.offset());
+ Self::Backend::io_read(view).into()
}
/// Generic infallible write with compile-time bounds check.
@@ -602,9 +1008,10 @@ pub trait Io {
/// use kernel::io::{
/// Io,
/// Mmio,
+ /// Region,
/// };
///
- /// fn do_writes(io: &Mmio<0x1000>) {
+ /// fn do_writes(io: Mmio<'_, Region<0x1000>>) {
/// // 32-bit write of value `1` at address `0x10`.
/// io.write(0x10, 1u32);
///
@@ -613,16 +1020,14 @@ pub trait Io {
/// }
/// ```
#[inline(always)]
- fn write<T, L>(&self, location: L, value: T)
+ fn write<T, L>(self, location: L, value: T)
where
- L: IoLoc<T>,
- Self: IoKnownSize + IoCapable<L::IoType>,
+ L: IoLoc<Self::Target, T>,
+ Self::Backend: IoCapable<L::IoType>,
{
- let address = self.io_addr_assert::<L::IoType>(location.offset());
+ let view = io_view_assert::<Self, L::IoType>(self, location.offset());
let io_value = value.into();
-
- // SAFETY: `address` has been validated by `io_addr_assert`.
- unsafe { self.io_write(io_value, address) }
+ Self::Backend::io_write(view, io_value);
}
/// Generic infallible write of a fully-located register value.
@@ -636,6 +1041,7 @@ pub trait Io {
/// register,
/// Io,
/// Mmio,
+ /// Region,
/// };
///
/// register! {
@@ -651,16 +1057,16 @@ pub trait Io {
/// }
/// }
///
- /// fn do_write_reg(io: &Mmio<0x1000>) {
+ /// fn do_write_reg(io: Mmio<'_, Region<0x1000>>) {
/// io.write_reg(VERSION::new(1, 0));
/// }
/// ```
#[inline(always)]
- fn write_reg<T, L, V>(&self, value: V)
+ fn write_reg<T, L, V>(self, value: V)
where
- L: IoLoc<T>,
- V: LocatedRegister<Location = L, Value = T>,
- Self: IoKnownSize + IoCapable<L::IoType>,
+ L: IoLoc<Self::Target, T>,
+ V: LocatedRegister<Self::Target, Location = L, Value = T>,
+ Self::Backend: IoCapable<L::IoType>,
{
let (location, value) = value.into_io_op();
@@ -680,143 +1086,208 @@ pub trait Io {
/// use kernel::io::{
/// Io,
/// Mmio,
+ /// Region,
/// };
///
- /// fn do_update(io: &Mmio<0x1000>) {
+ /// fn do_update(io: Mmio<'_, Region<0x1000>>) {
/// io.update(0x10, |v: u32| {
/// v + 1
/// })
/// }
/// ```
#[inline(always)]
- fn update<T, L, F>(&self, location: L, f: F)
+ fn update<T, L, F>(self, location: L, f: F)
where
- L: IoLoc<T>,
- Self: IoKnownSize + IoCapable<L::IoType> + Sized,
+ L: IoLoc<Self::Target, T>,
+ Self::Backend: IoCapable<L::IoType>,
F: FnOnce(T) -> T,
{
- let address = self.io_addr_assert::<L::IoType>(location.offset());
-
- // SAFETY: `address` has been validated by `io_addr_assert`.
- let value: T = unsafe { self.io_read(address) }.into();
+ let view = io_view_assert::<Self, L::IoType>(self, location.offset());
+ let value: T = Self::Backend::io_read(view).into();
let io_value = f(value).into();
-
- // SAFETY: `address` has been validated by `io_addr_assert`.
- unsafe { self.io_write(io_value, address) }
+ Self::Backend::io_write(view, io_value);
}
}
-/// Trait for types with a known size at compile time.
+// Blanket implementation ensures that provided methods cannot be arbitrarily overridden by
+// implementers, which is relied upon for correctness and soundness.
+impl<'a, T: IoBase<'a>> Io<'a> for T {}
+
+/// A view of memory-mapped I/O region.
///
-/// This trait is implemented by I/O backends that have a compile-time known size,
-/// enabling the use of infallible I/O accessors with compile-time bounds checking.
+/// # Invariant
///
-/// Types implementing this trait can use the infallible methods in [`Io`] trait
-/// (e.g., `read8`, `write32`), which require `Self: IoKnownSize` bound.
-pub trait IoKnownSize: Io {
- /// Minimum usable size of this region.
- const MIN_SIZE: usize;
+/// `ptr` points to a valid and aligned memory-mapped I/O region for the duration lifetime `'a`.
+pub struct Mmio<'a, T: ?Sized> {
+ ptr: *mut T,
+ phantom: PhantomData<&'a ()>,
+}
- /// Returns the absolute I/O address for a given `offset`,
- /// performing compile-time bound checks.
- // Always inline to optimize out error path of `build_assert`.
- #[inline(always)]
- fn io_addr_assert<U>(&self, offset: usize) -> usize {
- build_assert!(offset_valid::<U>(offset, Self::MIN_SIZE));
+impl<T: ?Sized> Copy for Mmio<'_, T> {}
+impl<T: ?Sized> Clone for Mmio<'_, T> {
+ #[inline]
+ fn clone(&self) -> Self {
+ *self
+ }
+}
+
+impl<'a, T: ?Sized> Mmio<'a, T> {
+ /// Create a `Mmio`, providing the accessors to the MMIO mapping.
+ ///
+ /// # Safety
+ ///
+ /// `raw` represents a valid and aligned memory-mapped I/O region while `'a` is alive.
+ #[inline]
+ pub unsafe fn from_raw(raw: MmioRaw<T>) -> Self {
+ // INVARIANT: Per safety requirement.
+ Self {
+ ptr: raw.ptr,
+ phantom: PhantomData,
+ }
+ }
+}
- self.addr() + offset
+// SAFETY: `Mmio<'_, T>` is conceptually `&T` but in I/O memory.
+unsafe impl<T: ?Sized + Sync> Send for Mmio<'_, T> {}
+
+// SAFETY: `Mmio<'_, T>` is conceptually `&T` but in I/O memory.
+unsafe impl<T: ?Sized + Sync> Sync for Mmio<'_, T> {}
+
+impl<'a, T: ?Sized + KnownSize> IoBase<'a> for Mmio<'a, T> {
+ type Backend = MmioBackend;
+ type Target = T;
+
+ #[inline]
+ fn as_view(self) -> Mmio<'a, T> {
+ self
}
}
-/// Implements [`IoCapable`] on `$mmio` for `$ty` using `$read_fn` and `$write_fn`.
+/// I/O Backend for memory-mapped I/O.
+pub struct MmioBackend;
+
+impl IoBackend for MmioBackend {
+ type View<'a, T: ?Sized + KnownSize> = Mmio<'a, T>;
+
+ #[inline]
+ fn as_ptr<'a, T: ?Sized + KnownSize>(view: Self::View<'a, T>) -> *mut T {
+ view.ptr
+ }
+
+ #[inline]
+ unsafe fn project_view<'a, T: ?Sized + KnownSize, U: ?Sized + KnownSize>(
+ _view: Self::View<'a, T>,
+ ptr: *mut U,
+ ) -> Self::View<'a, U> {
+ // INVARIANT: Per safety requirement, `ptr` is projection from `view`, so it is also a valid
+ // memory-mapped I/O region.
+ Mmio {
+ ptr,
+ phantom: PhantomData,
+ }
+ }
+}
+
+/// Implements [`IoCapable`] on `$backend` for `$ty` using `$read_fn` and `$write_fn`.
macro_rules! impl_mmio_io_capable {
- ($mmio:ident, $(#[$attr:meta])* $ty:ty, $read_fn:ident, $write_fn:ident) => {
- $(#[$attr])*
- impl<const SIZE: usize> IoCapable<$ty> for $mmio<SIZE> {
- unsafe fn io_read(&self, address: usize) -> $ty {
- // SAFETY: By the trait invariant `address` is a valid address for MMIO operations.
- unsafe { bindings::$read_fn(address as *const c_void) }
+ ($backend: ident, $ty:ty, $read_fn:ident, $write_fn:ident) => {
+ impl IoCapable<$ty> for $backend {
+ #[inline]
+ fn io_read(view: <$backend as IoBackend>::View<'_, $ty>) -> $ty {
+ // SAFETY: `$backend::as_ptr(view)` is a valid pointer for MMIO operations for both
+ // `MmioBackend` and `RelaxedMmioBackend`.
+ unsafe { bindings::$read_fn($backend::as_ptr(view).cast_const().cast()) }
}
- unsafe fn io_write(&self, value: $ty, address: usize) {
- // SAFETY: By the trait invariant `address` is a valid address for MMIO operations.
- unsafe { bindings::$write_fn(value, address as *mut c_void) }
+ #[inline]
+ fn io_write(view: <$backend as IoBackend>::View<'_, $ty>, value: $ty) {
+ // SAFETY: `$backend::as_ptr(view)` is a valid pointer for MMIO operations for both
+ // `MmioBackend` and `RelaxedMmioBackend`.
+ unsafe { bindings::$write_fn(value, $backend::as_ptr(view).cast()) }
}
}
};
}
// MMIO regions support 8, 16, and 32-bit accesses.
-impl_mmio_io_capable!(Mmio, u8, readb, writeb);
-impl_mmio_io_capable!(Mmio, u16, readw, writew);
-impl_mmio_io_capable!(Mmio, u32, readl, writel);
+impl_mmio_io_capable!(MmioBackend, u8, readb, writeb);
+impl_mmio_io_capable!(MmioBackend, u16, readw, writew);
+impl_mmio_io_capable!(MmioBackend, u32, readl, writel);
// MMIO regions on 64-bit systems also support 64-bit accesses.
-impl_mmio_io_capable!(
- Mmio,
- #[cfg(CONFIG_64BIT)]
- u64,
- readq,
- writeq
-);
+#[cfg(CONFIG_64BIT)]
+impl_mmio_io_capable!(MmioBackend, u64, readq, writeq);
-impl<const SIZE: usize> Io for Mmio<SIZE> {
- /// Returns the base address of this mapping.
+impl IoCopyable for MmioBackend {
#[inline]
- fn addr(&self) -> usize {
- self.0.addr()
+ unsafe fn copy_from_io(view: Self::View<'_, [u8]>, buffer: *mut u8) {
+ // SAFETY:
+ // - `view.ptr` is valid MMIO memory for `view.size()` bytes.
+ // - `buffer` is valid for write for `view.size()` bytes.
+ unsafe {
+ bindings::memcpy_fromio(buffer.cast(), view.ptr.cast(), view.size());
+ }
}
- /// Returns the maximum size of this mapping.
#[inline]
- fn maxsize(&self) -> usize {
- self.0.maxsize()
- }
-}
-
-impl<const SIZE: usize> IoKnownSize for Mmio<SIZE> {
- const MIN_SIZE: usize = SIZE;
-}
-
-impl<const SIZE: usize> Mmio<SIZE> {
- /// Converts an `MmioRaw` into an `Mmio` instance, providing the accessors to the MMIO mapping.
- ///
- /// # Safety
- ///
- /// Callers must ensure that `addr` is the start of a valid I/O mapped memory region of size
- /// `maxsize`.
- pub unsafe fn from_raw(raw: &MmioRaw<SIZE>) -> &Self {
- // SAFETY: `Mmio` is a transparent wrapper around `MmioRaw`.
- unsafe { &*core::ptr::from_ref(raw).cast() }
+ unsafe fn copy_to_io(view: Self::View<'_, [u8]>, buffer: *const u8) {
+ // SAFETY:
+ // - `view.ptr` is valid MMIO memory for `view.size()` bytes.
+ // - `buffer` is valid for read for `view.size()` bytes.
+ unsafe {
+ bindings::memcpy_toio(view.ptr.cast(), buffer.cast(), view.size());
+ }
}
}
-/// [`Mmio`] wrapper using relaxed accessors.
+/// [`Mmio`] but using relaxed accessors.
///
/// This type provides an implementation of [`Io`] that uses relaxed I/O MMIO operands instead of
/// the regular ones.
///
/// See [`Mmio::relaxed`] for a usage example.
-#[repr(transparent)]
-pub struct RelaxedMmio<const SIZE: usize = 0>(Mmio<SIZE>);
+pub struct RelaxedMmio<'a, T: ?Sized>(Mmio<'a, T>);
-impl<const SIZE: usize> Io for RelaxedMmio<SIZE> {
+impl<T: ?Sized> Copy for RelaxedMmio<'_, T> {}
+impl<T: ?Sized> Clone for RelaxedMmio<'_, T> {
#[inline]
- fn addr(&self) -> usize {
- self.0.addr()
+ fn clone(&self) -> Self {
+ *self
}
+}
+
+/// I/O Backend for memory-mapped I/O, with relaxed access semantics.
+pub struct RelaxedMmioBackend;
+
+impl IoBackend for RelaxedMmioBackend {
+ type View<'a, T: ?Sized + KnownSize> = RelaxedMmio<'a, T>;
#[inline]
- fn maxsize(&self) -> usize {
- self.0.maxsize()
+ fn as_ptr<'a, T: ?Sized + KnownSize>(view: Self::View<'a, T>) -> *mut T {
+ MmioBackend::as_ptr(view.0)
+ }
+
+ #[inline]
+ unsafe fn project_view<'a, T: ?Sized + KnownSize, U: ?Sized + KnownSize>(
+ view: Self::View<'a, T>,
+ ptr: *mut U,
+ ) -> Self::View<'a, U> {
+ // SAFETY: Per safety requirement.
+ RelaxedMmio(unsafe { MmioBackend::project_view(view.0, ptr) })
}
}
-impl<const SIZE: usize> IoKnownSize for RelaxedMmio<SIZE> {
- const MIN_SIZE: usize = SIZE;
+impl<'a, T: ?Sized + KnownSize> IoBase<'a> for RelaxedMmio<'a, T> {
+ type Backend = RelaxedMmioBackend;
+ type Target = T;
+
+ #[inline]
+ fn as_view(self) -> RelaxedMmio<'a, T> {
+ self
+ }
}
-impl<const SIZE: usize> Mmio<SIZE> {
- /// Returns a [`RelaxedMmio`] reference that performs relaxed I/O operations.
+impl<'a, T: ?Sized> Mmio<'a, T> {
+ /// Returns a [`RelaxedMmio`] that performs relaxed I/O operations.
///
/// Relaxed accessors do not provide ordering guarantees with respect to DMA or memory accesses
/// and can be used when such ordering is not required.
@@ -827,31 +1298,457 @@ impl<const SIZE: usize> Mmio<SIZE> {
/// use kernel::io::{
/// Io,
/// Mmio,
+ /// Region,
/// RelaxedMmio,
/// };
///
- /// fn do_io(io: &Mmio<0x100>) {
+ /// fn do_io(io: Mmio<'_, Region<0x100>>) {
/// // The access is performed using `readl_relaxed` instead of `readl`.
/// let v = io.relaxed().read32(0x10);
/// }
///
/// ```
- pub fn relaxed(&self) -> &RelaxedMmio<SIZE> {
- // SAFETY: `RelaxedMmio` is `#[repr(transparent)]` over `Mmio`, so `Mmio<SIZE>` and
- // `RelaxedMmio<SIZE>` have identical layout.
- unsafe { core::mem::transmute(self) }
+ #[inline]
+ pub fn relaxed(self) -> RelaxedMmio<'a, T> {
+ RelaxedMmio(self)
}
}
// MMIO regions support 8, 16, and 32-bit accesses.
-impl_mmio_io_capable!(RelaxedMmio, u8, readb_relaxed, writeb_relaxed);
-impl_mmio_io_capable!(RelaxedMmio, u16, readw_relaxed, writew_relaxed);
-impl_mmio_io_capable!(RelaxedMmio, u32, readl_relaxed, writel_relaxed);
+impl_mmio_io_capable!(RelaxedMmioBackend, u8, readb_relaxed, writeb_relaxed);
+impl_mmio_io_capable!(RelaxedMmioBackend, u16, readw_relaxed, writew_relaxed);
+impl_mmio_io_capable!(RelaxedMmioBackend, u32, readl_relaxed, writel_relaxed);
// MMIO regions on 64-bit systems also support 64-bit accesses.
-impl_mmio_io_capable!(
- RelaxedMmio,
- #[cfg(CONFIG_64BIT)]
- u64,
- readq_relaxed,
- writeq_relaxed
-);
+#[cfg(CONFIG_64BIT)]
+impl_mmio_io_capable!(RelaxedMmioBackend, u64, readq_relaxed, writeq_relaxed);
+
+/// I/O Backend for system memory.
+pub struct SysMemBackend;
+
+impl IoBackend for SysMemBackend {
+ type View<'a, T: ?Sized + KnownSize> = SysMem<'a, T>;
+
+ #[inline]
+ fn as_ptr<'a, T: ?Sized + KnownSize>(view: Self::View<'a, T>) -> *mut T {
+ view.ptr
+ }
+
+ #[inline]
+ unsafe fn project_view<'a, T: ?Sized + KnownSize, U: ?Sized + KnownSize>(
+ _view: Self::View<'a, T>,
+ ptr: *mut U,
+ ) -> Self::View<'a, U> {
+ // INVARIANT: Per safety requirement, `ptr` is projection from `view`, so it is also a valid
+ // kernel accessible memory region.
+ SysMem {
+ ptr,
+ phantom: PhantomData,
+ }
+ }
+}
+
+/// Implements [`IoCapable`] on `SysMemBackend` for `$ty` using `read_volatile` and
+/// `write_volatile`.
+macro_rules! impl_sysmem_io_capable {
+ ($ty:ty) => {
+ impl IoCapable<$ty> for SysMemBackend {
+ #[inline]
+ fn io_read(view: SysMem<'_, $ty>) -> $ty {
+ // SAFETY:
+ // - Per type invariant, `ptr` is valid and aligned.
+ // - Using read_volatile() here so that race with hardware is well-defined.
+ // - Using read_volatile() here is not sound if it races with other CPU per Rust
+ // rules, but this is allowed per LKMM.
+ // - The macro is only used on primitives so all bit patterns are valid.
+ unsafe { view.ptr.read_volatile() }
+ }
+
+ #[inline]
+ fn io_write(view: SysMem<'_, $ty>, value: $ty) {
+ // SAFETY:
+ // - Per type invariant, `ptr` is valid and aligned.
+ // - Using write_volatile() here so that race with hardware is well-defined.
+ // - Using write_volatile() here is not sound if it races with other CPU per Rust
+ // rules, but this is allowed per LKMM.
+ unsafe { view.ptr.write_volatile(value) }
+ }
+ }
+ };
+}
+
+impl_sysmem_io_capable!(u8);
+impl_sysmem_io_capable!(u16);
+impl_sysmem_io_capable!(u32);
+#[cfg(CONFIG_64BIT)]
+impl_sysmem_io_capable!(u64);
+
+impl IoCopyable for SysMemBackend {
+ #[inline]
+ unsafe fn copy_from_io(view: Self::View<'_, [u8]>, buffer: *mut u8) {
+ // Use `bindings::memcpy` instead of `copy_nonoverlapping` for volatile.
+ // SAFETY:
+ // - `view.ptr` is in CPU address space and valid for read.
+ // - `buffer` is valid for write for `view.size()` bytes which is equal to `view.ptr.len()`.
+ unsafe { bindings::memcpy(buffer.cast(), view.ptr.cast(), view.ptr.len()) };
+ }
+
+ #[inline]
+ unsafe fn copy_to_io(view: Self::View<'_, [u8]>, buffer: *const u8) {
+ // Use `bindings::memcpy` instead of `copy_nonoverlapping` for volatile.
+ // SAFETY:
+ // - `view.ptr` is in CPU address space and valid for write.
+ // - `buffer` is valid for read for `view.size()` bytes which is equal to `view.ptr.len()`.
+ unsafe { bindings::memcpy(view.ptr.cast(), buffer.cast(), view.ptr.len()) };
+ }
+
+ #[inline]
+ fn copy_read<T: FromBytes>(view: Self::View<'_, T>) -> T {
+ // SAFETY:
+ // - Per type invariant, `ptr` is valid and aligned.
+ // - Using read_volatile() here so that race with hardware is well-defined.
+ // - Using read_volatile() here is not sound if it races with other CPU per Rust
+ // rules, but this is allowed per LKMM.
+ // - `T: FromBytes` so all bit patterns are valid.
+ unsafe { view.ptr.read_volatile() }
+ }
+
+ #[inline]
+ fn copy_write<T: IntoBytes>(view: Self::View<'_, T>, value: T) {
+ // SAFETY:
+ // - Per type invariant, `ptr` is valid and aligned.
+ // - Using write_volatile() here so that race with hardware is well-defined.
+ // - Using write_volatile() here is not sound if it races with other CPU per Rust
+ // rules, but this is allowed per LKMM.
+ unsafe { view.ptr.write_volatile(value) }
+ }
+}
+
+/// A view of a system memory region.
+///
+/// Provides `Io` trait implementation for kernel virtual address ranges,
+/// using volatile read/write to safely access shared memory that may be
+/// concurrently accessed by external hardware.
+///
+/// # Invariants
+///
+/// `self.ptr.addr() .. self.ptr.addr() + KnownSize::size(self.ptr)` is valid and aligned kernel
+/// accessible memory region for the lifetime `'a`.
+pub struct SysMem<'a, T: ?Sized> {
+ ptr: *mut T,
+ phantom: PhantomData<&'a ()>,
+}
+
+impl<T: ?Sized> Copy for SysMem<'_, T> {}
+impl<T: ?Sized> Clone for SysMem<'_, T> {
+ #[inline]
+ fn clone(&self) -> Self {
+ *self
+ }
+}
+
+// SAFETY: `SysMem<'_, T>` is conceptually `&T`.
+unsafe impl<T: ?Sized + Sync> Send for SysMem<'_, T> {}
+
+// SAFETY: `SysMem<'_, T>` is conceptually `&T`.
+unsafe impl<T: ?Sized + Sync> Sync for SysMem<'_, T> {}
+
+impl<'a, T: ?Sized> SysMem<'a, T> {
+ /// Create a `SysMem` from a raw pointer.
+ ///
+ /// # Safety
+ ///
+ /// `ptr.addr() .. ptr.addr() + KnownSize::size(ptr)` must be valid and aligned kernel
+ /// accessible memory region for the lifetime `'a`.
+ #[inline]
+ pub unsafe fn new(ptr: *mut T) -> Self {
+ // INVARIANT: Per safety requirement.
+ Self {
+ ptr,
+ phantom: PhantomData,
+ }
+ }
+
+ /// Obtain the raw pointer to the memory.
+ #[inline]
+ pub fn as_ptr(self) -> *mut T {
+ self.ptr
+ }
+}
+
+impl<'a, T: ?Sized + KnownSize> IoBase<'a> for SysMem<'a, T> {
+ type Backend = SysMemBackend;
+ type Target = T;
+
+ #[inline]
+ fn as_view(self) -> <Self::Backend as IoBackend>::View<'a, Self::Target> {
+ self
+ }
+}
+
+/// I/O Backend for [`IoSysMap`].
+pub struct IoSysMapBackend;
+
+/// Either [`Mmio`] or [`SysMem`].
+///
+/// This can be used when a piece of logic may wish to handle both MMIO or system memory but does
+/// not want or cannot be generic over I/O backends. This serves a similar purpose to
+/// [`include/linux/iosys-map.h`] in C.
+///
+/// This type can be used like any other types that implements [`Io`]; this also include
+/// [`io_project!`], [`io_read!`], [`io_write!`].
+///
+/// [`include/linux/iosys-map.h`]: srctree/include/linux/iosys-map.h
+pub enum IoSysMap<'a, T: ?Sized> {
+ /// The view is I/O memory.
+ Io(Mmio<'a, T>),
+ /// The view is system memory.
+ Sys(SysMem<'a, T>),
+}
+
+impl<T: ?Sized> Copy for IoSysMap<'_, T> {}
+impl<T: ?Sized> Clone for IoSysMap<'_, T> {
+ #[inline]
+ fn clone(&self) -> Self {
+ *self
+ }
+}
+
+impl<'a, T: ?Sized> From<Mmio<'a, T>> for IoSysMap<'a, T> {
+ #[inline]
+ fn from(value: Mmio<'a, T>) -> Self {
+ IoSysMap::Io(value)
+ }
+}
+
+impl<'a, T: ?Sized> From<SysMem<'a, T>> for IoSysMap<'a, T> {
+ #[inline]
+ fn from(value: SysMem<'a, T>) -> Self {
+ IoSysMap::Sys(value)
+ }
+}
+
+impl IoBackend for IoSysMapBackend {
+ type View<'a, T: ?Sized + KnownSize> = IoSysMap<'a, T>;
+
+ #[inline]
+ fn as_ptr<'a, T: ?Sized + KnownSize>(view: Self::View<'a, T>) -> *mut T {
+ match view {
+ IoSysMap::Io(l) => MmioBackend::as_ptr(l),
+ IoSysMap::Sys(r) => SysMemBackend::as_ptr(r),
+ }
+ }
+
+ #[inline]
+ unsafe fn project_view<'a, T: ?Sized + KnownSize, U: ?Sized + KnownSize>(
+ view: Self::View<'a, T>,
+ ptr: *mut U,
+ ) -> Self::View<'a, U> {
+ match view {
+ // SAFETY: Per safety requirement.
+ IoSysMap::Io(l) => IoSysMap::Io(unsafe { MmioBackend::project_view(l, ptr) }),
+ // SAFETY: Per safety requirement.
+ IoSysMap::Sys(r) => IoSysMap::Sys(unsafe { SysMemBackend::project_view(r, ptr) }),
+ }
+ }
+}
+
+impl<T> IoCapable<T> for IoSysMapBackend
+where
+ MmioBackend: IoCapable<T>,
+ SysMemBackend: IoCapable<T>,
+{
+ #[inline]
+ fn io_read(view: Self::View<'_, T>) -> T {
+ match view {
+ IoSysMap::Io(l) => MmioBackend::io_read(l),
+ IoSysMap::Sys(r) => SysMemBackend::io_read(r),
+ }
+ }
+
+ #[inline]
+ fn io_write<'a>(view: Self::View<'a, T>, value: T) {
+ match view {
+ IoSysMap::Io(l) => MmioBackend::io_write(l, value),
+ IoSysMap::Sys(r) => SysMemBackend::io_write(r, value),
+ }
+ }
+}
+
+impl IoCopyable for IoSysMapBackend {
+ #[inline]
+ unsafe fn copy_from_io(view: Self::View<'_, [u8]>, buffer: *mut u8) {
+ match view {
+ // SAFETY: Per safety requirement.
+ IoSysMap::Io(l) => unsafe { MmioBackend::copy_from_io(l, buffer) },
+ // SAFETY: Per safety requirement.
+ IoSysMap::Sys(r) => unsafe { SysMemBackend::copy_from_io(r, buffer) },
+ }
+ }
+
+ #[inline]
+ unsafe fn copy_to_io(view: Self::View<'_, [u8]>, buffer: *const u8) {
+ match view {
+ // SAFETY: Per safety requirement.
+ IoSysMap::Io(l) => unsafe { MmioBackend::copy_to_io(l, buffer) },
+ // SAFETY: Per safety requirement.
+ IoSysMap::Sys(r) => unsafe { SysMemBackend::copy_to_io(r, buffer) },
+ }
+ }
+
+ #[inline]
+ fn copy_read<T: FromBytes>(view: Self::View<'_, T>) -> T {
+ match view {
+ IoSysMap::Io(l) => MmioBackend::copy_read(l),
+ IoSysMap::Sys(r) => SysMemBackend::copy_read(r),
+ }
+ }
+
+ #[inline]
+ fn copy_write<T: IntoBytes>(view: Self::View<'_, T>, value: T) {
+ match view {
+ IoSysMap::Io(l) => MmioBackend::copy_write(l, value),
+ IoSysMap::Sys(r) => SysMemBackend::copy_write(r, value),
+ }
+ }
+}
+
+impl<'a, T: ?Sized + KnownSize> IoBase<'a> for IoSysMap<'a, T> {
+ type Backend = IoSysMapBackend;
+ type Target = T;
+
+ #[inline]
+ fn as_view(self) -> IoSysMap<'a, T> {
+ self
+ }
+}
+
+// This helper turns associated functions to methods so it can be invoked in macro.
+// Used by `io_project!()` only.
+#[doc(hidden)]
+#[derive(Clone, Copy)]
+pub struct ProjectHelper<T>(pub T);
+
+impl<'a, T> ProjectHelper<T>
+where
+ T: Io<'a, Backend: IoBackend<View<'a, T::Target> = T>>,
+{
+ // These helper methods must not have symbols present in the binary to avoid confusion.
+ #[inline(always)]
+ pub fn as_ptr(self) -> *mut T::Target {
+ T::Backend::as_ptr(self.0)
+ }
+
+ /// # Safety
+ ///
+ /// Same as `IoBackend::project_view`
+ #[inline(always)]
+ pub unsafe fn project_view<U: ?Sized + KnownSize>(
+ self,
+ ptr: *mut U,
+ ) -> <T::Backend as IoBackend>::View<'a, U> {
+ // SAFETY: Per safety requirement.
+ unsafe { T::Backend::project_view::<T::Target, _>(self.0, ptr) }
+ }
+}
+
+/// Project an I/O type to a subview of it.
+///
+/// The syntax is of form `io_project!(io, proj)` where `io` is an expression to a type that
+/// implements [`Io`] and `proj` is a [projection specification](kernel::ptr::project!).
+///
+/// # Examples
+///
+/// ```
+/// use kernel::io::{
+/// io_project,
+/// Mmio,
+/// };
+/// #[repr(C)]
+/// struct MyStruct { field: u32, }
+///
+/// # fn test(mmio: Mmio<'_, [MyStruct]>) -> Result {
+/// // let mmio: Mmio<[MyStruct]>;
+/// let field: Mmio<'_, u32> = io_project!(mmio, [try: 1].field);
+/// let whole: Mmio<'_, MyStruct> = io_project!(mmio, [try: 2]);
+/// let nested: Mmio<'_, u32> = io_project!(whole, .field);
+/// # Ok::<(), Error>(()) }
+/// ```
+#[macro_export]
+#[doc(hidden)]
+macro_rules! io_project {
+ ($io:expr, $($proj:tt)*) => {{
+ #[allow(unused)]
+ use $crate::io::IoBase as _;
+ let view = $crate::io::ProjectHelper($io.as_view());
+ let ptr = $crate::ptr::project!(
+ mut view.as_ptr(), $($proj)*
+ );
+ #[allow(unused_unsafe)]
+ // SAFETY: `ptr` is a projection.
+ unsafe { view.project_view(ptr) }
+ }};
+}
+#[doc(inline)]
+pub use crate::io_project;
+
+/// Read from I/O memory.
+///
+/// The syntax is of form `io_read!(io, proj)` where `io` is an expression to a type that
+/// implements [`Io`] and `proj` is a [projection specification](kernel::ptr::project!).
+///
+/// # Examples
+///
+/// ```
+/// #[repr(C)]
+/// struct MyStruct { field: u32, }
+///
+/// # fn test(mmio: kernel::io::Mmio<'_, [MyStruct]>) -> Result {
+/// // let mmio: Mmio<'_, [MyStruct]>;
+/// let field: u32 = kernel::io::io_read!(mmio, [try: 2].field);
+/// # Ok::<(), Error>(()) }
+/// ```
+#[macro_export]
+#[doc(hidden)]
+macro_rules! io_read {
+ ($io:expr, $($proj:tt)*) => {
+ $crate::io::Io::read_val($crate::io_project!($io, $($proj)*))
+ };
+}
+#[doc(inline)]
+pub use crate::io_read;
+
+/// Writes to I/O memory.
+///
+/// The syntax is of form `io_write!(io, proj, val)` where `io` is an expression to a type that
+/// implements [`Io`] and `proj` is a [projection specification](kernel::ptr::project!),
+/// and `val` is the value to be written to the projected location.
+///
+/// # Examples
+///
+/// ```
+/// #[repr(C)]
+/// struct MyStruct { field: u32, }
+///
+/// # fn test(mmio: kernel::io::Mmio<'_, [MyStruct]>) -> Result {
+/// // let mmio: Mmio<'_, [MyStruct]>;
+/// kernel::io::io_write!(mmio, [try: 2].field, 10);
+/// # Ok::<(), Error>(()) }
+/// ```
+#[macro_export]
+#[doc(hidden)]
+macro_rules! io_write {
+ (@parse [$io:expr] [$($proj:tt)*] [, $val:expr]) => {
+ $crate::io::Io::write_val($crate::io_project!($io, $($proj)*), $val)
+ };
+ (@parse [$io:expr] [$($proj:tt)*] [.$field:tt $($rest:tt)*]) => {
+ $crate::io_write!(@parse [$io] [$($proj)* .$field] [$($rest)*])
+ };
+ (@parse [$io:expr] [$($proj:tt)*] [[$flavor:ident: $index:expr] $($rest:tt)*]) => {
+ $crate::io_write!(@parse [$io] [$($proj)* [$flavor: $index]] [$($rest)*])
+ };
+ ($io:expr, $($rest:tt)*) => {
+ $crate::io_write!(@parse [$io] [] [$($rest)*])
+ };
+}
+#[doc(inline)]
+pub use crate::io_write;
diff --git a/rust/kernel/io/mem.rs b/rust/kernel/io/mem.rs
index fc2a3e24f8d5..32a919099dcd 100644
--- a/rust/kernel/io/mem.rs
+++ b/rust/kernel/io/mem.rs
@@ -2,24 +2,28 @@
//! Generic memory-mapped IO.
-use core::ops::Deref;
-
use crate::{
device::{
Bound,
Device, //
},
- devres::Devres,
+ devres::DevresLt,
io::{
self,
resource::{
Region,
Resource, //
},
+ IoBase,
Mmio,
+ MmioBackend,
MmioRaw, //
},
prelude::*,
+ types::{
+ CovariantForLt,
+ ForLt, //
+ },
};
/// An IO request for a specific device and resource.
@@ -172,6 +176,19 @@ pub struct ExclusiveIoMem<'a, const SIZE: usize> {
_region: Region,
}
+impl<const SIZE: usize> ForLt for ExclusiveIoMem<'static, SIZE> {
+ type Of<'a> = ExclusiveIoMem<'a, SIZE>;
+}
+
+// SAFETY: `ExclusiveIoMem<'a, SIZE>` is covariant over `'a`; it holds an `IoMem<'a, SIZE>`,
+// which holds `&'a Device<Bound>`, which is covariant.
+unsafe impl<const SIZE: usize> CovariantForLt for ExclusiveIoMem<'static, SIZE> {}
+
+/// A device-managed exclusive I/O memory region.
+///
+/// See [`ExclusiveIoMem::into_devres`].
+pub type DevresExclusiveIoMem<const SIZE: usize> = DevresLt<ExclusiveIoMem<'static, SIZE>>;
+
impl<'a, const SIZE: usize> ExclusiveIoMem<'a, SIZE> {
/// Creates a new `ExclusiveIoMem` instance.
fn ioremap(dev: &'a Device<Bound>, resource: &Resource) -> Result<Self> {
@@ -198,23 +215,23 @@ impl<'a, const SIZE: usize> ExclusiveIoMem<'a, SIZE> {
/// Consume the `ExclusiveIoMem` and register it as a device-managed resource.
///
- /// The returned `Devres<ExclusiveIoMem<'static, SIZE>>` can outlive the original lifetime
- /// `'a`. Access to the I/O memory is revoked when the device is unbound.
- pub fn into_devres(self) -> Result<Devres<ExclusiveIoMem<'static, SIZE>>> {
- // SAFETY: Casting to `'static` is sound because `Devres` guarantees the
- // `ExclusiveIoMem` does not actually outlive the device -- access is revoked and the
- // resource is released when the device is unbound.
- let iomem: ExclusiveIoMem<'static, SIZE> = unsafe { core::mem::transmute(self) };
- let dev = iomem.iomem.dev;
- Devres::new(dev, iomem)
+ /// The returned [`DevresExclusiveIoMem`] can outlive the original borrow and be stored in
+ /// driver data. Access to the I/O memory is revoked automatically when the device is unbound.
+ pub fn into_devres(self) -> Result<DevresExclusiveIoMem<SIZE>> {
+ let dev = self.iomem.dev;
+ // SAFETY: `ExclusiveIoMem` only holds a device reference and an I/O mapping, both of
+ // which remain valid for the device's full bound scope, not just for `'a`.
+ unsafe { DevresLt::new(dev, self) }
}
}
-impl<const SIZE: usize> Deref for ExclusiveIoMem<'_, SIZE> {
- type Target = Mmio<SIZE>;
+impl<'a, const SIZE: usize> IoBase<'a> for &'a ExclusiveIoMem<'_, SIZE> {
+ type Backend = MmioBackend;
+ type Target = super::Region<SIZE>;
- fn deref(&self) -> &Self::Target {
- &self.iomem
+ #[inline]
+ fn as_view(self) -> Mmio<'a, Self::Target> {
+ self.iomem.as_view()
}
}
@@ -229,9 +246,22 @@ impl<const SIZE: usize> Deref for ExclusiveIoMem<'_, SIZE> {
/// start of the I/O memory mapped region.
pub struct IoMem<'a, const SIZE: usize = 0> {
dev: &'a Device<Bound>,
- io: MmioRaw<SIZE>,
+ io: MmioRaw<super::Region<SIZE>>,
}
+impl<const SIZE: usize> ForLt for IoMem<'static, SIZE> {
+ type Of<'a> = IoMem<'a, SIZE>;
+}
+
+// SAFETY: `IoMem<'a, SIZE>` is covariant over `'a`; it holds `&'a Device<Bound>`,
+// which is covariant.
+unsafe impl<const SIZE: usize> CovariantForLt for IoMem<'static, SIZE> {}
+
+/// A device-managed I/O memory region.
+///
+/// See [`IoMem::into_devres`].
+pub type DevresIoMem<const SIZE: usize = 0> = DevresLt<IoMem<'static, SIZE>>;
+
impl<'a, const SIZE: usize> IoMem<'a, SIZE> {
fn ioremap(dev: &'a Device<Bound>, resource: &Resource) -> Result<Self> {
// Note: Some ioremap() implementations use types that depend on the CPU
@@ -264,23 +294,19 @@ impl<'a, const SIZE: usize> IoMem<'a, SIZE> {
return Err(ENOMEM);
}
- let io = MmioRaw::new(addr as usize, size)?;
-
+ let io = MmioRaw::new_region(addr as usize, size)?;
Ok(IoMem { dev, io })
}
/// Consume the `IoMem` and register it as a device-managed resource.
///
- /// The returned `Devres<IoMem<'static, SIZE>>` can outlive the original
- /// lifetime `'a`. Access to the I/O memory is revoked when the device
- /// is unbound.
- pub fn into_devres(self) -> Result<Devres<IoMem<'static, SIZE>>> {
- // SAFETY: Casting to `'static` is sound because `Devres` guarantees the `IoMem` does not
- // actually outlive the device -- access is revoked and the resource is released when the
- // device is unbound.
- let iomem: IoMem<'static, SIZE> = unsafe { core::mem::transmute(self) };
- let dev = iomem.dev;
- Devres::new(dev, iomem)
+ /// The returned [`DevresIoMem`] can outlive the original borrow and be stored in driver data.
+ /// Access to the I/O memory is revoked automatically when the device is unbound.
+ pub fn into_devres(self) -> Result<DevresIoMem<SIZE>> {
+ let dev = self.dev;
+ // SAFETY: `IoMem` only holds a device reference and an I/O mapping, both of which
+ // remain valid for the device's full bound scope, not just for `'a`.
+ unsafe { DevresLt::new(dev, self) }
}
}
@@ -291,11 +317,13 @@ impl<const SIZE: usize> Drop for IoMem<'_, SIZE> {
}
}
-impl<const SIZE: usize> Deref for IoMem<'_, SIZE> {
- type Target = Mmio<SIZE>;
+impl<'a, const SIZE: usize> IoBase<'a> for &'a IoMem<'_, SIZE> {
+ type Backend = MmioBackend;
+ type Target = super::Region<SIZE>;
- fn deref(&self) -> &Self::Target {
+ #[inline]
+ fn as_view(self) -> Mmio<'a, Self::Target> {
// SAFETY: Safe as by the invariant of `IoMem`.
- unsafe { Mmio::from_raw(&self.io) }
+ unsafe { Mmio::from_raw(self.io) }
}
}
diff --git a/rust/kernel/io/poll.rs b/rust/kernel/io/poll.rs
index 75d1b3e8596c..d75f2fcf46f2 100644
--- a/rust/kernel/io/poll.rs
+++ b/rust/kernel/io/poll.rs
@@ -48,13 +48,14 @@ use crate::{
/// use kernel::io::{
/// Io,
/// Mmio,
+/// Region,
/// poll::read_poll_timeout, //
/// };
/// use kernel::time::Delta;
///
/// const HW_READY: u16 = 0x01;
///
-/// fn wait_for_hardware<const SIZE: usize>(io: &Mmio<SIZE>) -> Result {
+/// fn wait_for_hardware<const SIZE: usize>(io: Mmio<'_, Region<SIZE>>) -> Result {
/// read_poll_timeout(
/// // The `op` closure reads the value of a specific status register.
/// || io.try_read16(0x1000),
@@ -135,13 +136,14 @@ where
/// use kernel::io::{
/// Io,
/// Mmio,
+/// Region,
/// poll::read_poll_timeout_atomic, //
/// };
/// use kernel::time::Delta;
///
/// const HW_READY: u16 = 0x01;
///
-/// fn wait_for_hardware<const SIZE: usize>(io: &Mmio<SIZE>) -> Result {
+/// fn wait_for_hardware<const SIZE: usize>(io: Mmio<'_, Region<SIZE>>) -> Result {
/// read_poll_timeout_atomic(
/// // The `op` closure reads the value of a specific status register.
/// || io.try_read16(0x1000),
diff --git a/rust/kernel/io/register.rs b/rust/kernel/io/register.rs
index f924c7c7c1db..03dfd2ff48c7 100644
--- a/rust/kernel/io/register.rs
+++ b/rust/kernel/io/register.rs
@@ -58,7 +58,7 @@
//! },
//! num::Bounded,
//! };
-//! # use kernel::io::Mmio;
+//! # use kernel::io::{Mmio, Region};
//! # register! {
//! # pub BOOT_0(u32) @ 0x00000100 {
//! # 15:8 vendor_id;
@@ -66,7 +66,7 @@
//! # 3:0 minor_revision;
//! # }
//! # }
-//! # fn test(io: &Mmio<0x1000>) {
+//! # fn test(io: Mmio<'_, Region<0x1000>>) {
//! # fn obtain_vendor_id() -> u8 { 0xff }
//!
//! // Read from the register's defined offset (0x100).
@@ -113,6 +113,8 @@ use crate::{
io::IoLoc, //
};
+use super::Region;
+
/// Trait implemented by all registers.
pub trait Register: Sized {
/// Backing primitive type of the register.
@@ -129,7 +131,7 @@ pub trait FixedRegister: Register {}
/// Allows `()` to be used as the `location` parameter of [`Io::write`](super::Io::write) when
/// passing a [`FixedRegister`] value.
-impl<T> IoLoc<T> for ()
+impl<const SIZE: usize, T> IoLoc<Region<SIZE>, T> for ()
where
T: FixedRegister,
{
@@ -143,7 +145,7 @@ where
/// A [`FixedRegister`] carries its location in its type. Thus `FixedRegister` values can be used
/// as an [`IoLoc`].
-impl<T> IoLoc<T> for T
+impl<const SIZE: usize, T> IoLoc<Region<SIZE>, T> for T
where
T: FixedRegister,
{
@@ -168,7 +170,7 @@ impl<T: FixedRegister> FixedRegisterLoc<T> {
}
}
-impl<T> IoLoc<T> for FixedRegisterLoc<T>
+impl<const SIZE: usize, T> IoLoc<Region<SIZE>, T> for FixedRegisterLoc<T>
where
T: FixedRegister,
{
@@ -239,7 +241,7 @@ where
}
}
-impl<T, B> IoLoc<T> for RelativeRegisterLoc<T, B>
+impl<const SIZE: usize, T, B> IoLoc<Region<SIZE>, T> for RelativeRegisterLoc<T, B>
where
T: RelativeRegister,
B: RegisterBase<T::BaseFamily> + ?Sized,
@@ -283,7 +285,7 @@ impl<T: RegisterArray> RegisterArrayLoc<T> {
}
}
-impl<T> IoLoc<T> for RegisterArrayLoc<T>
+impl<const SIZE: usize, T> IoLoc<Region<SIZE>, T> for RegisterArrayLoc<T>
where
T: RegisterArray,
{
@@ -370,7 +372,7 @@ where
}
}
-impl<T, B> IoLoc<T> for RelativeRegisterArrayLoc<T, B>
+impl<const SIZE: usize, T, B> IoLoc<Region<SIZE>, T> for RelativeRegisterArrayLoc<T, B>
where
T: RelativeRegisterArray,
B: RegisterBase<T::BaseFamily> + ?Sized,
@@ -387,18 +389,18 @@ where
/// which to write it.
///
/// Implementors can be used with [`Io::write_reg`](super::Io::write_reg).
-pub trait LocatedRegister {
+pub trait LocatedRegister<Base: ?Sized> {
/// Register value to write.
type Value: Register;
/// Full location information at which to write the value.
- type Location: IoLoc<Self::Value>;
+ type Location: IoLoc<Base, Self::Value>;
/// Consumes `self` and returns a `(location, value)` tuple describing a valid I/O write
/// operation.
fn into_io_op(self) -> (Self::Location, Self::Value);
}
-impl<T> LocatedRegister for T
+impl<const SIZE: usize, T> LocatedRegister<Region<SIZE>> for T
where
T: FixedRegister,
{
@@ -444,16 +446,16 @@ where
/// Io,
/// },
/// };
-/// # use kernel::io::Mmio;
+/// # use kernel::io::{Mmio, Region};
///
/// register! {
/// FIXED_REG(u32) @ 0x100 {
-/// 16:8 high_byte;
+/// 15:8 high_byte;
/// 7:0 low_byte;
/// }
/// }
///
-/// # fn test(io: &Mmio<0x1000>) {
+/// # fn test(io: Mmio<'_, Region<0x1000>>) {
/// let val = io.read(FIXED_REG);
///
/// // Write from an already-existing value.
@@ -557,7 +559,7 @@ where
/// Io,
/// },
/// };
-/// # use kernel::io::Mmio;
+/// # use kernel::io::{Mmio, Region};
///
/// // Type used to identify the base.
/// pub struct CpuCtlBase;
@@ -582,7 +584,7 @@ where
/// }
/// }
///
-/// # fn test(io: Mmio<0x1000>) {
+/// # fn test(io: Mmio<'_, Region<0x1000>>) {
/// // Read the status of `Cpu0`.
/// let cpu0_started = io.read(CPU_CTL::of::<Cpu0>());
///
@@ -599,7 +601,7 @@ where
/// }
/// }
///
-/// # fn test2(io: Mmio<0x1000>) {
+/// # fn test2(io: Mmio<'_, Region<0x1000>>) {
/// // Start the aliased `CPU0`, leaving its other fields untouched.
/// io.update(CPU_CTL_ALIAS::of::<Cpu0>(), |r| r.with_alias_start(true));
/// # }
@@ -636,7 +638,7 @@ where
/// Io,
/// },
/// };
-/// # use kernel::io::Mmio;
+/// # use kernel::io::{Mmio, Region};
/// # fn get_scratch_idx() -> usize {
/// # 0x15
/// # }
@@ -649,7 +651,7 @@ where
/// }
/// }
///
-/// # fn test(io: &Mmio<0x1000>)
+/// # fn test(io: Mmio<'_, Region<0x1000>>)
/// # -> Result<(), Error>{
/// // Read scratch register 0, i.e. I/O address `0x80`.
/// let scratch_0 = io.read(SCRATCH::at(0)).value();
@@ -722,7 +724,7 @@ where
/// Io,
/// },
/// };
-/// # use kernel::io::Mmio;
+/// # use kernel::io::{Mmio, Region};
/// # fn get_scratch_idx() -> usize {
/// # 0x15
/// # }
@@ -750,7 +752,7 @@ where
/// }
/// }
///
-/// # fn test(io: &Mmio<0x1000>) -> Result<(), Error> {
+/// # fn test(io: Mmio<'_, Region<0x1000>>) -> Result<(), Error> {
/// // Read scratch register 0 of CPU0.
/// let scratch = io.read(CPU_SCRATCH::of::<Cpu0>().at(0));
///
@@ -792,7 +794,7 @@ where
/// }
/// }
///
-/// # fn test2(io: &Mmio<0x1000>) -> Result<(), Error> {
+/// # fn test2(io: Mmio<'_, Region<0x1000>>) -> Result<(), Error> {
/// let cpu0_status = io.read(CPU_FIRMWARE_STATUS::of::<Cpu0>()).status();
/// # Ok(())
/// # }
@@ -802,6 +804,9 @@ macro_rules! register {
// Entry point for the macro, allowing multiple registers to be defined in one call.
// It matches all possible register declaration patterns to dispatch them to corresponding
// `@reg` rule that defines a single register.
+ //
+ // TODO: change `alias:ident` to `alias:path` once relative registers are replaced by I/O
+ // projections.
(
$(
$(#[$attr:meta])* $vis:vis $name:ident ($storage:ty)
@@ -830,12 +835,12 @@ macro_rules! register {
) => {
$crate::register!(@bitfield $(#[$attr])* $vis struct $name($storage) { $($fields)* });
$crate::register!(@io_base $name($storage) @ $offset);
- $crate::register!(@io_fixed $(#[$attr])* $vis $name($storage));
+ $crate::register!(@io_fixed $(#[$attr])* $vis $name);
};
// Creates an alias register of fixed offset register `alias` with its own fields.
(
- @reg $(#[$attr:meta])* $vis:vis $name:ident ($storage:ty) => $alias:ident
+ @reg $(#[$attr:meta])* $vis:vis $name:ident ($storage:ty) => $alias:path
{ $($fields:tt)* }
) => {
$crate::register!(@bitfield $(#[$attr])* $vis struct $name($storage) { $($fields)* });
@@ -843,7 +848,7 @@ macro_rules! register {
@io_base $name($storage) @
<$alias as $crate::io::register::Register>::OFFSET
);
- $crate::register!(@io_fixed $(#[$attr])* $vis $name($storage));
+ $crate::register!(@io_fixed $(#[$attr])* $vis $name);
};
// Creates a register at a relative offset from a base address provider.
@@ -853,7 +858,7 @@ macro_rules! register {
) => {
$crate::register!(@bitfield $(#[$attr])* $vis struct $name($storage) { $($fields)* });
$crate::register!(@io_base $name($storage) @ $offset);
- $crate::register!(@io_relative $vis $name($storage) @ $base);
+ $crate::register!(@io_relative $name @ $base);
};
// Creates an alias register of relative offset register `alias` with its own fields.
@@ -865,7 +870,7 @@ macro_rules! register {
$crate::register!(
@io_base $name($storage) @ <$alias as $crate::io::register::Register>::OFFSET
);
- $crate::register!(@io_relative $vis $name($storage) @ $base);
+ $crate::register!(@io_relative $name @ $base);
};
// Creates an array of registers at a fixed offset of the MMIO space.
@@ -877,7 +882,7 @@ macro_rules! register {
$crate::register!(@bitfield $(#[$attr])* $vis struct $name($storage) { $($fields)* });
$crate::register!(@io_base $name($storage) @ $offset);
- $crate::register!(@io_array $vis $name($storage) [ $size, stride = $stride ]);
+ $crate::register!(@io_array $name [ $size, stride = $stride ]);
};
// Shortcut for contiguous array of registers (stride == size of element).
@@ -886,14 +891,15 @@ macro_rules! register {
{ $($fields:tt)* }
) => {
$crate::register!(
- $(#[$attr])* $vis $name($storage) [ $size, stride = ::core::mem::size_of::<$storage>() ]
+ @reg $(#[$attr])* $vis $name($storage)
+ [ $size, stride = ::core::mem::size_of::<$storage>() ]
@ $offset { $($fields)* }
);
};
// Creates an alias of register `idx` of array of registers `alias` with its own fields.
(
- @reg $(#[$attr:meta])* $vis:vis $name:ident ($storage:ty) => $alias:ident [ $idx:expr ]
+ @reg $(#[$attr:meta])* $vis:vis $name:ident ($storage:ty) => $alias:path [ $idx:expr ]
{ $($fields:tt)* }
) => {
$crate::build_assert::static_assert!(
@@ -906,7 +912,7 @@ macro_rules! register {
<$alias as $crate::io::register::Register>::OFFSET
+ $idx * <$alias as $crate::io::register::RegisterArray>::STRIDE
);
- $crate::register!(@io_fixed $(#[$attr])* $vis $name($storage));
+ $crate::register!(@io_fixed $(#[$attr])* $vis $name);
};
// Creates an array of registers at a relative offset from a base address provider.
@@ -919,9 +925,7 @@ macro_rules! register {
$crate::register!(@bitfield $(#[$attr])* $vis struct $name($storage) { $($fields)* });
$crate::register!(@io_base $name($storage) @ $offset);
- $crate::register!(
- @io_relative_array $vis $name($storage) [ $size, stride = $stride ] @ $base + $offset
- );
+ $crate::register!(@io_relative_array $name [ $size, stride = $stride ] @ $base);
};
// Shortcut for contiguous array of relative registers (stride == size of element).
@@ -930,7 +934,8 @@ macro_rules! register {
@ $base:ident + $offset:literal { $($fields:tt)* }
) => {
$crate::register!(
- $(#[$attr])* $vis $name($storage) [ $size, stride = ::core::mem::size_of::<$storage>() ]
+ @reg $(#[$attr])* $vis $name($storage)
+ [ $size, stride = ::core::mem::size_of::<$storage>() ]
@ $base + $offset { $($fields)* }
);
};
@@ -951,7 +956,7 @@ macro_rules! register {
<$alias as $crate::io::register::Register>::OFFSET +
$idx * <$alias as $crate::io::register::RegisterArray>::STRIDE
);
- $crate::register!(@io_relative $vis $name($storage) @ $base);
+ $crate::register!(@io_relative $name @ $base);
};
// Generates the bitfield for the register.
@@ -977,7 +982,7 @@ macro_rules! register {
};
// Implementations of fixed registers.
- (@io_fixed $(#[$attr:meta])* $vis:vis $name:ident ($storage:ty)) => {
+ (@io_fixed $(#[$attr:meta])* $vis:vis $name:ident) => {
impl $crate::io::register::FixedRegister for $name {}
$(#[$attr])*
@@ -986,7 +991,7 @@ macro_rules! register {
};
// Implementations of relative registers.
- (@io_relative $vis:vis $name:ident ($storage:ty) @ $base:ident) => {
+ (@io_relative $name:ident @ $base:ident) => {
impl $crate::io::register::WithBase for $name {
type BaseFamily = $base;
}
@@ -995,7 +1000,7 @@ macro_rules! register {
};
// Implementations of register arrays.
- (@io_array $vis:vis $name:ident ($storage:ty) [ $size:expr, stride = $stride:expr ]) => {
+ (@io_array $name:ident [ $size:expr, stride = $stride:expr ]) => {
impl $crate::io::register::Array for $name {}
impl $crate::io::register::RegisterArray for $name {
@@ -1006,8 +1011,7 @@ macro_rules! register {
// Implementations of relative array registers.
(
- @io_relative_array $vis:vis $name:ident ($storage:ty) [ $size:expr, stride = $stride:expr ]
- @ $base:ident + $offset:literal
+ @io_relative_array $name:ident [ $size:expr, stride = $stride:expr ] @ $base:ident
) => {
impl $crate::io::register::WithBase for $name {
type BaseFamily = $base;
diff --git a/rust/kernel/irq.rs b/rust/kernel/irq.rs
index 20abd4056655..09ef1e7f853c 100644
--- a/rust/kernel/irq.rs
+++ b/rust/kernel/irq.rs
@@ -8,7 +8,7 @@
//! The current abstractions handle IRQ requests and handlers, i.e.: it allows
//! drivers to register a handler for a given IRQ line.
//!
-//! C header: [`include/linux/device.h`](srctree/include/linux/interrupt.h)
+//! C header: [`include/linux/interrupt.h`](srctree/include/linux/interrupt.h)
/// Flags to be used when registering IRQ handlers.
mod flags;
diff --git a/rust/kernel/irq/request.rs b/rust/kernel/irq/request.rs
index f425fe12f7c8..c1c6525a676a 100644
--- a/rust/kernel/irq/request.rs
+++ b/rust/kernel/irq/request.rs
@@ -5,16 +5,21 @@
//! [`ThreadedRegistration`], which allow users to register handlers for a given
//! IRQ line.
-use core::marker::PhantomPinned;
+use core::marker::{
+ PhantomData,
+ PhantomPinned, //
+};
-use crate::alloc::Allocator;
-use crate::device::{Bound, Device};
-use crate::devres::Devres;
-use crate::error::to_result;
-use crate::irq::flags::Flags;
-use crate::prelude::*;
-use crate::str::CStr;
-use crate::sync::Arc;
+use crate::{
+ device::{
+ Bound,
+ Device, //
+ },
+ error::to_result,
+ irq::flags::Flags,
+ prelude::*,
+ str::CStr,
+};
/// The value that can be returned from a [`Handler`] or a [`ThreadedHandler`].
#[repr(u32)]
@@ -27,7 +32,7 @@ pub enum IrqReturn {
}
/// Callbacks for an IRQ handler.
-pub trait Handler: Sync + 'static {
+pub trait Handler: Sync {
/// The hard IRQ handler.
///
/// This is executed in interrupt context, hence all corresponding
@@ -36,73 +41,20 @@ pub trait Handler: Sync + 'static {
/// All work that does not necessarily need to be executed from
/// interrupt context, should be deferred to a threaded handler.
/// See also [`ThreadedRegistration`].
- fn handle(&self, device: &Device<Bound>) -> IrqReturn;
+ fn handle(&self) -> IrqReturn;
}
-impl<T: ?Sized + Handler + Send> Handler for Arc<T> {
- fn handle(&self, device: &Device<Bound>) -> IrqReturn {
- T::handle(self, device)
- }
-}
-
-impl<T: ?Sized + Handler, A: Allocator + 'static> Handler for Box<T, A> {
- fn handle(&self, device: &Device<Bound>) -> IrqReturn {
- T::handle(self, device)
- }
-}
-
-/// # Invariants
-///
-/// - `self.irq` is the same as the one passed to `request_{threaded}_irq`.
-/// - `cookie` was passed to `request_{threaded}_irq` as the cookie. It is guaranteed to be unique
-/// by the type system, since each call to `new` will return a different instance of
-/// `Registration`.
-#[pin_data(PinnedDrop)]
-struct RegistrationInner {
- irq: u32,
- cookie: *mut c_void,
-}
-
-impl RegistrationInner {
- fn synchronize(&self) {
- // SAFETY: safe as per the invariants of `RegistrationInner`
- unsafe { bindings::synchronize_irq(self.irq) };
- }
-}
-
-#[pinned_drop]
-impl PinnedDrop for RegistrationInner {
- fn drop(self: Pin<&mut Self>) {
- // SAFETY:
- //
- // Safe as per the invariants of `RegistrationInner` and:
- //
- // - The containing struct is `!Unpin` and was initialized using
- // pin-init, so it occupied the same memory location for the entirety of
- // its lifetime.
- //
- // Notice that this will block until all handlers finish executing,
- // i.e.: at no point will &self be invalid while the handler is running.
- unsafe { bindings::free_irq(self.irq, self.cookie) };
- }
-}
-
-// SAFETY: We only use `inner` on drop, which called at most once with no
-// concurrent access.
-unsafe impl Sync for RegistrationInner {}
-
-// SAFETY: It is safe to send `RegistrationInner` across threads.
-unsafe impl Send for RegistrationInner {}
-
/// A request for an IRQ line for a given device.
///
/// # Invariants
///
/// - `ìrq` is the number of an interrupt source of `dev`.
-/// - `irq` has not been registered yet.
+/// - `irq` has not been registered yet; this is consumed by [`Registration::new()`].
pub struct IrqRequest<'a> {
- dev: &'a Device<Bound>,
irq: u32,
+ /// Proves the device is bound at registration time and ties `'a` to the device's bound
+ /// lifetime, ensuring the [`Registration`] cannot outlive it.
+ _dev: PhantomData<&'a Device<Bound>>,
}
impl<'a> IrqRequest<'a> {
@@ -111,12 +63,16 @@ impl<'a> IrqRequest<'a> {
/// # Safety
///
/// - `irq` should be a valid IRQ number for `dev`.
- pub(crate) unsafe fn new(dev: &'a Device<Bound>, irq: u32) -> Self {
+ pub(crate) unsafe fn new(_dev: &'a Device<Bound>, irq: u32) -> Self {
// INVARIANT: `irq` is a valid IRQ number for `dev`.
- IrqRequest { dev, irq }
+ IrqRequest {
+ irq,
+ _dev: PhantomData,
+ }
}
/// Returns the IRQ number of an [`IrqRequest`].
+ #[inline]
pub fn irq(&self) -> u32 {
self.irq
}
@@ -139,10 +95,18 @@ impl<'a> IrqRequest<'a> {
/// [`Completion::wait_for_completion()`]: kernel::sync::Completion::wait_for_completion
///
/// ```
-/// use kernel::device::{Bound, Device};
-/// use kernel::irq::{self, Flags, IrqRequest, IrqReturn, Registration};
-/// use kernel::prelude::*;
-/// use kernel::sync::{Arc, Completion};
+/// use core::pin::Pin;
+/// use kernel::{
+/// irq::{
+/// self,
+/// Flags,
+/// IrqRequest,
+/// IrqReturn,
+/// Registration,
+/// },
+/// prelude::*,
+/// sync::Completion,
+/// };
///
/// // Data shared between process and IRQ context.
/// #[pin_data]
@@ -153,7 +117,7 @@ impl<'a> IrqRequest<'a> {
///
/// impl irq::Handler for Data {
/// // Executed in IRQ context.
-/// fn handle(&self, _dev: &Device<Bound>) -> IrqReturn {
+/// fn handle(&self) -> IrqReturn {
/// self.completion.complete_all();
/// IrqReturn::Handled
/// }
@@ -163,12 +127,21 @@ impl<'a> IrqRequest<'a> {
/// //
/// // This runs in process context and assumes `request` was previously acquired from a device.
/// fn register_irq(
-/// handler: impl PinInit<Data, Error>,
/// request: IrqRequest<'_>,
-/// ) -> Result<Arc<Registration<Data>>> {
-/// let registration = Registration::new(request, Flags::SHARED, c"my_device", handler);
+/// ) -> Result<Pin<KBox<Registration<'_, Data>>>> {
+/// // SAFETY: The returned Registration is not leaked.
+/// let registration = unsafe {
+/// Registration::new(
+/// request,
+/// Flags::SHARED,
+/// c"my_device",
+/// try_pin_init!(Data {
+/// completion <- Completion::new(),
+/// }? Error),
+/// )
+/// };
///
-/// let registration = Arc::pin_init(registration, GFP_KERNEL)?;
+/// let registration = KBox::pin_init(registration, GFP_KERNEL)?;
///
/// registration.handler().completion.wait_for_completion();
///
@@ -179,11 +152,10 @@ impl<'a> IrqRequest<'a> {
///
/// # Invariants
///
-/// * We own an irq handler whose cookie is a pointer to `Self`.
-#[pin_data]
-pub struct Registration<T: Handler> {
- #[pin]
- inner: Devres<RegistrationInner>,
+/// * We own an irq handler registered via `request_irq` whose cookie is a pointer to `Self`.
+#[pin_data(PinnedDrop)]
+pub struct Registration<'a, T: Handler> {
+ request: IrqRequest<'a>,
#[pin]
handler: T,
@@ -194,44 +166,46 @@ pub struct Registration<T: Handler> {
_pin: PhantomPinned,
}
-impl<T: Handler> Registration<T> {
+impl<'a, T: Handler> Registration<'a, T> {
/// Registers the IRQ handler with the system for the given IRQ number.
- pub fn new<'a>(
+ ///
+ /// # Safety
+ ///
+ /// Callers must not `mem::forget()` the returned [`Registration`] or otherwise prevent its
+ /// [`Drop`] implementation from running.
+ pub unsafe fn new(
request: IrqRequest<'a>,
flags: Flags,
name: &'static CStr,
handler: impl PinInit<T, Error> + 'a,
- ) -> impl PinInit<Self, Error> + 'a {
+ ) -> impl PinInit<Self, Error> + 'a
+ where
+ T: 'a,
+ {
+ // INVARIANT: If initialization completes successfully, we own an IRQ handler registered
+ // via `request_irq` whose cookie is a pointer to `Self`.
try_pin_init!(&this in Self {
handler <- handler,
- inner <- Devres::new(
- request.dev,
- try_pin_init!(RegistrationInner {
- // INVARIANT: `this` is a valid pointer to the `Registration` instance
- cookie: this.as_ptr().cast::<c_void>(),
- irq: {
- // SAFETY:
- // - The callbacks are valid for use with request_irq.
- // - If this succeeds, the slot is guaranteed to be valid until the
- // destructor of Self runs, which will deregister the callbacks
- // before the memory location becomes invalid.
- // - When request_irq is called, everything that handle_irq_callback will
- // touch has already been initialized, so it's safe for the callback to
- // be called immediately.
- to_result(unsafe {
- bindings::request_irq(
- request.irq,
- Some(handle_irq_callback::<T>),
- flags.into_inner(),
- name.as_char_ptr(),
- this.as_ptr().cast::<c_void>(),
- )
- })?;
- request.irq
- }
- })
- ),
+ request,
_pin: PhantomPinned,
+ _: {
+ // SAFETY:
+ // - The callbacks are valid for use with request_irq.
+ // - If this succeeds, the slot is guaranteed to be valid until the destructor of
+ // Self runs, which will deregister the callbacks before the memory location
+ // becomes invalid.
+ // - All fields are already initialized, so it's safe for the callback to be
+ // called immediately.
+ to_result(unsafe {
+ bindings::request_irq(
+ request.irq,
+ Some(handle_irq_callback::<T>),
+ flags.into_inner(),
+ name.as_char_ptr(),
+ this.as_ptr().cast::<c_void>(),
+ )
+ })?;
+ },
})
}
@@ -241,19 +215,25 @@ impl<T: Handler> Registration<T> {
}
/// Wait for pending IRQ handlers on other CPUs.
- ///
- /// This will attempt to access the inner [`Devres`] container.
- pub fn try_synchronize(&self) -> Result {
- let inner = self.inner.try_access().ok_or(ENODEV)?;
- inner.synchronize();
- Ok(())
+ #[inline]
+ pub fn synchronize(&self) {
+ // SAFETY: `self.request.irq` is a valid registered IRQ number (type invariant).
+ unsafe { bindings::synchronize_irq(self.request.irq) };
}
+}
- /// Wait for pending IRQ handlers on other CPUs.
- pub fn synchronize(&self, dev: &Device<Bound>) -> Result {
- let inner = self.inner.access(dev)?;
- inner.synchronize();
- Ok(())
+#[pinned_drop]
+impl<T: Handler> PinnedDrop for Registration<'_, T> {
+ fn drop(self: Pin<&mut Self>) {
+ // SAFETY: The cookie was set to a pointer to `Self` in `Registration::new()`. This blocks
+ // until all in-flight handlers complete, so no references to `self` remain after this
+ // returns.
+ unsafe {
+ bindings::free_irq(
+ self.request.irq,
+ core::ptr::from_mut::<Self>(self.get_unchecked_mut()).cast::<c_void>(),
+ )
+ };
}
}
@@ -261,13 +241,11 @@ impl<T: Handler> Registration<T> {
///
/// This function should be only used as the callback in `request_irq`.
unsafe extern "C" fn handle_irq_callback<T: Handler>(_irq: i32, ptr: *mut c_void) -> c_uint {
- // SAFETY: `ptr` is a pointer to `Registration<T>` set in `Registration::new`
- let registration = unsafe { &*(ptr as *const Registration<T>) };
- // SAFETY: The irq callback is removed before the device is unbound, so the fact that the irq
- // callback is running implies that the device has not yet been unbound.
- let device = unsafe { registration.inner.device().as_bound() };
+ let ptr = ptr.cast_const().cast::<Registration<'_, T>>();
+ // SAFETY: `ptr` is a pointer to `Registration<'_, T>` set in `Registration::new()`.
+ let registration = unsafe { &*ptr };
- T::handle(&registration.handler, device) as c_uint
+ T::handle(&registration.handler) as c_uint
}
/// The value that can be returned from [`ThreadedHandler::handle`].
@@ -284,7 +262,7 @@ pub enum ThreadedIrqReturn {
}
/// Callbacks for a threaded IRQ handler.
-pub trait ThreadedHandler: Sync + 'static {
+pub trait ThreadedHandler: Sync {
/// The hard IRQ handler.
///
/// This is executed in interrupt context, hence all corresponding
@@ -293,8 +271,7 @@ pub trait ThreadedHandler: Sync + 'static {
/// handler, i.e. [`ThreadedHandler::handle_threaded`].
///
/// The default implementation returns [`ThreadedIrqReturn::WakeThread`].
- #[expect(unused_variables)]
- fn handle(&self, device: &Device<Bound>) -> ThreadedIrqReturn {
+ fn handle(&self) -> ThreadedIrqReturn {
ThreadedIrqReturn::WakeThread
}
@@ -302,27 +279,7 @@ pub trait ThreadedHandler: Sync + 'static {
///
/// This is executed in process context. The kernel creates a dedicated
/// `kthread` for this purpose.
- fn handle_threaded(&self, device: &Device<Bound>) -> IrqReturn;
-}
-
-impl<T: ?Sized + ThreadedHandler + Send> ThreadedHandler for Arc<T> {
- fn handle(&self, device: &Device<Bound>) -> ThreadedIrqReturn {
- T::handle(self, device)
- }
-
- fn handle_threaded(&self, device: &Device<Bound>) -> IrqReturn {
- T::handle_threaded(self, device)
- }
-}
-
-impl<T: ?Sized + ThreadedHandler, A: Allocator + 'static> ThreadedHandler for Box<T, A> {
- fn handle(&self, device: &Device<Bound>) -> ThreadedIrqReturn {
- T::handle(self, device)
- }
-
- fn handle_threaded(&self, device: &Device<Bound>) -> IrqReturn {
- T::handle_threaded(self, device)
- }
+ fn handle_threaded(&self) -> IrqReturn;
}
/// A registration of a threaded IRQ handler for a given IRQ line.
@@ -339,13 +296,20 @@ impl<T: ?Sized + ThreadedHandler, A: Allocator + 'static> ThreadedHandler for Bo
/// [`Mutex`](kernel::sync::Mutex) to provide interior mutability.
///
/// ```
-/// use kernel::device::{Bound, Device};
-/// use kernel::irq::{
-/// self, Flags, IrqRequest, IrqReturn, ThreadedHandler, ThreadedIrqReturn,
-/// ThreadedRegistration,
+/// use core::pin::Pin;
+/// use kernel::{
+/// irq::{
+/// self,
+/// Flags,
+/// IrqRequest,
+/// IrqReturn,
+/// ThreadedHandler,
+/// ThreadedIrqReturn,
+/// ThreadedRegistration,
+/// },
+/// prelude::*,
+/// sync::Mutex,
/// };
-/// use kernel::prelude::*;
-/// use kernel::sync::{Arc, Mutex};
///
/// // Declare a struct that will be passed in when the interrupt fires. The u32
/// // merely serves as an example of some internal data.
@@ -363,7 +327,7 @@ impl<T: ?Sized + ThreadedHandler, A: Allocator + 'static> ThreadedHandler for Bo
/// // This will run (in a separate kthread) if and only if
/// // [`ThreadedHandler::handle`] returns [`WakeThread`], which it does by
/// // default.
-/// fn handle_threaded(&self, _dev: &Device<Bound>) -> IrqReturn {
+/// fn handle_threaded(&self) -> IrqReturn {
/// let mut data = self.value.lock();
/// *data += 1;
/// IrqReturn::Handled
@@ -375,13 +339,21 @@ impl<T: ?Sized + ThreadedHandler, A: Allocator + 'static> ThreadedHandler for Bo
/// // This is executing in process context and assumes that `request` was
/// // previously acquired from a device.
/// fn register_threaded_irq(
-/// handler: impl PinInit<Data, Error>,
/// request: IrqRequest<'_>,
-/// ) -> Result<Arc<ThreadedRegistration<Data>>> {
-/// let registration =
-/// ThreadedRegistration::new(request, Flags::SHARED, c"my_device", handler);
+/// ) -> Result<Pin<KBox<ThreadedRegistration<'_, Data>>>> {
+/// // SAFETY: The returned Registration is not leaked.
+/// let registration = unsafe {
+/// ThreadedRegistration::new(
+/// request,
+/// Flags::SHARED,
+/// c"my_device",
+/// try_pin_init!(Data {
+/// value <- kernel::new_mutex!(0),
+/// }? Error),
+/// )
+/// };
///
-/// let registration = Arc::pin_init(registration, GFP_KERNEL)?;
+/// let registration = KBox::pin_init(registration, GFP_KERNEL)?;
///
/// {
/// // The data can be accessed from process context too.
@@ -396,11 +368,11 @@ impl<T: ?Sized + ThreadedHandler, A: Allocator + 'static> ThreadedHandler for Bo
///
/// # Invariants
///
-/// * We own an irq handler whose cookie is a pointer to `Self`.
-#[pin_data]
-pub struct ThreadedRegistration<T: ThreadedHandler> {
- #[pin]
- inner: Devres<RegistrationInner>,
+/// * We own an irq handler registered via `request_threaded_irq` whose cookie is a pointer to
+/// `Self`.
+#[pin_data(PinnedDrop)]
+pub struct ThreadedRegistration<'a, T: ThreadedHandler> {
+ request: IrqRequest<'a>,
#[pin]
handler: T,
@@ -411,45 +383,47 @@ pub struct ThreadedRegistration<T: ThreadedHandler> {
_pin: PhantomPinned,
}
-impl<T: ThreadedHandler> ThreadedRegistration<T> {
+impl<'a, T: ThreadedHandler> ThreadedRegistration<'a, T> {
/// Registers the IRQ handler with the system for the given IRQ number.
- pub fn new<'a>(
+ ///
+ /// # Safety
+ ///
+ /// Callers must not `mem::forget()` the returned [`ThreadedRegistration`] or otherwise prevent
+ /// its [`Drop`] implementation from running.
+ pub unsafe fn new(
request: IrqRequest<'a>,
flags: Flags,
name: &'static CStr,
handler: impl PinInit<T, Error> + 'a,
- ) -> impl PinInit<Self, Error> + 'a {
+ ) -> impl PinInit<Self, Error> + 'a
+ where
+ T: 'a,
+ {
+ // INVARIANT: If initialization completes successfully, we own an IRQ handler registered
+ // via `request_threaded_irq` whose cookie is a pointer to `Self`.
try_pin_init!(&this in Self {
handler <- handler,
- inner <- Devres::new(
- request.dev,
- try_pin_init!(RegistrationInner {
- // INVARIANT: `this` is a valid pointer to the `ThreadedRegistration` instance.
- cookie: this.as_ptr().cast::<c_void>(),
- irq: {
- // SAFETY:
- // - The callbacks are valid for use with request_threaded_irq.
- // - If this succeeds, the slot is guaranteed to be valid until the
- // destructor of Self runs, which will deregister the callbacks
- // before the memory location becomes invalid.
- // - When request_threaded_irq is called, everything that the two callbacks
- // will touch has already been initialized, so it's safe for the
- // callbacks to be called immediately.
- to_result(unsafe {
- bindings::request_threaded_irq(
- request.irq,
- Some(handle_threaded_irq_callback::<T>),
- Some(thread_fn_callback::<T>),
- flags.into_inner(),
- name.as_char_ptr(),
- this.as_ptr().cast::<c_void>(),
- )
- })?;
- request.irq
- }
- })
- ),
+ request,
_pin: PhantomPinned,
+ _: {
+ // SAFETY:
+ // - The callbacks are valid for use with request_threaded_irq.
+ // - If this succeeds, the slot is guaranteed to be valid until the destructor of
+ // Self runs, which will deregister the callbacks before the memory location
+ // becomes invalid.
+ // - All fields are already initialized, so it's safe for the callbacks to be
+ // called immediately.
+ to_result(unsafe {
+ bindings::request_threaded_irq(
+ request.irq,
+ Some(handle_threaded_irq_callback::<T>),
+ Some(thread_fn_callback::<T>),
+ flags.into_inner(),
+ name.as_char_ptr(),
+ this.as_ptr().cast::<c_void>(),
+ )
+ })?;
+ },
})
}
@@ -459,19 +433,25 @@ impl<T: ThreadedHandler> ThreadedRegistration<T> {
}
/// Wait for pending IRQ handlers on other CPUs.
- ///
- /// This will attempt to access the inner [`Devres`] container.
- pub fn try_synchronize(&self) -> Result {
- let inner = self.inner.try_access().ok_or(ENODEV)?;
- inner.synchronize();
- Ok(())
+ #[inline]
+ pub fn synchronize(&self) {
+ // SAFETY: `self.request.irq` is a valid registered IRQ number (type invariant).
+ unsafe { bindings::synchronize_irq(self.request.irq) };
}
+}
- /// Wait for pending IRQ handlers on other CPUs.
- pub fn synchronize(&self, dev: &Device<Bound>) -> Result {
- let inner = self.inner.access(dev)?;
- inner.synchronize();
- Ok(())
+#[pinned_drop]
+impl<T: ThreadedHandler> PinnedDrop for ThreadedRegistration<'_, T> {
+ fn drop(self: Pin<&mut Self>) {
+ // SAFETY: The cookie was set to a pointer to `Self` in `ThreadedRegistration::new()`. This
+ // blocks until all in-flight handlers complete, so no references to `self` remain after
+ // this returns.
+ unsafe {
+ bindings::free_irq(
+ self.request.irq,
+ core::ptr::from_mut::<Self>(self.get_unchecked_mut()).cast::<c_void>(),
+ )
+ };
}
}
@@ -482,24 +462,22 @@ unsafe extern "C" fn handle_threaded_irq_callback<T: ThreadedHandler>(
_irq: i32,
ptr: *mut c_void,
) -> c_uint {
- // SAFETY: `ptr` is a pointer to `ThreadedRegistration<T>` set in `ThreadedRegistration::new`
- let registration = unsafe { &*(ptr as *const ThreadedRegistration<T>) };
- // SAFETY: The irq callback is removed before the device is unbound, so the fact that the irq
- // callback is running implies that the device has not yet been unbound.
- let device = unsafe { registration.inner.device().as_bound() };
+ let ptr = ptr.cast_const().cast::<ThreadedRegistration<'_, T>>();
+ // SAFETY: `ptr` is a pointer to `ThreadedRegistration<'_, T>` set in
+ // `ThreadedRegistration::new()`.
+ let registration = unsafe { &*ptr };
- T::handle(&registration.handler, device) as c_uint
+ T::handle(&registration.handler) as c_uint
}
/// # Safety
///
/// This function should be only used as the callback in `request_threaded_irq`.
unsafe extern "C" fn thread_fn_callback<T: ThreadedHandler>(_irq: i32, ptr: *mut c_void) -> c_uint {
- // SAFETY: `ptr` is a pointer to `ThreadedRegistration<T>` set in `ThreadedRegistration::new`
- let registration = unsafe { &*(ptr as *const ThreadedRegistration<T>) };
- // SAFETY: The irq callback is removed before the device is unbound, so the fact that the irq
- // callback is running implies that the device has not yet been unbound.
- let device = unsafe { registration.inner.device().as_bound() };
+ let ptr = ptr.cast_const().cast::<ThreadedRegistration<'_, T>>();
+ // SAFETY: `ptr` is a pointer to `ThreadedRegistration<'_, T>` set in
+ // `ThreadedRegistration::new()`.
+ let registration = unsafe { &*ptr };
- T::handle_threaded(&registration.handler, device) as c_uint
+ T::handle_threaded(&registration.handler) as c_uint
}
diff --git a/rust/kernel/lib.rs b/rust/kernel/lib.rs
index 73fd046ee8ed..abb208518413 100644
--- a/rust/kernel/lib.rs
+++ b/rust/kernel/lib.rs
@@ -16,6 +16,9 @@
// Please see https://github.com/Rust-for-Linux/linux/issues/2 for details on
// the unstable features in use.
//
+// Stable since Rust 1.87.0.
+#![feature(unsigned_is_multiple_of)]
+//
// Stable since Rust 1.89.0.
#![feature(generic_arg_infer)]
//
@@ -121,6 +124,8 @@ pub mod safety;
pub mod scatterlist;
pub mod security;
pub mod seq_file;
+#[cfg(CONFIG_RUST_SERIAL_DEV_BUS_ABSTRACTIONS)]
+pub mod serdev;
pub mod sizes;
#[cfg(CONFIG_SOC_BUS)]
pub mod soc;
diff --git a/rust/kernel/net/phy.rs b/rust/kernel/net/phy.rs
index 8b7036b8fe48..956cda573ddb 100644
--- a/rust/kernel/net/phy.rs
+++ b/rust/kernel/net/phy.rs
@@ -804,62 +804,6 @@ impl DeviceMask {
/// }
/// # }
/// ```
-///
-/// This expands to the following code:
-///
-/// ```ignore
-/// use kernel::net::phy::{self, DeviceId};
-/// use kernel::prelude::*;
-///
-/// struct Module {
-/// _reg: ::kernel::net::phy::Registration,
-/// }
-///
-/// module! {
-/// type: Module,
-/// name: "rust_sample_phy",
-/// authors: ["Rust for Linux Contributors"],
-/// description: "Rust sample PHYs driver",
-/// license: "GPL",
-/// }
-///
-/// struct PhySample;
-///
-/// #[vtable]
-/// impl phy::Driver for PhySample {
-/// const NAME: &'static CStr = c"PhySample";
-/// const PHY_DEVICE_ID: phy::DeviceId = phy::DeviceId::new_with_exact_mask(0x00000001);
-/// }
-///
-/// const _: () = {
-/// static mut DRIVERS: [::kernel::net::phy::DriverVTable; 1] =
-/// [::kernel::net::phy::create_phy_driver::<PhySample>()];
-///
-/// impl ::kernel::Module for Module {
-/// fn init(module: &'static ::kernel::ThisModule) -> Result<Self> {
-/// let drivers = unsafe { &mut DRIVERS };
-/// let mut reg = ::kernel::net::phy::Registration::register(
-/// module,
-/// ::core::pin::Pin::static_mut(drivers),
-/// )?;
-/// Ok(Module { _reg: reg })
-/// }
-/// }
-/// };
-///
-/// const N: usize = 1;
-///
-/// const TABLE: ::kernel::device_id::IdArray<::kernel::net::phy::DeviceId, (), N> =
-/// ::kernel::device_id::IdArray::new_without_index([
-/// ::kernel::net::phy::DeviceId(
-/// ::kernel::bindings::mdio_device_id {
-/// phy_id: 0x00000001,
-/// phy_id_mask: 0xffffffff,
-/// }),
-/// ]);
-///
-/// ::kernel::module_device_table!("mdio", phydev, TABLE);
-/// ```
#[macro_export]
macro_rules! module_phy_driver {
(@replace_expr $_t:tt $sub:expr) => {$sub};
@@ -869,12 +813,10 @@ macro_rules! module_phy_driver {
};
(@device_table [$($dev:expr),+]) => {
- const N: usize = $crate::module_phy_driver!(@count_devices $($dev),+);
-
- const TABLE: $crate::device_id::IdArray<$crate::net::phy::DeviceId, (), N> =
- $crate::device_id::IdArray::new_without_index([ $(($dev,())),+, ]);
-
- $crate::module_device_table!("mdio", phydev, TABLE);
+ $crate::module_device_table!(
+ "mdio", $crate::net::phy::DeviceId,
+ TABLE, @none, [$($dev),+]
+ );
};
(drivers: [$($driver:ident),+ $(,)?], device_table: [$($dev:expr),+ $(,)?], $($f:tt)*) => {
diff --git a/rust/kernel/of.rs b/rust/kernel/of.rs
index 58b20c367f99..d0318f62afd7 100644
--- a/rust/kernel/of.rs
+++ b/rust/kernel/of.rs
@@ -25,10 +25,6 @@ unsafe impl RawDeviceId for DeviceId {
// SAFETY: `DRIVER_DATA_OFFSET` is the offset to the `data` field.
unsafe impl RawDeviceIdIndex for DeviceId {
const DRIVER_DATA_OFFSET: usize = core::mem::offset_of!(bindings::of_device_id, data);
-
- fn index(&self) -> usize {
- self.0.data as usize
- }
}
impl DeviceId {
@@ -53,13 +49,7 @@ impl DeviceId {
/// Create an OF `IdTable` with an "alias" for modpost.
#[macro_export]
macro_rules! of_device_table {
- ($table_name:ident, $module_table_name:ident, $id_info_type: ty, $table_data: expr) => {
- const $table_name: $crate::device_id::IdArray<
- $crate::of::DeviceId,
- $id_info_type,
- { $table_data.len() },
- > = $crate::device_id::IdArray::new($table_data);
-
- $crate::module_device_table!("of", $module_table_name, $table_name);
+ ($($tt:tt)*) => {
+ $crate::module_device_table!("of", $crate::of::DeviceId, $($tt)*);
};
}
diff --git a/rust/kernel/pci.rs b/rust/kernel/pci.rs
index 4def9ca1824c..aee6485dc58e 100644
--- a/rust/kernel/pci.rs
+++ b/rust/kernel/pci.rs
@@ -43,15 +43,16 @@ pub use self::id::{
pub use self::io::{
Bar,
ConfigSpace,
- ConfigSpaceKind,
ConfigSpaceSize,
+ DevresBar,
Extended,
Normal, //
};
pub use self::irq::{
IrqType,
IrqTypes,
- IrqVector, //
+ IrqVector,
+ IrqVectorRegistration, //
};
/// An adapter for the registration of PCI drivers.
@@ -110,7 +111,11 @@ impl<T: Driver> Adapter<T> {
// SAFETY: `DeviceId` is a `#[repr(transparent)]` wrapper of `struct pci_device_id` and
// does not add additional invariants, so it's safe to transmute.
let id = unsafe { &*id.cast::<DeviceId>() };
- let info = T::ID_TABLE.info(id.index());
+
+ // SAFETY: `id` comes from `T::ID_TABLE` which is of type `IdArray<_, T::IdInfo>` or
+ // `pci_device_id_any` which has 0 as driver_data. It can also come from dynamic IDs, which
+ // will ensure that `driver_data` exists in `T::ID_TABLE`.
+ let info = unsafe { id.info_unchecked_opt::<T::IdInfo>() };
from_result(|| {
let data = T::probe(pdev, info);
@@ -233,10 +238,6 @@ unsafe impl RawDeviceId for DeviceId {
// SAFETY: `DRIVER_DATA_OFFSET` is the offset to the `driver_data` field.
unsafe impl RawDeviceIdIndex for DeviceId {
const DRIVER_DATA_OFFSET: usize = core::mem::offset_of!(bindings::pci_device_id, driver_data);
-
- fn index(&self) -> usize {
- self.0.driver_data
- }
}
/// `IdTable` type for PCI.
@@ -245,14 +246,8 @@ pub type IdTable<T> = &'static dyn kernel::device_id::IdTable<DeviceId, T>;
/// Create a PCI `IdTable` with its alias for modpost.
#[macro_export]
macro_rules! pci_device_table {
- ($table_name:ident, $module_table_name:ident, $id_info_type: ty, $table_data: expr) => {
- const $table_name: $crate::device_id::IdArray<
- $crate::pci::DeviceId,
- $id_info_type,
- { $table_data.len() },
- > = $crate::device_id::IdArray::new($table_data);
-
- $crate::module_device_table!("pci", $module_table_name, $table_name);
+ ($($tt:tt)*) => {
+ $crate::module_device_table!("pci", $crate::pci::DeviceId, $($tt)*);
};
}
@@ -267,7 +262,6 @@ macro_rules! pci_device_table {
///
/// kernel::pci_device_table!(
/// PCI_TABLE,
-/// MODULE_PCI_TABLE,
/// <MyDriver as pci::Driver>::IdInfo,
/// [
/// (
@@ -284,7 +278,7 @@ macro_rules! pci_device_table {
///
/// fn probe<'bound>(
/// _pdev: &'bound pci::Device<Core<'_>>,
-/// _id_info: &'bound Self::IdInfo,
+/// _id_info: Option<&'bound Self::IdInfo>,
/// ) -> impl PinInit<Self::Data<'bound>, Error> + 'bound {
/// Err(ENODEV)
/// }
@@ -313,7 +307,7 @@ pub trait Driver {
/// attempt to initialize the device here.
fn probe<'bound>(
dev: &'bound Device<device::Core<'_>>,
- id_info: &'bound Self::IdInfo,
+ id_info: Option<&'bound Self::IdInfo>,
) -> impl PinInit<Self::Data<'bound>, Error> + 'bound;
/// PCI driver unbind.
@@ -482,11 +476,13 @@ impl<'a> crate::dma::Device<'a> for Device<device::Core<'a>> {}
// SAFETY: Instances of `Device` are always reference-counted.
unsafe impl crate::sync::aref::AlwaysRefCounted for Device {
+ #[inline]
fn inc_ref(&self) {
// SAFETY: The existence of a shared reference guarantees that the refcount is non-zero.
unsafe { bindings::pci_dev_get(self.as_raw()) };
}
+ #[inline]
unsafe fn dec_ref(obj: NonNull<Self>) {
// SAFETY: The safety requirements guarantee that the refcount is non-zero.
unsafe { bindings::pci_dev_put(obj.cast().as_ptr()) }
diff --git a/rust/kernel/pci/io.rs b/rust/kernel/pci/io.rs
index 0461e01aaa20..953e16735c6e 100644
--- a/rust/kernel/pci/io.rs
+++ b/rust/kernel/pci/io.rs
@@ -6,19 +6,22 @@ use super::Device;
use crate::{
bindings,
device,
- devres::Devres,
+ devres::DevresLt,
io::{
- Io,
+ IoBackend,
+ IoBase,
IoCapable,
- IoKnownSize,
Mmio,
- MmioRaw, //
+ MmioBackend,
+ MmioRaw,
+ Region, //
},
- prelude::*, //
-};
-use core::{
- marker::PhantomData,
- ops::Deref, //
+ prelude::*,
+ ptr::KnownSize,
+ types::{
+ CovariantForLt,
+ ForLt, //
+ }, //
};
/// Represents the size of a PCI configuration space.
@@ -46,68 +49,95 @@ impl ConfigSpaceSize {
}
}
-/// Marker type for normal (256-byte) PCI configuration space.
-pub struct Normal;
+/// Alias for normal (256-byte) PCI configuration space.
+pub type Normal = Region<256>;
-/// Marker type for extended (4096-byte) PCIe configuration space.
-pub struct Extended;
+/// Alias for extended (4096-byte) PCIe configuration space.
+pub type Extended = Region<4096>;
-/// Trait for PCI configuration space size markers.
+/// A view of PCI configuration space of a device.
+///
+/// Provides typed read and write accessors for configuration registers
+/// using the standard `pci_read_config_*` and `pci_write_config_*` helpers.
+///
+/// The generic parameter `T` is the type of the view. The full configuration space is also a
+/// special type of view; in such cases, `T` can be [`Normal`] for 256-byte legacy configuration
+/// space or [`Extended`] for 4096-byte PCIe extended configuration space (default).
///
-/// This trait is implemented by [`Normal`] and [`Extended`] to provide
-/// compile-time knowledge of the configuration space size.
-pub trait ConfigSpaceKind {
- /// The size of this configuration space in bytes.
- const SIZE: usize;
+/// # Invariants
+///
+/// `ptr` is aligned and range `ptr..ptr + KnownSize::size(ptr)` is within
+/// `0..pdev.cfg_size().into_raw()`.
+pub struct ConfigSpace<'a, T: ?Sized = Extended> {
+ pub(crate) pdev: &'a Device<device::Bound>,
+ ptr: *mut T,
}
-impl ConfigSpaceKind for Normal {
- const SIZE: usize = 256;
+impl<T: ?Sized> Copy for ConfigSpace<'_, T> {}
+impl<T: ?Sized> Clone for ConfigSpace<'_, T> {
+ #[inline]
+ fn clone(&self) -> Self {
+ *self
+ }
}
-impl ConfigSpaceKind for Extended {
- const SIZE: usize = 4096;
-}
+// SAFETY: `ConfigSpace<'_, T>` is conceptually `&T` but in I/O memory.
+unsafe impl<T: ?Sized + Sync> Send for ConfigSpace<'_, T> {}
-/// The PCI configuration space of a device.
-///
-/// Provides typed read and write accessors for configuration registers
-/// using the standard `pci_read_config_*` and `pci_write_config_*` helpers.
-///
-/// The generic parameter `S` indicates the maximum size of the configuration space.
-/// Use [`Normal`] for 256-byte legacy configuration space or [`Extended`] for
-/// 4096-byte PCIe extended configuration space (default).
-pub struct ConfigSpace<'a, S: ConfigSpaceKind = Extended> {
- pub(crate) pdev: &'a Device<device::Bound>,
- _marker: PhantomData<S>,
+// SAFETY: `ConfigSpace<'_, T>` is conceptually `&T` but in I/O memory.
+unsafe impl<T: ?Sized + Sync> Sync for ConfigSpace<'_, T> {}
+
+/// I/O Backend for PCI configuration space.
+pub struct ConfigSpaceBackend;
+
+impl IoBackend for ConfigSpaceBackend {
+ type View<'a, T: ?Sized + KnownSize> = ConfigSpace<'a, T>;
+
+ #[inline]
+ fn as_ptr<'a, T: ?Sized + KnownSize>(view: ConfigSpace<'a, T>) -> *mut T {
+ view.ptr
+ }
+
+ #[inline]
+ unsafe fn project_view<'a, T: ?Sized + KnownSize, U: ?Sized + KnownSize>(
+ view: Self::View<'a, T>,
+ ptr: *mut U,
+ ) -> Self::View<'a, U> {
+ // INVARIANT: Per safety requirement.
+ ConfigSpace {
+ pdev: view.pdev,
+ ptr,
+ }
+ }
}
/// Implements [`IoCapable`] on [`ConfigSpace`] for `$ty` using `$read_fn` and `$write_fn`.
macro_rules! impl_config_space_io_capable {
($ty:ty, $read_fn:ident, $write_fn:ident) => {
- impl<'a, S: ConfigSpaceKind> IoCapable<$ty> for ConfigSpace<'a, S> {
- unsafe fn io_read(&self, address: usize) -> $ty {
+ impl IoCapable<$ty> for ConfigSpaceBackend {
+ fn io_read(view: ConfigSpace<'_, $ty>) -> $ty {
+ // CAST: The offset is cast to `i32` because the C functions expect a 32-bit
+ // signed offset parameter. PCI configuration space size is at most 4096 bytes,
+ // so the value always fits within `i32` without truncation or sign change.
+ let addr = view.ptr.addr() as i32;
+
let mut val: $ty = 0;
// Return value from C function is ignored in infallible accessors.
- let _ret =
- // SAFETY: By the type invariant `self.pdev` is a valid address.
- // CAST: The offset is cast to `i32` because the C functions expect a 32-bit
- // signed offset parameter. PCI configuration space size is at most 4096 bytes,
- // so the value always fits within `i32` without truncation or sign change.
- unsafe { bindings::$read_fn(self.pdev.as_raw(), address as i32, &mut val) };
-
+ // SAFETY: By the type invariant `pdev` is a valid address.
+ let _ = unsafe { bindings::$read_fn(view.pdev.as_raw(), addr, &mut val) };
val
}
- unsafe fn io_write(&self, value: $ty, address: usize) {
+ fn io_write(view: ConfigSpace<'_, $ty>, value: $ty) {
+ // CAST: The offset is cast to `i32` because the C functions expect a 32-bit
+ // signed offset parameter. PCI configuration space size is at most 4096 bytes,
+ // so the value always fits within `i32` without truncation or sign change.
+ let addr = view.ptr.addr() as i32;
+
// Return value from C function is ignored in infallible accessors.
- let _ret =
- // SAFETY: By the type invariant `self.pdev` is a valid address.
- // CAST: The offset is cast to `i32` because the C functions expect a 32-bit
- // signed offset parameter. PCI configuration space size is at most 4096 bytes,
- // so the value always fits within `i32` without truncation or sign change.
- unsafe { bindings::$write_fn(self.pdev.as_raw(), address as i32, value) };
+ // SAFETY: By the type invariant `pdev` is a valid address.
+ let _ = unsafe { bindings::$write_fn(view.pdev.as_raw(), addr, value) };
}
}
};
@@ -118,24 +148,16 @@ impl_config_space_io_capable!(u8, pci_read_config_byte, pci_write_config_byte);
impl_config_space_io_capable!(u16, pci_read_config_word, pci_write_config_word);
impl_config_space_io_capable!(u32, pci_read_config_dword, pci_write_config_dword);
-impl<'a, S: ConfigSpaceKind> Io for ConfigSpace<'a, S> {
- /// Returns the base address of the I/O region. It is always 0 for configuration space.
- #[inline]
- fn addr(&self) -> usize {
- 0
- }
+impl<'a, T: ?Sized + KnownSize> IoBase<'a> for ConfigSpace<'a, T> {
+ type Backend = ConfigSpaceBackend;
+ type Target = T;
- /// Returns the maximum size of the configuration space.
#[inline]
- fn maxsize(&self) -> usize {
- self.pdev.cfg_size().into_raw()
+ fn as_view(self) -> ConfigSpace<'a, T> {
+ self
}
}
-impl<'a, S: ConfigSpaceKind> IoKnownSize for ConfigSpace<'a, S> {
- const MIN_SIZE: usize = S::SIZE;
-}
-
/// A PCI BAR to perform I/O-Operations on.
///
/// I/O backend assumes that the device is little-endian and will automatically
@@ -147,10 +169,23 @@ impl<'a, S: ConfigSpaceKind> IoKnownSize for ConfigSpace<'a, S> {
/// memory mapped PCI BAR and its size.
pub struct Bar<'a, const SIZE: usize = 0> {
pdev: &'a Device<device::Bound>,
- io: MmioRaw<SIZE>,
+ io: MmioRaw<crate::io::Region<SIZE>>,
num: i32,
}
+impl<const SIZE: usize> ForLt for Bar<'static, SIZE> {
+ type Of<'a> = Bar<'a, SIZE>;
+}
+
+// SAFETY: `Bar<'a, SIZE>` is covariant over `'a`; it holds `&'a Device<Bound>`,
+// which is covariant.
+unsafe impl<const SIZE: usize> CovariantForLt for Bar<'static, SIZE> {}
+
+/// A device-managed PCI BAR mapping.
+///
+/// See [`Bar::into_devres`].
+pub type DevresBar<const SIZE: usize = 0> = DevresLt<Bar<'static, SIZE>>;
+
impl<'a, const SIZE: usize> Bar<'a, SIZE> {
pub(super) fn new(
pdev: &'a Device<device::Bound>,
@@ -187,7 +222,7 @@ impl<'a, const SIZE: usize> Bar<'a, SIZE> {
return Err(ENOMEM);
}
- let io = match MmioRaw::new(ioptr, len as usize) {
+ let io = match MmioRaw::new_region(ioptr, len as usize) {
Ok(io) => io,
Err(err) => {
// SAFETY:
@@ -223,15 +258,13 @@ impl<'a, const SIZE: usize> Bar<'a, SIZE> {
/// Consume the `Bar` and register it as a device-managed resource.
///
- /// The returned `Devres<Bar<'static, SIZE>>` can outlive the original lifetime `'a`. Access
- /// to the BAR is revoked when the device is unbound.
- pub fn into_devres(self) -> Result<Devres<Bar<'static, SIZE>>> {
- // SAFETY: Casting to `'static` is sound because `Devres` guarantees the `Bar` does not
- // actually outlive the device -- access is revoked and the resource is released when the
- // device is unbound.
- let bar: Bar<'static, SIZE> = unsafe { core::mem::transmute(self) };
- let pdev = bar.pdev;
- Devres::new(pdev.as_ref(), bar)
+ /// The returned [`DevresBar`] can outlive the original borrow and be stored in driver data.
+ /// Access to the BAR is revoked automatically when the device is unbound.
+ pub fn into_devres(self) -> Result<DevresBar<SIZE>> {
+ let pdev = self.pdev;
+ // SAFETY: `Bar` only holds a reference to the device and an I/O mapping, both of which
+ // remain valid for the device's full bound scope, not just for `'a`.
+ unsafe { DevresLt::new(pdev.as_ref(), self) }
}
}
@@ -249,12 +282,14 @@ impl<const SIZE: usize> Drop for Bar<'_, SIZE> {
}
}
-impl<const SIZE: usize> Deref for Bar<'_, SIZE> {
- type Target = Mmio<SIZE>;
+impl<'a, const SIZE: usize> IoBase<'a> for &'a Bar<'_, SIZE> {
+ type Backend = MmioBackend;
+ type Target = crate::io::Region<SIZE>;
- fn deref(&self) -> &Self::Target {
+ #[inline]
+ fn as_view(self) -> Mmio<'a, Self::Target> {
// SAFETY: By the type invariant of `Self`, the MMIO range in `self.io` is properly mapped.
- unsafe { Mmio::from_raw(&self.io) }
+ unsafe { Mmio::from_raw(self.io) }
}
}
@@ -289,23 +324,25 @@ impl Device<device::Bound> {
}
}
- /// Return an initialized normal (256-byte) config space object.
+ /// Return a view of the normal (256-byte) config space.
pub fn config_space<'a>(&'a self) -> ConfigSpace<'a, Normal> {
+ // INVARIANT: null is aligned and the range is within config space.
ConfigSpace {
pdev: self,
- _marker: PhantomData,
+ ptr: Normal::ptr_from_raw_parts_mut(core::ptr::null_mut(), self.cfg_size().into_raw()),
}
}
- /// Return an initialized extended (4096-byte) config space object.
+ /// Return a view of the extended (4096-byte) config space.
pub fn config_space_extended<'a>(&'a self) -> Result<ConfigSpace<'a, Extended>> {
if self.cfg_size() != ConfigSpaceSize::Extended {
return Err(EINVAL);
}
+ // INVARIANT: null is aligned and we just checked the `cfg_size`.
Ok(ConfigSpace {
pdev: self,
- _marker: PhantomData,
+ ptr: Extended::ptr_from_raw_parts_mut(core::ptr::null_mut(), 4096),
})
}
}
diff --git a/rust/kernel/pci/irq.rs b/rust/kernel/pci/irq.rs
index d9230e105541..6741046ec1c0 100644
--- a/rust/kernel/pci/irq.rs
+++ b/rust/kernel/pci/irq.rs
@@ -7,17 +7,11 @@ use crate::{
bindings,
device,
device::Bound,
- devres,
error::to_result,
- irq::{
- self,
- IrqRequest, //
- },
- prelude::*,
- str::CStr,
- sync::aref::ARef, //
+ irq::IrqRequest,
+ prelude::*, //
};
-use core::ops::RangeInclusive;
+use core::num::NonZero;
/// IRQ type flags for PCI interrupt allocation.
#[derive(Debug, Clone, Copy)]
@@ -39,6 +33,16 @@ impl IrqType {
IrqType::MsiX => bindings::PCI_IRQ_MSIX,
}
}
+
+ /// Construct from raw value.
+ #[inline]
+ const fn from_raw(raw: u32) -> Self {
+ match raw {
+ bindings::PCI_IRQ_MSIX => IrqType::MsiX,
+ bindings::PCI_IRQ_MSI => IrqType::Msi,
+ _ => IrqType::Intx,
+ }
+ }
}
/// Set of IRQ types that can be used for PCI interrupt allocation.
@@ -71,148 +75,113 @@ impl IrqTypes {
}
}
-/// Represents an allocated IRQ vector for a specific PCI device.
+/// A resolved IRQ vector from a PCI interrupt vector allocation.
///
-/// This type ties an IRQ vector to the device it was allocated for,
-/// ensuring the vector is only used with the correct device.
-#[derive(Clone, Copy)]
+/// Created by [`IrqVectorRegistration::index`]. Convert to [`IrqRequest`] via [`From`] to register
+/// a handler with [`irq::Registration::new`](crate::irq::Registration::new).
pub struct IrqVector<'a> {
- dev: &'a Device<Bound>,
- index: u32,
+ request: IrqRequest<'a>,
+ reg: &'a IrqVectorRegistration<'a>,
}
impl<'a> IrqVector<'a> {
- /// Creates a new [`IrqVector`] for the given device and index.
+ /// Creates a new [`IrqVector`] with an already resolved [`IrqRequest`].
///
/// # Safety
///
- /// - `index` must be a valid IRQ vector index for `dev`.
- /// - `dev` must point to a [`Device`] that has successfully allocated IRQ vectors.
- unsafe fn new(dev: &'a Device<Bound>, index: u32) -> Self {
- Self { dev, index }
+ /// `request` must have been resolved from `reg`.
+ #[inline]
+ unsafe fn new(request: IrqRequest<'a>, reg: &'a IrqVectorRegistration<'a>) -> Self {
+ Self { request, reg }
}
- /// Returns the raw vector index.
- fn index(&self) -> u32 {
- self.index
+ /// Returns the [`IrqVectorRegistration`] this vector was derived from.
+ #[inline]
+ pub fn vectors(&self) -> &'a IrqVectorRegistration<'a> {
+ self.reg
}
-}
-impl<'a> TryInto<IrqRequest<'a>> for IrqVector<'a> {
- type Error = Error;
+ /// Returns the interrupt type the PCI core selected for this vector's allocation.
+ #[inline]
+ pub fn irq_type(&self) -> IrqType {
+ self.reg.irq_type()
+ }
+}
- fn try_into(self) -> Result<IrqRequest<'a>> {
- // SAFETY: `self.as_raw` returns a valid pointer to a `struct pci_dev`.
- let irq = unsafe { bindings::pci_irq_vector(self.dev.as_raw(), self.index()) };
- if irq < 0 {
- return Err(crate::error::Error::from_errno(irq));
- }
- // SAFETY: `irq` is guaranteed to be a valid IRQ number for `&self`.
- Ok(unsafe { IrqRequest::new(self.dev.as_ref(), irq as u32) })
+impl<'a> From<IrqVector<'a>> for IrqRequest<'a> {
+ #[inline]
+ fn from(vector: IrqVector<'a>) -> Self {
+ vector.request
}
}
-/// Represents an IRQ vector allocation for a PCI device.
+/// An allocation of PCI interrupt vectors for a device.
///
-/// This type ensures that IRQ vectors are properly allocated and freed by
-/// tying the allocation to the lifetime of this registration object.
+/// This type owns the vector allocation; dropping it frees the vectors. IRQ handlers borrow from
+/// this registration and must be dropped before it is.
///
/// # Invariants
///
-/// The [`Device`] has successfully allocated IRQ vectors.
-struct IrqVectorRegistration {
- dev: ARef<Device>,
+/// `dev` has an allocation of `len` interrupt vectors.
+pub struct IrqVectorRegistration<'a> {
+ dev: &'a Device<Bound>,
+ len: NonZero<usize>,
}
-impl IrqVectorRegistration {
- /// Allocate and register IRQ vectors for the given PCI device.
+impl<'a> IrqVectorRegistration<'a> {
+ /// Returns the number of allocated vectors.
///
- /// Allocates IRQ vectors and registers them with devres for automatic cleanup.
- /// Returns a range of valid IRQ vectors.
- fn register<'a>(
- dev: &'a Device<Bound>,
- min_vecs: u32,
- max_vecs: u32,
- irq_types: IrqTypes,
- ) -> Result<RangeInclusive<IrqVector<'a>>> {
- // SAFETY:
- // - `dev.as_raw()` is guaranteed to be a valid pointer to a `struct pci_dev`
- // by the type invariant of `Device`.
- // - `pci_alloc_irq_vectors` internally validates all other parameters
- // and returns error codes.
- let ret = unsafe {
- bindings::pci_alloc_irq_vectors(dev.as_raw(), min_vecs, max_vecs, irq_types.as_raw())
- };
-
- to_result(ret)?;
- let count = ret as u32;
+ /// This is at least the `min_vecs` that [`Device::alloc_irq_vectors`] was asked for.
+ #[inline]
+ #[allow(clippy::len_without_is_empty)]
+ pub fn len(&self) -> usize {
+ self.len.get()
+ }
- // SAFETY:
- // - `pci_alloc_irq_vectors` returns the number of allocated vectors on success.
- // - Vectors are 0-based, so valid indices are [0, count-1].
- // - `pci_alloc_irq_vectors` guarantees `count >= min_vecs > 0`, so both `0` and
- // `count - 1` are valid IRQ vector indices for `dev`.
- let range = unsafe { IrqVector::new(dev, 0)..=IrqVector::new(dev, count - 1) };
+ /// Returns the interrupt type the PCI core selected for this allocation.
+ #[inline]
+ pub fn irq_type(&self) -> IrqType {
+ // SAFETY: `self.dev.as_raw()` is a valid pointer to a `struct pci_dev`.
+ IrqType::from_raw(unsafe { bindings::pci_irq_type(self.dev.as_raw()) })
+ }
- // INVARIANT: The IRQ vector allocation for `dev` above was successful.
- let irq_vecs = Self { dev: dev.into() };
- devres::register(dev.as_ref(), irq_vecs, GFP_KERNEL)?;
+ /// Returns the [`IrqVector`] at `index`.
+ ///
+ /// Returns [`EINVAL`] if the `index` is out of bounds for the length reported by
+ /// [`Self::len()`].
+ #[inline]
+ pub fn index(&self, index: usize) -> Result<IrqVector<'_>> {
+ // SAFETY: `self.dev.as_raw()` is a valid pointer to a `struct pci_dev`.
+ let irq = unsafe { bindings::pci_irq_vector(self.dev.as_raw(), index as u32) };
+ if irq < 0 {
+ return Err(Error::from_errno(irq));
+ }
- Ok(range)
+ // SAFETY: `irq` is a valid IRQ number for `self.dev`, resolved from this registration.
+ Ok(unsafe { IrqVector::new(IrqRequest::new(self.dev.as_ref(), irq as u32), self) })
}
}
-impl Drop for IrqVectorRegistration {
+impl Drop for IrqVectorRegistration<'_> {
+ #[inline]
fn drop(&mut self) {
- // SAFETY:
- // - By the type invariant, `self.dev.as_raw()` is a valid pointer to a `struct pci_dev`.
- // - `self.dev` has successfully allocated IRQ vectors.
+ // SAFETY: By the type invariant, `self.dev.as_raw()` is a valid pointer to a
+ // `struct pci_dev` that has successfully allocated IRQ vectors.
unsafe { bindings::pci_free_irq_vectors(self.dev.as_raw()) };
}
}
impl Device<device::Bound> {
- /// Returns a [`kernel::irq::Registration`] for the given IRQ vector.
- pub fn request_irq<'a, T: crate::irq::Handler + 'static>(
- &'a self,
- vector: IrqVector<'a>,
- flags: irq::Flags,
- name: &'static CStr,
- handler: impl PinInit<T, Error> + 'a,
- ) -> impl PinInit<irq::Registration<T>, Error> + 'a {
- pin_init::pin_init_scope(move || {
- let request = vector.try_into()?;
-
- Ok(irq::Registration::<T>::new(request, flags, name, handler))
- })
- }
-
- /// Returns a [`kernel::irq::ThreadedRegistration`] for the given IRQ vector.
- pub fn request_threaded_irq<'a, T: crate::irq::ThreadedHandler + 'static>(
- &'a self,
- vector: IrqVector<'a>,
- flags: irq::Flags,
- name: &'static CStr,
- handler: impl PinInit<T, Error> + 'a,
- ) -> impl PinInit<irq::ThreadedRegistration<T>, Error> + 'a {
- pin_init::pin_init_scope(move || {
- let request = vector.try_into()?;
-
- Ok(irq::ThreadedRegistration::<T>::new(
- request, flags, name, handler,
- ))
- })
- }
-
- /// Allocate IRQ vectors for this PCI device with automatic cleanup.
+ /// Allocate IRQ vectors for this PCI device.
///
/// Allocates between `min_vecs` and `max_vecs` interrupt vectors for the device.
/// The allocation will use MSI-X, MSI, or INTx interrupts based on the `irq_types`
/// parameter and hardware capabilities. When multiple types are specified, the kernel
/// will try them in order of preference: MSI-X first, then MSI, then INTx interrupts.
///
- /// The allocated vectors are automatically freed when the device is unbound, using the
- /// devres (device resource management) system.
+ /// The allocated vectors are freed when the returned [`IrqVectorRegistration`] is dropped.
+ /// Use [`IrqVectorRegistration::index`] to obtain an [`IrqVector`] for a given vector
+ /// index.
///
/// # Arguments
///
@@ -222,8 +191,8 @@ impl Device<device::Bound> {
///
/// # Returns
///
- /// Returns a range of IRQ vectors that were successfully allocated, or an error if the
- /// allocation fails or cannot meet the minimum requirement.
+ /// Returns the IRQ vector registration, or an error if `min_vecs` vectors cannot be
+ /// allocated.
///
/// # Examples
///
@@ -246,7 +215,20 @@ impl Device<device::Bound> {
min_vecs: u32,
max_vecs: u32,
irq_types: IrqTypes,
- ) -> Result<RangeInclusive<IrqVector<'_>>> {
- IrqVectorRegistration::register(self, min_vecs, max_vecs, irq_types)
+ ) -> Result<IrqVectorRegistration<'_>> {
+ // SAFETY:
+ // - `self.as_raw()` is guaranteed to be a valid pointer to a `struct pci_dev`
+ // by the type invariant of `Device`.
+ // - `pci_alloc_irq_vectors` internally validates all other parameters
+ // and returns error codes.
+ let ret = unsafe {
+ bindings::pci_alloc_irq_vectors(self.as_raw(), min_vecs, max_vecs, irq_types.as_raw())
+ };
+ to_result(ret)?;
+
+ let len = NonZero::new(ret as usize).ok_or(EINVAL)?;
+
+ // INVARIANT: `pci_alloc_irq_vectors()` allocated `len` vectors for `self`.
+ Ok(IrqVectorRegistration { dev: self, len })
}
}
diff --git a/rust/kernel/platform.rs b/rust/kernel/platform.rs
index 5a5f4156d79b..ac0a012ae1bb 100644
--- a/rust/kernel/platform.rs
+++ b/rust/kernel/platform.rs
@@ -100,7 +100,8 @@ impl<T: Driver> Adapter<T> {
//
// INVARIANT: `pdev` is valid for the duration of `probe_callback()`.
let pdev = unsafe { &*pdev.cast::<Device<device::CoreInternal<'_>>>() };
- let info = <Self as driver::Adapter>::id_info(pdev.as_ref());
+ // SAFETY: `pdev` matched data is of type `Self::IdInfo`.
+ let info = unsafe { <Self as driver::Adapter>::id_info(pdev.as_ref()) };
from_result(|| {
let data = T::probe(pdev, info);
@@ -176,7 +177,6 @@ macro_rules! module_platform_driver {
///
/// kernel::of_device_table!(
/// OF_TABLE,
-/// MODULE_OF_TABLE,
/// <MyDriver as platform::Driver>::IdInfo,
/// [
/// (of::DeviceId::new(c"test,device"), ())
@@ -185,7 +185,6 @@ macro_rules! module_platform_driver {
///
/// kernel::acpi_device_table!(
/// ACPI_TABLE,
-/// MODULE_ACPI_TABLE,
/// <MyDriver as platform::Driver>::IdInfo,
/// [
/// (acpi::DeviceId::new(c"LNUXBEEF"), ())
@@ -340,22 +339,30 @@ macro_rules! define_irq_accessor_by_index {
$handler_trait:ident
) => {
$(#[$meta])*
- pub fn $fn_name<'a, T: irq::$handler_trait + 'static>(
+ ///
+ /// # Safety
+ ///
+ /// Callers must not `mem::forget()` the resulting registration or otherwise prevent its
+ /// [`Drop`] implementation from running.
+ pub unsafe fn $fn_name<'a, T: irq::$handler_trait + 'a>(
&'a self,
flags: irq::Flags,
index: u32,
name: &'static CStr,
handler: impl PinInit<T, Error> + 'a,
- ) -> impl PinInit<irq::$reg_type<T>, Error> + 'a {
+ ) -> impl PinInit<irq::$reg_type<'a, T>, Error> + 'a {
pin_init::pin_init_scope(move || {
let request = self.$request_fn(index)?;
- Ok(irq::$reg_type::<T>::new(
- request,
- flags,
- name,
- handler,
- ))
+ // SAFETY: Caller guarantees the Registration will not be leaked.
+ Ok(unsafe {
+ irq::$reg_type::<T>::new(
+ request,
+ flags,
+ name,
+ handler,
+ )
+ })
})
}
};
@@ -369,22 +376,30 @@ macro_rules! define_irq_accessor_by_name {
$handler_trait:ident
) => {
$(#[$meta])*
- pub fn $fn_name<'a, T: irq::$handler_trait + 'static>(
+ ///
+ /// # Safety
+ ///
+ /// Callers must not `mem::forget()` the resulting registration or otherwise prevent its
+ /// [`Drop`] implementation from running.
+ pub unsafe fn $fn_name<'a, T: irq::$handler_trait + 'a>(
&'a self,
flags: irq::Flags,
irq_name: &'a CStr,
name: &'static CStr,
handler: impl PinInit<T, Error> + 'a,
- ) -> impl PinInit<irq::$reg_type<T>, Error> + 'a {
+ ) -> impl PinInit<irq::$reg_type<'a, T>, Error> + 'a {
pin_init::pin_init_scope(move || {
let request = self.$request_fn(irq_name)?;
- Ok(irq::$reg_type::<T>::new(
- request,
- flags,
- name,
- handler,
- ))
+ // SAFETY: Caller guarantees the Registration will not be leaked.
+ Ok(unsafe {
+ irq::$reg_type::<T>::new(
+ request,
+ flags,
+ name,
+ handler,
+ )
+ })
})
}
};
diff --git a/rust/kernel/ptr.rs b/rust/kernel/ptr.rs
index 3f3e529e9f58..82acb531b17b 100644
--- a/rust/kernel/ptr.rs
+++ b/rust/kernel/ptr.rs
@@ -235,11 +235,20 @@ impl_alignable_uint!(u8, u16, u32, u64, usize);
///
/// This is a generalization of [`size_of`] that works for dynamically sized types.
pub trait KnownSize {
+ /// Minimum size of this type known at compile-time.
+ const MIN_SIZE: usize;
+
+ /// Minimum alignment of this type known at compile-time.
+ const MIN_ALIGN: Alignment;
+
/// Get the size of an object of this type in bytes, with the metadata of the given pointer.
fn size(p: *const Self) -> usize;
}
impl<T> KnownSize for T {
+ const MIN_SIZE: usize = size_of::<T>();
+ const MIN_ALIGN: Alignment = Alignment::of::<T>();
+
#[inline(always)]
fn size(_: *const Self) -> usize {
size_of::<T>()
@@ -247,6 +256,9 @@ impl<T> KnownSize for T {
}
impl<T> KnownSize for [T] {
+ const MIN_SIZE: usize = 0;
+ const MIN_ALIGN: Alignment = Alignment::of::<T>();
+
#[inline(always)]
fn size(p: *const Self) -> usize {
p.len() * size_of::<T>()
diff --git a/rust/kernel/serdev.rs b/rust/kernel/serdev.rs
new file mode 100644
index 000000000000..17ca504b7f8d
--- /dev/null
+++ b/rust/kernel/serdev.rs
@@ -0,0 +1,604 @@
+// SPDX-License-Identifier: GPL-2.0
+
+//! Abstractions for the serial device bus.
+//!
+//! C header: [`include/linux/serdev.h`](srctree/include/linux/serdev.h)
+
+use crate::{
+ acpi,
+ device,
+ driver,
+ error::{
+ from_result,
+ to_result,
+ VTABLE_DEFAULT_ERROR, //
+ },
+ new_mutex,
+ of,
+ prelude::*,
+ sync::{
+ aref::AlwaysRefCounted,
+ Mutex, //
+ },
+ time::Jiffies,
+ types::{
+ Opaque,
+ ScopeGuard, //
+ }, //
+};
+
+use core::{
+ cell::UnsafeCell,
+ marker::PhantomData,
+ mem::{offset_of, MaybeUninit},
+ ptr::NonNull, //
+};
+
+/// Parity bit to use with a serial device.
+#[repr(u32)]
+pub enum Parity {
+ /// No parity bit.
+ None = bindings::serdev_parity_SERDEV_PARITY_NONE,
+ /// Even partiy.
+ Even = bindings::serdev_parity_SERDEV_PARITY_EVEN,
+ /// Odd parity.
+ Odd = bindings::serdev_parity_SERDEV_PARITY_ODD,
+}
+
+/// An adapter for the registration of serial device bus device drivers.
+pub struct Adapter<T: Driver>(T);
+
+// SAFETY:
+// - `bindings::serdev_device_driver` is a C type declared as `repr(C)`.
+// - `PrivateData<'bound, T>` is the type of the driver's device private data.
+// - `struct serdev_device_driver` embeds a `struct device_driver`.
+// - `DEVICE_DRIVER_OFFSET` is the correct byte offset to the embedded `struct device_driver`.
+unsafe impl<T: Driver> driver::DriverLayout for Adapter<T> {
+ type DriverType = bindings::serdev_device_driver;
+ type DriverData<'bound> = PrivateData<'bound, T>;
+ const DEVICE_DRIVER_OFFSET: usize = core::mem::offset_of!(Self::DriverType, driver);
+}
+
+// SAFETY: A call to `unregister` for a given instance of `DriverType` is guaranteed to be valid if
+// a preceding call to `register` has been successful.
+unsafe impl<T: Driver> driver::RegistrationOps for Adapter<T> {
+ unsafe fn register(
+ sdrv: &Opaque<Self::DriverType>,
+ name: &'static CStr,
+ module: &'static ThisModule,
+ ) -> Result {
+ let of_table = match T::OF_ID_TABLE {
+ Some(table) => table.as_ptr(),
+ None => core::ptr::null(),
+ };
+
+ let acpi_table = match T::ACPI_ID_TABLE {
+ Some(table) => table.as_ptr(),
+ None => core::ptr::null(),
+ };
+
+ // SAFETY: It's safe to set the fields of `struct serdev_device_driver` on initialization.
+ unsafe {
+ (*sdrv.get()).driver.name = name.as_char_ptr();
+ (*sdrv.get()).probe = Some(Self::probe_callback);
+ (*sdrv.get()).remove = Some(Self::remove_callback);
+ (*sdrv.get()).driver.of_match_table = of_table;
+ (*sdrv.get()).driver.acpi_match_table = acpi_table;
+ }
+
+ // SAFETY: `sdrv` is guaranteed to be a valid `DriverType`.
+ to_result(unsafe { bindings::__serdev_device_driver_register(sdrv.get(), module.as_ptr()) })
+ }
+
+ unsafe fn unregister(sdrv: &Opaque<Self::DriverType>) {
+ // SAFETY: `sdrv` is guaranteed to be a valid `DriverType`.
+ unsafe { bindings::serdev_device_driver_unregister(sdrv.get()) };
+ }
+}
+
+#[doc(hidden)]
+#[pin_data(PinnedDrop)]
+pub struct PrivateData<'bound, T: Driver> {
+ sdev: &'bound Device<device::Bound>,
+ #[pin]
+ driver: UnsafeCell<MaybeUninit<T::Data<'bound>>>,
+ open: UnsafeCell<bool>,
+ /// Whether `receive_buf_callback` is allowed to call `Driver::receive`.
+ ///
+ /// If locked, the receive_buf_callback will be blocked on data reception.
+ /// This is the case while the driver is being probed or while [`PrivateData`] is being dropped.
+ /// This is necessary, because we need to open the serdev device before the driver has been
+ /// probed in order to allow it to be configured, which allows `receive_buf_callback` to be
+ /// called. Thus we need to block data until probe completes and the driver data becomes
+ /// initialized.
+ ///
+ /// If unlocked and true, the receive_buf_callback will forward the data to
+ /// `Driver::receive`. This is the normal state of operation.
+ ///
+ /// If unlocked and false, the receive_buf_callback will throw away the data.
+ /// This is only the case, if the serdev device is open and
+ /// - the driver returned an error in probe
+ /// or
+ /// - the driver data already has been dropped, because it was unbound.
+ #[pin]
+ active: Mutex<bool>,
+}
+
+#[pinned_drop]
+impl<T: Driver> PinnedDrop for PrivateData<'_, T> {
+ fn drop(self: Pin<&mut Self>) {
+ let mut active = self.active.lock();
+ if *active {
+ // SAFETY:
+ // - We have exclusive access to `self.driver`.
+ // - `self.driver` is guaranteed to be initialized.
+ unsafe { (*self.driver.get()).assume_init_drop() };
+ *active = false;
+ }
+ drop(active);
+
+ // SAFETY: We have exclusive access to `self.open`.
+ if unsafe { *self.open.get() } {
+ // SAFETY: `self.sdev.as_raw()` is guaranteed to be a pointer to a valid
+ // `struct serdev_device`.
+ unsafe { bindings::serdev_device_close(self.sdev.as_raw()) };
+ }
+ }
+}
+
+impl<T: Driver> Adapter<T> {
+ const OPS: &'static bindings::serdev_device_ops = &bindings::serdev_device_ops {
+ receive_buf: if T::HAS_RECEIVE {
+ Some(Self::receive_buf_callback)
+ } else {
+ None
+ },
+ write_wakeup: Some(bindings::serdev_device_write_wakeup),
+ };
+
+ extern "C" fn probe_callback(sdev: *mut bindings::serdev_device) -> kernel::ffi::c_int {
+ // SAFETY: The serial device bus only ever calls the probe callback with a valid pointer to
+ // a `struct serdev_device`.
+ //
+ // INVARIANT: `sdev` is valid for the duration of `probe_callback()`.
+ let sdev = unsafe { &*sdev.cast::<Device<device::CoreInternal<'_>>>() };
+ // SAFETY: `sdev` matched data is of type `Self::IdInfo`.
+ let info = unsafe { <Self as driver::Adapter>::id_info(sdev.as_ref()) };
+
+ from_result(|| {
+ sdev.as_ref().set_drvdata(try_pin_init!(PrivateData::<T> {
+ sdev: &**sdev,
+ driver: MaybeUninit::<T::Data<'_>>::zeroed().into(),
+ open: false.into(),
+ active <- new_mutex!(false),
+ }))?;
+ // SAFETY: We just set drvdata to `PrivateData<'_, T>`.
+ let private_data = unsafe { sdev.as_ref().drvdata_borrow::<PrivateData<'_, T>>() };
+ let private_data = ScopeGuard::new_with_data(private_data, |_| {
+ // SAFETY: We just set drvdata to `PrivateData<'_, T>`.
+ drop(unsafe { sdev.as_ref().drvdata_obtain::<PrivateData<'_, T>>() });
+ });
+ let mut active = private_data.active.lock();
+
+ // SAFETY: `sdev.as_raw()` is guaranteed to be a valid pointer to `serdev_device`.
+ unsafe { bindings::serdev_device_set_client_ops(sdev.as_raw(), Self::OPS) };
+
+ // SAFETY: The serial device bus only ever calls the probe callback with a valid pointer
+ // to a `serdev_device`.
+ to_result(unsafe { bindings::serdev_device_open(sdev.as_raw()) })?;
+
+ // SAFETY: We have exclusive access to `private_data.open`.
+ unsafe { *private_data.open.get() = true };
+
+ let data = T::probe(sdev, info);
+
+ // SAFETY: We have exclusive access to `private_data.driver`.
+ let driver = unsafe { &mut *private_data.driver.get() };
+ // SAFETY:
+ // - `driver.as_mut_ptr()` is a valid pointer to uninitialized data.
+ // - `private_data.driver` is pinned.
+ let result = unsafe { pin_init::raw_try_init(driver.as_mut_ptr(), data) };
+
+ *active = result.is_ok();
+
+ drop(active);
+
+ result.map(|()| {
+ private_data.dismiss();
+ 0
+ })
+ })
+ }
+
+ extern "C" fn remove_callback(sdev: *mut bindings::serdev_device) {
+ // SAFETY: The serial device bus only ever calls the remove callback with a valid pointer
+ // to a `struct serdev_device`.
+ //
+ // INVARIANT: `sdev` is valid for the duration of `remove_callback()`.
+ let sdev = unsafe { &*sdev.cast::<Device<device::CoreInternal<'_>>>() };
+
+ // SAFETY: `remove_callback` is only ever called after a successful call to
+ // `probe_callback`, hence it's guaranteed that `Device::set_drvdata()` has been called
+ // and stored a `Pin<KBox<PrivateData<'_, T>>>`.
+ let private_data = unsafe { sdev.as_ref().drvdata_borrow::<PrivateData<'_, T>>() };
+
+ // SAFETY: No one has exclusive access to `private_data.driver`.
+ let data = unsafe { &*private_data.driver.get() };
+ // SAFETY:
+ // - `private_data.driver` is pinned.
+ // - `remove_callback` is only ever called after a successful call to `probe_callback`,
+ // hence it's guaranteed that `private_data.driver` was initialized.
+ let data_pinned = unsafe { Pin::new_unchecked(data.assume_init_ref()) };
+
+ T::unbind(sdev, data_pinned);
+ }
+
+ extern "C" fn receive_buf_callback(
+ sdev: *mut bindings::serdev_device,
+ buf: *const u8,
+ length: usize,
+ ) -> usize {
+ // SAFETY: The serial device bus only ever calls the receive buf callback with a valid
+ // pointer to a `struct serdev_device`.
+ //
+ // INVARIANT: `sdev` is valid for the duration of `receive_buf_callback()`.
+ let sdev = unsafe { &*sdev.cast::<Device<device::BoundInternal>>() };
+
+ // SAFETY: `receive_buf_callback` is only ever called after a successful call to
+ // `probe_callback`, hence it's guaranteed that `Device::set_drvdata()` has been called
+ // and stored a `Pin<KBox<PrivateData<'_, T>>>`.
+ let private_data = unsafe { sdev.as_ref().drvdata_borrow::<PrivateData<'_, T>>() };
+ let active = private_data.active.lock();
+
+ if !*active {
+ return length;
+ }
+
+ // SAFETY: No one has exclusive access to `private_data.driver`.
+ let data = unsafe { &*private_data.driver.get() };
+ // SAFETY:
+ // - `private_data.driver` is pinned.
+ // - `receive_buf_callback` is only ever called after a successful call to `probe_callback`,
+ // hence it's guaranteed that `private_data.driver` was initialized.
+ let data_pinned = unsafe { Pin::new_unchecked(data.assume_init_ref()) };
+
+ // SAFETY: `buf` is guaranteed to be non-null and has the size of `length`.
+ let buf = unsafe { core::slice::from_raw_parts(buf, length) };
+
+ T::receive(sdev, data_pinned, buf)
+ }
+}
+
+impl<T: Driver> driver::Adapter for Adapter<T> {
+ type IdInfo = T::IdInfo;
+
+ fn of_id_table() -> Option<of::IdTable<Self::IdInfo>> {
+ T::OF_ID_TABLE
+ }
+
+ fn acpi_id_table() -> Option<acpi::IdTable<Self::IdInfo>> {
+ T::ACPI_ID_TABLE
+ }
+}
+
+/// Declares a kernel module that exposes a single serial device bus device driver.
+///
+/// # Examples
+///
+/// ```ignore
+/// kernel::module_serdev_device_driver! {
+/// type: MyDriver,
+/// name: "Module name",
+/// authors: ["Author name"],
+/// description: "Description",
+/// license: "GPL v2",
+/// }
+/// ```
+#[macro_export]
+macro_rules! module_serdev_device_driver {
+ ($($f:tt)*) => {
+ $crate::module_driver!(<T>, $crate::serdev::Adapter<T>, { $($f)* });
+ };
+}
+
+/// The serial device bus device driver trait.
+///
+/// Drivers must implement this trait in order to get a serial device bus device driver registered.
+///
+/// # Examples
+///
+///```
+/// # use kernel::{
+/// acpi,
+/// bindings,
+/// device::{
+/// Bound,
+/// Core, //
+/// },
+/// of,
+/// serdev, //
+/// };
+///
+/// struct MyDriver;
+///
+/// kernel::of_device_table!(
+/// OF_TABLE,
+/// <MyDriver as serdev::Driver>::IdInfo,
+/// [
+/// (of::DeviceId::new(c"test,device"), ())
+/// ]
+/// );
+///
+/// kernel::acpi_device_table!(
+/// ACPI_TABLE,
+/// <MyDriver as serdev::Driver>::IdInfo,
+/// [
+/// (acpi::DeviceId::new(c"LNUXBEEF"), ())
+/// ]
+/// );
+///
+/// #[vtable]
+/// impl serdev::Driver for MyDriver {
+/// type IdInfo = ();
+/// type Data<'bound> = Self;
+/// const OF_ID_TABLE: Option<of::IdTable<Self::IdInfo>> = Some(&OF_TABLE);
+/// const ACPI_ID_TABLE: Option<acpi::IdTable<Self::IdInfo>> = Some(&ACPI_TABLE);
+///
+/// fn probe<'bound>(
+/// sdev: &'bound serdev::Device<Core<'_>>,
+/// _id_info: Option<&'bound Self::IdInfo>,
+/// ) -> impl PinInit<Self::Data<'bound>, Error> + 'bound {
+/// sdev.set_baudrate(115200);
+/// sdev.write_all(b"Hello\n", 0)?;
+/// Ok(MyDriver)
+/// }
+/// }
+///```
+#[vtable]
+pub trait Driver {
+ /// The type holding driver private data about each device id supported by the driver.
+ // TODO: Use associated_type_defaults once stabilized:
+ //
+ // ```
+ // type IdInfo: 'static = ();
+ // ```
+ type IdInfo: 'static;
+
+ /// The type of the driver's bus device private data.
+ type Data<'bound>: Send + Sync + 'bound;
+
+ /// The table of OF device ids supported by the driver.
+ const OF_ID_TABLE: Option<of::IdTable<Self::IdInfo>> = None;
+
+ /// The table of ACPI device ids supported by the driver.
+ const ACPI_ID_TABLE: Option<acpi::IdTable<Self::IdInfo>> = None;
+
+ /// Serial device bus device driver probe.
+ ///
+ /// Called when a new serial device bus device is added or discovered.
+ /// Implementers should attempt to initialize the device here.
+ fn probe<'bound>(
+ sdev: &'bound Device<device::Core<'_>>,
+ id_info: Option<&'bound Self::IdInfo>,
+ ) -> impl PinInit<Self::Data<'bound>, Error> + 'bound;
+
+ /// Serial device bus device driver unbind.
+ ///
+ /// Called when a [`Device`] is unbound from its bound [`Driver`]. Implementing this callback
+ /// is optional.
+ ///
+ /// This callback serves as a place for drivers to perform teardown operations that require a
+ /// `&Device<Core>` or `&Device<Bound>` reference. For instance.
+ ///
+ /// Otherwise, release operations for driver resources should be performed in `Drop`.
+ fn unbind<'bound>(sdev: &'bound Device<device::Core<'_>>, this: Pin<&Self::Data<'bound>>) {
+ let _ = (sdev, this);
+ }
+
+ /// Serial device bus device data receive callback.
+ ///
+ /// Called when data got received from device.
+ ///
+ /// Returns the number of bytes accepted.
+ fn receive<'bound>(
+ sdev: &'bound Device<device::Bound>,
+ this: Pin<&Self::Data<'bound>>,
+ data: &[u8],
+ ) -> usize {
+ let _ = (sdev, this, data);
+ build_error!(VTABLE_DEFAULT_ERROR)
+ }
+}
+
+/// The serial device bus device representation.
+///
+/// This structure represents the Rust abstraction for a C `struct serdev_device`. The
+/// implementation abstracts the usage of an already existing C `struct serdev_device` within Rust
+/// code that we get passed from the C side.
+///
+/// # Invariants
+///
+/// A [`Device`] instance represents a valid `struct serdev_device` created by the C portion of
+/// the kernel.
+#[repr(transparent)]
+pub struct Device<Ctx: device::DeviceContext = device::Normal>(
+ Opaque<bindings::serdev_device>,
+ PhantomData<Ctx>,
+);
+
+impl<Ctx: device::DeviceContext> Device<Ctx> {
+ #[inline]
+ fn as_raw(&self) -> *mut bindings::serdev_device {
+ self.0.get()
+ }
+}
+
+impl Device<device::Bound> {
+ /// Set the baudrate in bits per second.
+ ///
+ /// Common baudrates are 115200, 9600, 19200, 57600, 4800.
+ ///
+ /// Use [`Device::write_flush`] before calling this if you have written data prior to this call.
+ #[inline]
+ pub fn set_baudrate(&self, speed: u32) -> Result<(), u32> {
+ // SAFETY: `self.as_raw()` is guaranteed to be a pointer to a valid `serdev_device`.
+ let ret = unsafe { bindings::serdev_device_set_baudrate(self.as_raw(), speed) };
+ if ret == speed {
+ Ok(())
+ } else {
+ Err(ret)
+ }
+ }
+
+ /// Set if flow control should be enabled.
+ ///
+ /// Use [`Device::write_flush`] before calling this if you have written data prior to this call.
+ #[inline]
+ pub fn set_flow_control(&self, enable: bool) {
+ // SAFETY: `self.as_raw()` is guaranteed to be a pointer to a valid `serdev_device`.
+ unsafe { bindings::serdev_device_set_flow_control(self.as_raw(), enable) };
+ }
+
+ /// Set parity to use.
+ ///
+ /// Use [`Device::write_flush`] before calling this if you have written data prior to this call.
+ #[inline]
+ pub fn set_parity(&self, parity: Parity) -> Result {
+ // SAFETY: `self.as_raw()` is guaranteed to be a pointer to a valid `serdev_device`.
+ to_result(unsafe { bindings::serdev_device_set_parity(self.as_raw(), parity as u32) })
+ }
+
+ /// Write data to the serial device until the controller has accepted all the data or has
+ /// been interrupted by a timeout or signal.
+ ///
+ /// Note that any accepted data has only been buffered by the controller. Use
+ /// [`Device::wait_until_sent`] to make sure the controller write buffer has actually been
+ /// emptied.
+ ///
+ /// Use a timeout of 0 to wait indefinitely.
+ ///
+ /// Returns the number of bytes written (less than `data.len()` if interrupted).
+ /// [`kernel::error::code::ETIMEDOUT`] or [`kernel::error::code::ERESTARTSYS`] if interrupted
+ /// before any bytes were written. [`kernel::error::code::EINVAL`] if `data.len() > i32::MAX`.
+ #[inline]
+ pub fn write_all(&self, data: &[u8], timeout: Jiffies) -> Result<usize> {
+ if data.len() > i32::MAX as usize {
+ return Err(EINVAL);
+ }
+
+ // SAFETY:
+ // - `self.as_raw()` is guaranteed to be a pointer to a valid `serdev_device`.
+ // - `data.as_ptr()` is guaranteed to be a valid array pointer with the size of
+ // `data.len()`.
+ let ret = unsafe {
+ bindings::serdev_device_write(
+ self.as_raw(),
+ data.as_ptr(),
+ data.len(),
+ isize::try_from(timeout).unwrap_or_default(),
+ )
+ };
+ // CAST: negative return values are guaranteed to be between `-MAX_ERRNO` and `-1`,
+ // which always fit into a `i32`.
+ to_result(ret as i32).map(|()| ret.unsigned_abs())
+ }
+
+ /// Write data to the serial device.
+ ///
+ /// If you want to write until the controller has accepted all the data, use
+ /// [`Device::write_all`].
+ ///
+ /// Note that any accepted data has only been buffered by the controller. Use
+ /// [`Device::wait_until_sent`] to make sure the controller write buffer has actually been
+ /// emptied.
+ ///
+ /// Returns the number of bytes written (less than `data.len()` if not enough room in the
+ /// write buffer).
+ #[inline]
+ pub fn write(&self, data: &[u8]) -> Result<u32> {
+ if data.len() > i32::MAX as usize {
+ return Err(EINVAL);
+ }
+
+ // SAFETY:
+ // - `self.as_raw()` is guaranteed to be a pointer to a valid `serdev_device`.
+ // - `data.as_ptr()` is guaranteed to be a valid array pointer with the size of
+ // `data.len()`.
+ let ret =
+ unsafe { bindings::serdev_device_write_buf(self.as_raw(), data.as_ptr(), data.len()) };
+
+ to_result(ret as i32).map(|()| ret.unsigned_abs())
+ }
+
+ /// Send data to the serial device immediately.
+ ///
+ /// Note that this doesn't guarantee that the data has been transmitted.
+ /// Use [`Device::wait_until_sent`] for this purpose.
+ #[inline]
+ pub fn write_flush(&self) {
+ // SAFETY: `self.as_raw()` is guaranteed to be a pointer to a valid `serdev_device`.
+ unsafe { bindings::serdev_device_write_flush(self.as_raw()) };
+ }
+
+ /// Wait for the data to be sent.
+ ///
+ /// After this function, the write buffer of the controller should be empty or the timeout
+ /// elapsed.
+ ///
+ /// Use a timeout of 0 to wait indefinitely.
+ #[inline]
+ pub fn wait_until_sent(&self, timeout: Jiffies) {
+ // SAFETY: `self.as_raw()` is guaranteed to be a pointer to a valid `serdev_device`.
+ unsafe {
+ bindings::serdev_device_wait_until_sent(
+ self.as_raw(),
+ isize::try_from(timeout).unwrap_or_default(),
+ )
+ };
+ }
+}
+
+// SAFETY: `serdev::Device` is a transparent wrapper of `struct serdev_device`.
+// The offset is guaranteed to point to a valid device field inside `serdev::Device`.
+unsafe impl<Ctx: device::DeviceContext> device::AsBusDevice<Ctx> for Device<Ctx> {
+ const OFFSET: usize = offset_of!(bindings::serdev_device, dev);
+}
+
+// SAFETY: `Device` is a transparent wrapper of a type that doesn't depend on `Device`'s generic
+// argument.
+kernel::impl_device_context_deref!(unsafe { Device });
+kernel::impl_device_context_into_aref!(Device);
+
+// SAFETY: Instances of `Device` are always reference-counted.
+unsafe impl AlwaysRefCounted for Device {
+ fn inc_ref(&self) {
+ self.as_ref().inc_ref();
+ }
+
+ unsafe fn dec_ref(obj: NonNull<Self>) {
+ // SAFETY: The safety requirements guarantee that the refcount is non-zero.
+ unsafe { bindings::serdev_device_put(obj.cast().as_ptr()) }
+ }
+}
+
+impl<Ctx: device::DeviceContext> AsRef<device::Device<Ctx>> for Device<Ctx> {
+ fn as_ref(&self) -> &device::Device<Ctx> {
+ // SAFETY: By the type invariant of `Self`, `self.as_raw()` is a pointer to a valid
+ // `struct serdev_device`.
+ let dev = unsafe { &raw mut (*self.as_raw()).dev };
+
+ // SAFETY: `dev` points to a valid `struct device`.
+ unsafe { device::Device::from_raw(dev) }
+ }
+}
+
+// SAFETY: A `Device` is always reference-counted and can be released from any thread.
+unsafe impl Send for Device {}
+
+// SAFETY: `Device` can be shared among threads because all methods of `Device`
+// (i.e. `Device<Normal>) are thread safe.
+unsafe impl Sync for Device {}
+
+// SAFETY: Same as `Device<Normal>` -- the underlying `struct serdev_device` is the same;
+// `Bound` is a zero-sized type-state marker that does not affect thread safety.
+unsafe impl Sync for Device<device::Bound> {}
diff --git a/rust/kernel/types.rs b/rust/kernel/types.rs
index 67b3874cb3d2..132dd428c1f6 100644
--- a/rust/kernel/types.rs
+++ b/rust/kernel/types.rs
@@ -13,7 +13,10 @@ use pin_init::{PinInit, Wrapper, Zeroable};
#[doc(hidden)]
pub mod for_lt;
-pub use for_lt::ForLt;
+pub use for_lt::{
+ CovariantForLt,
+ ForLt, //
+};
/// Used to transfer ownership to and from foreign (non-Rust) languages.
///
diff --git a/rust/kernel/types/for_lt.rs b/rust/kernel/types/for_lt.rs
index d44323c28e8d..b8f422c802dc 100644
--- a/rust/kernel/types/for_lt.rs
+++ b/rust/kernel/types/for_lt.rs
@@ -1,22 +1,19 @@
// SPDX-License-Identifier: Apache-2.0 OR MIT
-//! Provide implementation and test of the `ForLt` trait and macro.
+//! Provide implementation and test of the [`trait@ForLt`] and [`trait@CovariantForLt`] traits and
+//! macros.
//!
-//! This module is hidden and user should just use `ForLt!` directly.
+//! This module is hidden and users should just use [`ForLt!`](macro@ForLt) /
+//! [`CovariantForLt!`](macro@CovariantForLt) directly.
use core::marker::PhantomData;
/// Representation of types generic over a lifetime.
///
-/// The type must be covariant over the generic lifetime, i.e. the lifetime parameter
-/// can be soundly shortened.
-///
-/// The lifetime involved must be covariant.
-///
/// # Macro
///
-/// It is not recommended to implement this trait directly. `ForLt!` macro is provided to obtain a
-/// type that implements this trait.
+/// It is not recommended to implement this trait directly. [`ForLt!`](macro@ForLt) macro is
+/// provided to obtain a type that implements this trait.
///
/// The full syntax is
///
@@ -49,16 +46,65 @@ use core::marker::PhantomData;
/// ForLt!(u32) // Equivalent to `ForLt!(for<'a> u32)`.
/// # >();
/// ```
+pub trait ForLt {
+ /// The type parameterized by the lifetime.
+ type Of<'a>: 'a;
+}
+pub use macros::ForLt;
+
+/// [`trait@ForLt`] subtrait for types that are covariant over their lifetime parameter.
+///
+/// Provides a safe [`cast_ref`](CovariantForLt::cast_ref) method for types that are proven to be
+/// covariant. The `CovariantForLt!` macro syntax is the same as `ForLt!`.
+///
+/// # Macro
+///
+/// It is not recommended to implement this trait directly.
+/// [`CovariantForLt!`](macro@CovariantForLt) macro is provided to obtain a type that implements
+/// this trait.
+///
+/// The full syntax is
+///
+/// ```
+/// # use kernel::types::CovariantForLt;
+/// # fn expect_lt<F: CovariantForLt>() {}
+/// # struct TypeThatUse<'a>(&'a ());
+/// # expect_lt::<
+/// CovariantForLt!(for<'a> TypeThatUse<'a>)
+/// # >();
+/// ```
+///
+/// which gives a type so that
+/// `<CovariantForLt!(for<'a> TypeThatUse<'a>) as CovariantForLt>::Of<'b>`
+/// is `TypeThatUse<'b>`.
+///
+/// You may also use a short-hand syntax which works similar to lifetime elision.
+/// The macro also accepts types that do not involve a lifetime at all.
+///
+/// ```
+/// # use kernel::types::CovariantForLt;
+/// # fn expect_lt<F: CovariantForLt>() {}
+/// # struct TypeThatUse<'a>(&'a ());
+/// # expect_lt::<
+/// CovariantForLt!(TypeThatUse<'_>) // Equivalent to `CovariantForLt!(for<'a> TypeThatUse<'a>)`.
+/// # >();
+/// # expect_lt::<
+/// CovariantForLt!(&u32) // Equivalent to `CovariantForLt!(for<'a> &'a u32)`.
+/// # >();
+/// # expect_lt::<
+/// CovariantForLt!(u32) // Equivalent to `CovariantForLt!(for<'a> u32)`.
+/// # >();
+/// ```
///
/// The macro will attempt to prove that the type is indeed covariant over the lifetime supplied.
/// When it cannot be syntactically proven, it will emit checks to ask the Rust compiler to prove
/// it.
///
/// ```ignore,compile_fail
-/// # use kernel::types::ForLt;
-/// # fn expect_lt<F: ForLt>() {}
+/// # use kernel::types::CovariantForLt;
+/// # fn expect_lt<F: CovariantForLt>() {}
/// # expect_lt::<
-/// ForLt!(fn(&u32)) // Contravariant, will fail compilation.
+/// CovariantForLt!(fn(&u32)) // Contravariant, will fail compilation.
/// # >();
/// ```
///
@@ -67,26 +113,23 @@ use core::marker::PhantomData;
/// the generic parameter but is in a separate item.
///
/// ```
-/// # use kernel::types::ForLt;
-/// fn expect_lt<F: ForLt>() {}
+/// # use kernel::types::CovariantForLt;
+/// fn expect_lt<F: CovariantForLt>() {}
/// # #[allow(clippy::unnecessary_safety_comment, reason = "false positive")]
/// fn generic_fn<T: 'static>() {
/// // Syntactically proven by the macro
-/// expect_lt::<ForLt!(&T)>();
+/// expect_lt::<CovariantForLt!(&T)>();
/// // Syntactically proven by the macro
-/// expect_lt::<ForLt!(&KBox<T>)>();
+/// expect_lt::<CovariantForLt!(&KBox<T>)>();
/// // Cannot be syntactically proven, need to check covariance of `KBox`
-/// // expect_lt::<ForLt!(&KBox<&T>)>();
+/// // expect_lt::<CovariantForLt!(&KBox<&T>)>();
/// }
/// ```
///
/// # Safety
///
/// `Self::Of<'a>` must be covariant over the lifetime `'a`.
-pub unsafe trait ForLt {
- /// The type parameterized by the lifetime.
- type Of<'a>: 'a;
-
+pub unsafe trait CovariantForLt: ForLt {
/// Cast a reference to a shorter lifetime.
#[inline(always)]
fn cast_ref<'r, 'short: 'r, 'long: 'short>(long: &'r Self::Of<'long>) -> &'r Self::Of<'short> {
@@ -94,29 +137,33 @@ pub unsafe trait ForLt {
unsafe { core::mem::transmute(long) }
}
}
-pub use macros::ForLt;
+pub use macros::CovariantForLt;
/// This is intended to be an "unsafe-to-refer-to" type.
///
-/// Must only be used by the `ForLt!` macro.
+/// Must only be used by the [`ForLt!`](macro@ForLt) / [`CovariantForLt!`](macro@CovariantForLt)
+/// macros.
///
/// `T` is the magic `dyn for<'a> WithLt<'a, TypeThatUse<'a>>` generated by macro.
///
/// `WF` is a type that the macro can use to assert some specific type is well-formed.
///
/// `N` is to provide the macro a place to emit arbitrary items, in case it needs to prove
-/// additional properties.
+/// additional properties. [`ForLt!`](macro@ForLt) emits `N = 0`;
+/// [`CovariantForLt!`](macro@CovariantForLt) emits `N = 1` after a covariance proof.
#[doc(hidden)]
pub struct UnsafeForLtImpl<T: ?Sized, WF, const N: usize>(PhantomData<(WF, T)>);
-// This is a helper trait for implementation `ForLt` to be able to use HRTB.
+// This is a helper trait for implementation of `ForLt` / `CovariantForLt` to be able to use HRTB.
#[doc(hidden)]
pub trait WithLt<'a> {
type Of: 'a;
}
-// SAFETY: In `ForLt!` macro, a covariance proof is generated when naming `UnsafeForLtImpl`
-// and it will fail to evaluate if the type is not covariant.
-unsafe impl<T: ?Sized + for<'a> WithLt<'a>, WF> ForLt for UnsafeForLtImpl<T, WF, 0> {
+impl<T: ?Sized + for<'a> WithLt<'a>, WF, const N: usize> ForLt for UnsafeForLtImpl<T, WF, N> {
type Of<'a> = <T as WithLt<'a>>::Of;
}
+
+// SAFETY: In `CovariantForLt!` macro, a covariance proof is generated in the `N` const generic
+// and it will fail to evaluate if the type is not covariant. Only `N = 1` gets this impl.
+unsafe impl<T: ?Sized + for<'a> WithLt<'a>, WF> CovariantForLt for UnsafeForLtImpl<T, WF, 1> {}
diff --git a/rust/kernel/usb.rs b/rust/kernel/usb.rs
index 870423806e4f..558e15a08478 100644
--- a/rust/kernel/usb.rs
+++ b/rust/kernel/usb.rs
@@ -89,7 +89,10 @@ impl<T: Driver> Adapter<T> {
// does not add additional invariants, so it's safe to transmute.
let id = unsafe { &*id.cast::<DeviceId>() };
- let info = T::ID_TABLE.info(id.index());
+ // SAFETY: `id` comes from `T::ID_TABLE` which is of type `IdArray<_, T::IdInfo>`. It
+ // can also come from dynamic IDs, which will ensure that `driver_data` exists in
+ // `T::ID_TABLE` or is 0.
+ let info = unsafe { id.info_unchecked_opt::<T::IdInfo>() };
let data = T::probe(intf, id, info);
let dev: &device::Device<device::CoreInternal<'_>> = intf.as_ref();
@@ -242,10 +245,6 @@ unsafe impl RawDeviceId for DeviceId {
// SAFETY: `DRIVER_DATA_OFFSET` is the offset to the `driver_info` field.
unsafe impl RawDeviceIdIndex for DeviceId {
const DRIVER_DATA_OFFSET: usize = core::mem::offset_of!(bindings::usb_device_id, driver_info);
-
- fn index(&self) -> usize {
- self.0.driver_info
- }
}
/// [`IdTable`](kernel::device_id::IdTable) type for USB.
@@ -254,14 +253,8 @@ pub type IdTable<T> = &'static dyn kernel::device_id::IdTable<DeviceId, T>;
/// Create a USB `IdTable` with its alias for modpost.
#[macro_export]
macro_rules! usb_device_table {
- ($table_name:ident, $module_table_name:ident, $id_info_type: ty, $table_data: expr) => {
- const $table_name: $crate::device_id::IdArray<
- $crate::usb::DeviceId,
- $id_info_type,
- { $table_data.len() },
- > = $crate::device_id::IdArray::new($table_data);
-
- $crate::module_device_table!("usb", $module_table_name, $table_name);
+ ($($tt:tt)*) => {
+ $crate::module_device_table!("usb", $crate::usb::DeviceId, $($tt)*);
};
}
@@ -277,7 +270,6 @@ macro_rules! usb_device_table {
///
/// kernel::usb_device_table!(
/// USB_TABLE,
-/// MODULE_USB_TABLE,
/// <MyDriver as usb::Driver>::IdInfo,
/// [
/// (usb::DeviceId::from_id(0x1234, 0x5678), ()),
@@ -293,7 +285,7 @@ macro_rules! usb_device_table {
/// fn probe<'bound>(
/// _interface: &'bound usb::Interface<Core<'_>>,
/// _id: &usb::DeviceId,
-/// _info: &'bound Self::IdInfo,
+/// _info: Option<&'bound Self::IdInfo>,
/// ) -> impl PinInit<Self::Data<'bound>, Error> + 'bound {
/// Err(ENODEV)
/// }
@@ -322,7 +314,7 @@ pub trait Driver {
fn probe<'bound>(
interface: &'bound Interface<device::Core<'_>>,
id: &DeviceId,
- id_info: &'bound Self::IdInfo,
+ id_info: Option<&'bound Self::IdInfo>,
) -> impl PinInit<Self::Data<'bound>, Error> + 'bound;
/// USB driver disconnect.
diff --git a/rust/macros/for_lt.rs b/rust/macros/for_lt.rs
index 364d4113cd10..4372cbad3ec4 100644
--- a/rust/macros/for_lt.rs
+++ b/rust/macros/for_lt.rs
@@ -154,8 +154,8 @@ impl<'a> Prover<'a> {
// Note that if we encounter `&'other_lt T`, then we still need to make sure the type
// is wellformed if `T` involves `&'lt`, so we defer to the compiler.
//
- // This is to block cases like `ForLt!(for<'a> &'static &'a u32)`, as the presence of
- // the type implies `'a: 'static` but this is unsound.
+ // This is to block cases like `CovariantForLt!(for<'a> &'static &'a u32)`, as the
+ // presence of the type implies `'a: 'static` but this is unsound.
Type::Reference(ty)
if ty.mutability.is_none() && ty.lifetime.as_ref() == Some(self.0) =>
{
@@ -176,7 +176,12 @@ impl<'a> Prover<'a> {
}
}
-pub(crate) fn for_lt(input: HigherRankedType) -> TokenStream {
+/// Shared implementation for both `ForLt!` and `CovariantForLt!`.
+///
+/// Both macros run the prover and emit `ProveWf` structs to check well-formedness for all lifetime
+/// instances (workaround for <https://github.com/rust-lang/rust/issues/152489>). `CovariantForLt!`
+/// additionally emits covariance proof functions and sets `N = 1`.
+fn for_lt_inner(input: HigherRankedType, prove_covariance: bool) -> TokenStream {
let (ty, lifetime) = match input {
HigherRankedType::Explicit { lifetime, ty, .. } => (ty, lifetime),
HigherRankedType::Implicit { ty } => {
@@ -211,14 +216,16 @@ pub(crate) fn for_lt(input: HigherRankedType) -> TokenStream {
));
// Insert a proof that the type is covariant.
- let cov_proof_name = format_ident!("prove_covariant_{idx}");
- proof.push(quote!(
- fn #cov_proof_name<'__short, '__long: '__short>(
- long: #wf_proof_name<'__long>
- ) -> #wf_proof_name<'__short> {
- long
- }
- ));
+ if prove_covariance {
+ let cov_proof_name = format_ident!("prove_covariant_{idx}");
+ proof.push(quote!(
+ fn #cov_proof_name<'__short, '__long: '__short>(
+ long: #wf_proof_name<'__long>
+ ) -> #wf_proof_name<'__short> {
+ long
+ }
+ ));
+ }
}
// Make sure that the type is wellformed when substituting lifetime with `'static`.
@@ -234,6 +241,8 @@ pub(crate) fn for_lt(input: HigherRankedType) -> TokenStream {
},
);
+ let n: usize = prove_covariance.into();
+
quote!(
::kernel::types::for_lt::UnsafeForLtImpl::<
dyn for<#lifetime> ::kernel::types::for_lt::WithLt<#lifetime, Of = #ty>,
@@ -241,8 +250,16 @@ pub(crate) fn for_lt(input: HigherRankedType) -> TokenStream {
{
#(#proof)*
- 0
+ #n
}
>
)
}
+
+pub(crate) fn for_lt(input: HigherRankedType) -> TokenStream {
+ for_lt_inner(input, false)
+}
+
+pub(crate) fn covariant_for_lt(input: HigherRankedType) -> TokenStream {
+ for_lt_inner(input, true)
+}
diff --git a/rust/macros/lib.rs b/rust/macros/lib.rs
index 408a90567f7e..37a6643db76e 100644
--- a/rust/macros/lib.rs
+++ b/rust/macros/lib.rs
@@ -515,8 +515,24 @@ pub fn kunit_tests(attr: TokenStream, input: TokenStream) -> TokenStream {
///
/// [`ForLt`]: trait.ForLt.html
#[proc_macro]
-// The macro shares the name with the trait.
#[allow(non_snake_case)]
pub fn ForLt(input: TokenStream) -> TokenStream {
for_lt::for_lt(parse_macro_input!(input)).into()
}
+
+/// Obtain a type that implements [`CovariantForLt`] (and [`ForLt`]) for the given higher-ranked
+/// type.
+///
+/// Unlike [`ForLt!`], this macro additionally proves that the type is covariant over the lifetime,
+/// providing a safe [`CovariantForLt::cast_ref`] method.
+///
+/// Please refer to the documentation of the [`CovariantForLt`] trait.
+///
+/// [`CovariantForLt`]: trait.CovariantForLt.html
+/// [`CovariantForLt::cast_ref`]: trait.CovariantForLt.html#method.cast_ref
+/// [`ForLt`]: trait.ForLt.html
+#[proc_macro]
+#[allow(non_snake_case)]
+pub fn CovariantForLt(input: TokenStream) -> TokenStream {
+ for_lt::covariant_for_lt(parse_macro_input!(input)).into()
+}