2022-02-11 20:25:34 +01:00
|
|
|
// SPDX-License-Identifier: GPL-2.0
|
|
|
|
|
|
|
|
|
|
//! The `kernel` crate.
|
|
|
|
|
//!
|
|
|
|
|
//! This crate contains the kernel APIs that have been ported or wrapped for
|
|
|
|
|
//! usage by Rust code in the kernel and is shared by all of them.
|
|
|
|
|
//!
|
|
|
|
|
//! In other words, all the rest of the Rust code in the kernel (e.g. kernel
|
2025-03-03 18:10:30 +01:00
|
|
|
//! modules written in Rust) depends on [`core`] and this crate.
|
2022-02-11 20:25:34 +01:00
|
|
|
//!
|
|
|
|
|
//! If you need a kernel C API that is not ported or wrapped yet here, then
|
|
|
|
|
//! do so first instead of bypassing this crate.
|
|
|
|
|
|
|
|
|
|
#![no_std]
|
2025-03-27 22:13:02 +01:00
|
|
|
//
|
|
|
|
|
// Please see https://github.com/Rust-for-Linux/linux/issues/2 for details on
|
|
|
|
|
// the unstable features in use.
|
|
|
|
|
//
|
2026-03-14 10:06:11 +09:00
|
|
|
// Stable since Rust 1.89.0.
|
|
|
|
|
#![feature(generic_arg_infer)]
|
|
|
|
|
//
|
2025-03-27 22:13:02 +01:00
|
|
|
// Expected to become stable.
|
|
|
|
|
#![feature(arbitrary_self_types)]
|
2026-04-06 01:52:46 +02:00
|
|
|
#![feature(derive_coerce_pointee)]
|
2025-03-27 22:13:02 +01:00
|
|
|
//
|
rust: use `#[used(compiler)]` to fix build and `modpost` with Rust >= 1.89.0
Starting with Rust 1.89.0 (expected 2025-08-07), the Rust compiler fails
to build the `rusttest` target due to undefined references such as:
kernel...-cgu.0:(.text....+0x116): undefined reference to
`rust_helper_kunit_get_current_test'
Moreover, tooling like `modpost` gets confused:
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/gpu/drm/nova/nova.o
ERROR: modpost: missing MODULE_LICENSE() in drivers/gpu/nova-core/nova_core.o
The reason behind both issues is that the Rust compiler will now [1]
treat `#[used]` as `#[used(linker)]` instead of `#[used(compiler)]`
for our targets. This means that the retain section flag (`R`,
`SHF_GNU_RETAIN`) will be used and that they will be marked as `unique`
too, with different IDs. In turn, that means we end up with undefined
references that did not get discarded in `rusttest` and that multiple
`.modinfo` sections are generated, which confuse tooling like `modpost`
because they only expect one.
Thus start using `#[used(compiler)]` to keep the previous behavior
and to be explicit about what we want. Sadly, it is an unstable feature
(`used_with_arg`) [2] -- we will talk to upstream Rust about it. The good
news is that it has been available for a long time (Rust >= 1.60) [3].
The changes should also be fine for previous Rust versions, since they
behave the same way as before [4].
Alternatively, we could use `#[no_mangle]` or `#[export_name = ...]`
since those still behave like `#[used(compiler)]`, but of course it is
not really what we want to express, and it requires other changes to
avoid symbol conflicts.
Cc: David Wood <david@davidtw.co>
Cc: Wesley Wiser <wwiser@gmail.com>
Cc: stable@vger.kernel.org # Needed in 6.12.y and later (Rust is pinned in older LTSs).
Link: https://github.com/rust-lang/rust/pull/140872 [1]
Link: https://github.com/rust-lang/rust/issues/93798 [2]
Link: https://github.com/rust-lang/rust/pull/91504 [3]
Link: https://godbolt.org/z/sxzWTMfzW [4]
Reviewed-by: Alice Ryhl <aliceryhl@google.com>
Acked-by: Björn Roy Baron <bjorn3_gh@protonmail.com>
Link: https://lore.kernel.org/r/20250712160103.1244945-3-ojeda@kernel.org
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
2025-07-12 18:01:03 +02:00
|
|
|
// To be determined.
|
|
|
|
|
#![feature(used_with_arg)]
|
|
|
|
|
//
|
2026-04-06 11:58:19 +02:00
|
|
|
// `feature(file_with_nul)` is stable since Rust 1.92.0. Before Rust 1.89.0, it did not exist, so
|
2025-06-15 15:14:00 -07:00
|
|
|
// enable it conditionally.
|
|
|
|
|
#![cfg_attr(CONFIG_RUSTC_HAS_FILE_WITH_NUL, feature(file_with_nul))]
|
2022-02-11 20:25:34 +01:00
|
|
|
|
|
|
|
|
// Ensure conditional compilation based on the kernel configuration works;
|
|
|
|
|
// otherwise we may silently break things like initcall handling.
|
|
|
|
|
#[cfg(not(CONFIG_RUST))]
|
|
|
|
|
compile_error!("Missing kernel configuration for conditional compilation");
|
|
|
|
|
|
rust: add pin-init API core
This API is used to facilitate safe pinned initialization of structs. It
replaces cumbersome `unsafe` manual initialization with elegant safe macro
invocations.
Due to the size of this change it has been split into six commits:
1. This commit introducing the basic public interface: traits and
functions to represent and create initializers.
2. Adds the `#[pin_data]`, `pin_init!`, `try_pin_init!`, `init!` and
`try_init!` macros along with their internal types.
3. Adds the `InPlaceInit` trait that allows using an initializer to create
an object inside of a `Box<T>` and other smart pointers.
4. Adds the `PinnedDrop` trait and adds macro support for it in
the `#[pin_data]` macro.
5. Adds the `stack_pin_init!` macro allowing to pin-initialize a struct on
the stack.
6. Adds the `Zeroable` trait and `init::zeroed` function to initialize
types that have `0x00` in all bytes as a valid bit pattern.
--
In this section the problem that the new pin-init API solves is outlined.
This message describes the entirety of the API, not just the parts
introduced in this commit. For a more granular explanation and additional
information on pinning and this issue, view [1].
Pinning is Rust's way of enforcing the address stability of a value. When a
value gets pinned it will be impossible for safe code to move it to another
location. This is done by wrapping pointers to said object with `Pin<P>`.
This wrapper prevents safe code from creating mutable references to the
object, preventing mutable access, which is needed to move the value.
`Pin<P>` provides `unsafe` functions to circumvent this and allow
modifications regardless. It is then the programmer's responsibility to
uphold the pinning guarantee.
Many kernel data structures require a stable address, because there are
foreign pointers to them which would get invalidated by moving the
structure. Since these data structures are usually embedded in structs to
use them, this pinning property propagates to the container struct.
Resulting in most structs in both Rust and C code needing to be pinned.
So if we want to have a `mutex` field in a Rust struct, this struct also
needs to be pinned, because a `mutex` contains a `list_head`. Additionally
initializing a `list_head` requires already having the final memory
location available, because it is initialized by pointing it to itself. But
this presents another challenge in Rust: values have to be initialized at
all times. There is the `MaybeUninit<T>` wrapper type, which allows
handling uninitialized memory, but this requires using the `unsafe` raw
pointers and a casting the type to the initialized variant.
This problem gets exacerbated when considering encapsulation and the normal
safety requirements of Rust code. The fields of the Rust `Mutex<T>` should
not be accessible to normal driver code. After all if anyone can modify
the fields, there is no way to ensure the invariants of the `Mutex<T>` are
upheld. But if the fields are inaccessible, then initialization of a
`Mutex<T>` needs to be somehow achieved via a function or a macro. Because
the `Mutex<T>` must be pinned in memory, the function cannot return it by
value. It also cannot allocate a `Box` to put the `Mutex<T>` into, because
that is an unnecessary allocation and indirection which would hurt
performance.
The solution in the rust tree (e.g. this commit: [2]) that is replaced by
this API is to split this function into two parts:
1. A `new` function that returns a partially initialized `Mutex<T>`,
2. An `init` function that requires the `Mutex<T>` to be pinned and that
fully initializes the `Mutex<T>`.
Both of these functions have to be marked `unsafe`, since a call to `new`
needs to be accompanied with a call to `init`, otherwise using the
`Mutex<T>` could result in UB. And because calling `init` twice also is not
safe. While `Mutex<T>` initialization cannot fail, other structs might
also have to allocate memory, which would result in conditional successful
initialization requiring even more manual accommodation work.
Combine this with the problem of pin-projections -- the way of accessing
fields of a pinned struct -- which also have an `unsafe` API, pinned
initialization is riddled with `unsafe` resulting in very poor ergonomics.
Not only that, but also having to call two functions possibly multiple
lines apart makes it very easy to forget it outright or during refactoring.
Here is an example of the current way of initializing a struct with two
synchronization primitives (see [3] for the full example):
struct SharedState {
state_changed: CondVar,
inner: Mutex<SharedStateInner>,
}
impl SharedState {
fn try_new() -> Result<Arc<Self>> {
let mut state = Pin::from(UniqueArc::try_new(Self {
// SAFETY: `condvar_init!` is called below.
state_changed: unsafe { CondVar::new() },
// SAFETY: `mutex_init!` is called below.
inner: unsafe {
Mutex::new(SharedStateInner { token_count: 0 })
},
})?);
// SAFETY: `state_changed` is pinned when `state` is.
let pinned = unsafe {
state.as_mut().map_unchecked_mut(|s| &mut s.state_changed)
};
kernel::condvar_init!(pinned, "SharedState::state_changed");
// SAFETY: `inner` is pinned when `state` is.
let pinned = unsafe {
state.as_mut().map_unchecked_mut(|s| &mut s.inner)
};
kernel::mutex_init!(pinned, "SharedState::inner");
Ok(state.into())
}
}
The pin-init API of this patch solves this issue by providing a
comprehensive solution comprised of macros and traits. Here is the example
from above using the pin-init API:
#[pin_data]
struct SharedState {
#[pin]
state_changed: CondVar,
#[pin]
inner: Mutex<SharedStateInner>,
}
impl SharedState {
fn new() -> impl PinInit<Self> {
pin_init!(Self {
state_changed <- new_condvar!("SharedState::state_changed"),
inner <- new_mutex!(
SharedStateInner { token_count: 0 },
"SharedState::inner",
),
})
}
}
Notably the way the macro is used here requires no `unsafe` and thus comes
with the usual Rust promise of safe code not introducing any memory
violations. Additionally it is now up to the caller of `new()` to decide
the memory location of the `SharedState`. They can choose at the moment
`Arc<T>`, `Box<T>` or the stack.
--
The API has the following architecture:
1. Initializer traits `PinInit<T, E>` and `Init<T, E>` that act like
closures.
2. Macros to create these initializer traits safely.
3. Functions to allow manually writing initializers.
The initializers (an `impl PinInit<T, E>`) receive a raw pointer pointing
to uninitialized memory and their job is to fully initialize a `T` at that
location. If initialization fails, they return an error (`E`) by value.
This way of initializing cannot be safely exposed to the user, since it
relies upon these properties outside of the control of the trait:
- the memory location (slot) needs to be valid memory,
- if initialization fails, the slot should not be read from,
- the value in the slot should be pinned, so it cannot move and the memory
cannot be deallocated until the value is dropped.
This is why using an initializer is facilitated by another trait that
ensures these requirements.
These initializers can be created manually by just supplying a closure that
fulfills the same safety requirements as `PinInit<T, E>`. But this is an
`unsafe` operation. To allow safe initializer creation, the `pin_init!` is
provided along with three other variants: `try_pin_init!`, `try_init!` and
`init!`. These take a modified struct initializer as a parameter and
generate a closure that initializes the fields in sequence.
The macros take great care in upholding the safety requirements:
- A shadowed struct type is used as the return type of the closure instead
of `()`. This is to prevent early returns, as these would prevent full
initialization.
- To ensure every field is only initialized once, a normal struct
initializer is placed in unreachable code. The type checker will emit
errors if a field is missing or specified multiple times.
- When initializing a field fails, the whole initializer will fail and
automatically drop fields that have been initialized earlier.
- Only the correct initializer type is allowed for unpinned fields. You
cannot use a `impl PinInit<T, E>` to initialize a structurally not pinned
field.
To ensure the last point, an additional macro `#[pin_data]` is needed. This
macro annotates the struct itself and the user specifies structurally
pinned and not pinned fields.
Because dropping a pinned struct is also not allowed to break the pinning
invariants, another macro attribute `#[pinned_drop]` is needed. This
macro is introduced in a following commit.
These two macros also have mechanisms to ensure the overall safety of the
API. Additionally, they utilize a combined proc-macro, declarative macro
design: first a proc-macro enables the outer attribute syntax `#[...]` and
does some important pre-parsing. Notably this prepares the generics such
that the declarative macro can handle them using token trees. Then the
actual parsing of the structure and the emission of code is handled by a
declarative macro.
For pin-projections the crates `pin-project` [4] and `pin-project-lite` [5]
had been considered, but were ultimately rejected:
- `pin-project` depends on `syn` [6] which is a very big dependency, around
50k lines of code.
- `pin-project-lite` is a more reasonable 5k lines of code, but contains a
very complex declarative macro to parse generics. On top of that it
would require modification that would need to be maintained
independently.
Link: https://rust-for-linux.com/the-safe-pinned-initialization-problem [1]
Link: https://github.com/Rust-for-Linux/linux/tree/0a04dc4ddd671efb87eef54dde0fb38e9074f4be [2]
Link: https://github.com/Rust-for-Linux/linux/blob/f509ede33fc10a07eba3da14aa00302bd4b5dddd/samples/rust/rust_miscdev.rs [3]
Link: https://crates.io/crates/pin-project [4]
Link: https://crates.io/crates/pin-project-lite [5]
Link: https://crates.io/crates/syn [6]
Co-developed-by: Gary Guo <gary@garyguo.net>
Signed-off-by: Gary Guo <gary@garyguo.net>
Signed-off-by: Benno Lossin <benno.lossin@proton.me>
Reviewed-by: Alice Ryhl <aliceryhl@google.com>
Reviewed-by: Wedson Almeida Filho <wedsonaf@gmail.com>
Reviewed-by: Andreas Hindborg <a.hindborg@samsung.com>
Link: https://lore.kernel.org/r/20230408122429.1103522-7-y86-dev@protonmail.com
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
2023-04-08 12:25:45 +00:00
|
|
|
// Allow proc-macros to refer to `::kernel` inside the `kernel` crate (this crate).
|
|
|
|
|
extern crate self as kernel;
|
|
|
|
|
|
2024-09-13 22:29:23 +01:00
|
|
|
pub use ffi;
|
|
|
|
|
|
2025-06-20 16:24:25 +01:00
|
|
|
pub mod acpi;
|
2024-03-27 22:35:54 -03:00
|
|
|
pub mod alloc;
|
2025-04-14 15:18:06 +02:00
|
|
|
#[cfg(CONFIG_AUXILIARY_BUS)]
|
|
|
|
|
pub mod auxiliary;
|
rust: add bitmap API.
Provides an abstraction for C bitmap API and bitops operations.
This commit enables a Rust implementation of an Android Binder
data structure from commit 15d9da3f818c ("binder: use bitmap for faster
descriptor lookup"), which can be found in drivers/android/dbitmap.h.
It is a step towards upstreaming the Rust port of Android Binder driver.
We follow the C Bitmap API closely in naming and semantics, with
a few differences that take advantage of Rust language facilities
and idioms. The main types are `BitmapVec` for owned bitmaps and
`Bitmap` for references to C bitmaps.
* We leverage Rust type system guarantees as follows:
* all (non-atomic) mutating operations require a &mut reference which
amounts to exclusive access.
* the `BitmapVec` type implements Send. This enables transferring
ownership between threads and is needed for Binder.
* the `BitmapVec` type implements Sync, which enables passing shared
references &Bitmap between threads. Atomic operations can be
used to safely modify from multiple threads (interior
mutability), though without ordering guarantees.
* The Rust API uses `{set,clear}_bit` vs `{set,clear}_bit_atomic` as
names for clarity, which differs from the C naming convention
`set_bit` for atomic vs `__set_bit` for non-atomic.
* we include enough operations for the API to be useful. Not all
operations are exposed yet in order to avoid dead code. The missing
ones can be added later.
* We take a fine-grained approach to safety:
* Low-level bit-ops get a safe API with bounds checks. Calling with
an out-of-bounds arguments to {set,clear}_bit becomes a no-op and
get logged as errors.
* We also introduce a RUST_BITMAP_HARDENED config, which
causes invocations with out-of-bounds arguments to panic.
* methods correspond to find_* C methods tolerate out-of-bounds
since the C implementation does. Also here, out-of-bounds
arguments are logged as errors, or panic in RUST_BITMAP_HARDENED
mode.
* We add a way to "borrow" bitmaps from C in Rust, to make C bitmaps
that were allocated in C directly usable in Rust code (`Bitmap`).
* the Rust API is optimized to represent the bitmap inline if it would
fit into a pointer. This saves allocations which is
relevant in the Binder use case.
The underlying C bitmap is *not* exposed for raw access in Rust. Doing so
would permit bypassing the Rust API and lose static guarantees.
An alternative route of vendoring an existing Rust bitmap package was
considered but suboptimal overall. Reusing the C implementation is
preferable for a basic data structure like bitmaps. It enables Rust
code to be a lot more similar and predictable with respect to C code
that uses the same data structures and enables the use of code that
has been tried-and-tested in the kernel, with the same performance
characteristics whenever possible.
We use the `usize` type for sizes and indices into the bitmap,
because Rust generally always uses that type for indices and lengths
and it will be more convenient if the API accepts that type. This means
that we need to perform some casts to/from u32 and usize, since the C
headers use unsigned int instead of size_t/unsigned long for these
numbers in some places.
Adds new MAINTAINERS section BITMAP API [RUST].
Suggested-by: Alice Ryhl <aliceryhl@google.com>
Suggested-by: Yury Norov <yury.norov@gmail.com>
Signed-off-by: Burak Emir <bqe@google.com>
Reviewed-by: Alice Ryhl <aliceryhl@google.com>
Signed-off-by: Yury Norov (NVIDIA) <yury.norov@gmail.com>
2025-09-08 07:21:53 +00:00
|
|
|
pub mod bitmap;
|
2025-07-14 20:29:58 -03:00
|
|
|
pub mod bits;
|
2024-06-11 13:45:49 +02:00
|
|
|
#[cfg(CONFIG_BLOCK)]
|
|
|
|
|
pub mod block;
|
2025-05-02 18:45:36 +09:00
|
|
|
pub mod bug;
|
2024-11-23 23:28:48 +01:00
|
|
|
pub mod build_assert;
|
2025-02-20 16:45:47 +05:30
|
|
|
pub mod clk;
|
2025-05-08 10:52:09 +02:00
|
|
|
#[cfg(CONFIG_CONFIGFS_FS)]
|
|
|
|
|
pub mod configfs;
|
2025-01-08 16:18:38 +05:30
|
|
|
pub mod cpu;
|
2025-01-07 12:51:08 +05:30
|
|
|
#[cfg(CONFIG_CPU_FREQ)]
|
|
|
|
|
pub mod cpufreq;
|
2024-06-06 15:23:39 +05:30
|
|
|
pub mod cpumask;
|
2024-09-15 14:31:30 +00:00
|
|
|
pub mod cred;
|
2025-09-04 21:13:52 +00:00
|
|
|
pub mod debugfs;
|
2024-06-18 17:48:34 +02:00
|
|
|
pub mod device;
|
2024-12-19 18:04:05 +01:00
|
|
|
pub mod device_id;
|
2024-12-19 18:04:10 +01:00
|
|
|
pub mod devres;
|
2025-03-17 20:52:09 +02:00
|
|
|
pub mod dma;
|
2024-12-19 18:04:04 +01:00
|
|
|
pub mod driver;
|
2025-04-11 01:55:21 +02:00
|
|
|
#[cfg(CONFIG_DRM = "y")]
|
|
|
|
|
pub mod drm;
|
2022-02-11 20:25:34 +01:00
|
|
|
pub mod error;
|
2025-02-10 13:30:26 +01:00
|
|
|
pub mod faux;
|
2024-06-18 17:48:35 +02:00
|
|
|
#[cfg(CONFIG_RUST_FW_LOADER_ABSTRACTIONS)]
|
|
|
|
|
pub mod firmware;
|
2025-07-04 16:14:53 -04:00
|
|
|
pub mod fmt;
|
rust: file: add Rust abstraction for `struct file`
This abstraction makes it possible to manipulate the open files for a
process. The new `File` struct wraps the C `struct file`. When accessing
it using the smart pointer `ARef<File>`, the pointer will own a
reference count to the file. When accessing it as `&File`, then the
reference does not own a refcount, but the borrow checker will ensure
that the reference count does not hit zero while the `&File` is live.
Since this is intended to manipulate the open files of a process, we
introduce an `fget` constructor that corresponds to the C `fget`
method. In future patches, it will become possible to create a new fd in
a process and bind it to a `File`. Rust Binder will use these to send
fds from one process to another.
We also provide a method for accessing the file's flags. Rust Binder
will use this to access the flags of the Binder fd to check whether the
non-blocking flag is set, which affects what the Binder ioctl does.
This introduces a struct for the EBADF error type, rather than just
using the Error type directly. This has two advantages:
* `File::fget` returns a `Result<ARef<File>, BadFdError>`, which the
compiler will represent as a single pointer, with null being an error.
This is possible because the compiler understands that `BadFdError`
has only one possible value, and it also understands that the
`ARef<File>` smart pointer is guaranteed non-null.
* Additionally, we promise to users of the method that the method can
only fail with EBADF, which means that they can rely on this promise
without having to inspect its implementation.
That said, there are also two disadvantages:
* Defining additional error types involves boilerplate.
* The question mark operator will only utilize the `From` trait once,
which prevents you from using the question mark operator on
`BadFdError` in methods that return some third error type that the
kernel `Error` is convertible into. (However, it works fine in methods
that return `Error`.)
Signed-off-by: Wedson Almeida Filho <wedsonaf@gmail.com>
Co-developed-by: Daniel Xu <dxu@dxuuu.xyz>
Signed-off-by: Daniel Xu <dxu@dxuuu.xyz>
Co-developed-by: Alice Ryhl <aliceryhl@google.com>
Reviewed-by: Benno Lossin <benno.lossin@proton.me>
Signed-off-by: Alice Ryhl <aliceryhl@google.com>
Link: https://lore.kernel.org/r/20240915-alice-file-v10-3-88484f7a3dcf@google.com
Reviewed-by: Gary Guo <gary@garyguo.net>
Signed-off-by: Christian Brauner <brauner@kernel.org>
2024-09-15 14:31:29 +00:00
|
|
|
pub mod fs;
|
2026-03-20 00:57:10 -04:00
|
|
|
#[cfg(CONFIG_GPU_BUDDY = "y")]
|
|
|
|
|
pub mod gpu;
|
2025-11-16 16:21:44 +00:00
|
|
|
#[cfg(CONFIG_I2C = "y")]
|
|
|
|
|
pub mod i2c;
|
2025-09-08 07:21:55 +00:00
|
|
|
pub mod id_pool;
|
2026-01-17 07:41:25 -03:00
|
|
|
#[doc(hidden)]
|
|
|
|
|
pub mod impl_flags;
|
rust: add pin-init API core
This API is used to facilitate safe pinned initialization of structs. It
replaces cumbersome `unsafe` manual initialization with elegant safe macro
invocations.
Due to the size of this change it has been split into six commits:
1. This commit introducing the basic public interface: traits and
functions to represent and create initializers.
2. Adds the `#[pin_data]`, `pin_init!`, `try_pin_init!`, `init!` and
`try_init!` macros along with their internal types.
3. Adds the `InPlaceInit` trait that allows using an initializer to create
an object inside of a `Box<T>` and other smart pointers.
4. Adds the `PinnedDrop` trait and adds macro support for it in
the `#[pin_data]` macro.
5. Adds the `stack_pin_init!` macro allowing to pin-initialize a struct on
the stack.
6. Adds the `Zeroable` trait and `init::zeroed` function to initialize
types that have `0x00` in all bytes as a valid bit pattern.
--
In this section the problem that the new pin-init API solves is outlined.
This message describes the entirety of the API, not just the parts
introduced in this commit. For a more granular explanation and additional
information on pinning and this issue, view [1].
Pinning is Rust's way of enforcing the address stability of a value. When a
value gets pinned it will be impossible for safe code to move it to another
location. This is done by wrapping pointers to said object with `Pin<P>`.
This wrapper prevents safe code from creating mutable references to the
object, preventing mutable access, which is needed to move the value.
`Pin<P>` provides `unsafe` functions to circumvent this and allow
modifications regardless. It is then the programmer's responsibility to
uphold the pinning guarantee.
Many kernel data structures require a stable address, because there are
foreign pointers to them which would get invalidated by moving the
structure. Since these data structures are usually embedded in structs to
use them, this pinning property propagates to the container struct.
Resulting in most structs in both Rust and C code needing to be pinned.
So if we want to have a `mutex` field in a Rust struct, this struct also
needs to be pinned, because a `mutex` contains a `list_head`. Additionally
initializing a `list_head` requires already having the final memory
location available, because it is initialized by pointing it to itself. But
this presents another challenge in Rust: values have to be initialized at
all times. There is the `MaybeUninit<T>` wrapper type, which allows
handling uninitialized memory, but this requires using the `unsafe` raw
pointers and a casting the type to the initialized variant.
This problem gets exacerbated when considering encapsulation and the normal
safety requirements of Rust code. The fields of the Rust `Mutex<T>` should
not be accessible to normal driver code. After all if anyone can modify
the fields, there is no way to ensure the invariants of the `Mutex<T>` are
upheld. But if the fields are inaccessible, then initialization of a
`Mutex<T>` needs to be somehow achieved via a function or a macro. Because
the `Mutex<T>` must be pinned in memory, the function cannot return it by
value. It also cannot allocate a `Box` to put the `Mutex<T>` into, because
that is an unnecessary allocation and indirection which would hurt
performance.
The solution in the rust tree (e.g. this commit: [2]) that is replaced by
this API is to split this function into two parts:
1. A `new` function that returns a partially initialized `Mutex<T>`,
2. An `init` function that requires the `Mutex<T>` to be pinned and that
fully initializes the `Mutex<T>`.
Both of these functions have to be marked `unsafe`, since a call to `new`
needs to be accompanied with a call to `init`, otherwise using the
`Mutex<T>` could result in UB. And because calling `init` twice also is not
safe. While `Mutex<T>` initialization cannot fail, other structs might
also have to allocate memory, which would result in conditional successful
initialization requiring even more manual accommodation work.
Combine this with the problem of pin-projections -- the way of accessing
fields of a pinned struct -- which also have an `unsafe` API, pinned
initialization is riddled with `unsafe` resulting in very poor ergonomics.
Not only that, but also having to call two functions possibly multiple
lines apart makes it very easy to forget it outright or during refactoring.
Here is an example of the current way of initializing a struct with two
synchronization primitives (see [3] for the full example):
struct SharedState {
state_changed: CondVar,
inner: Mutex<SharedStateInner>,
}
impl SharedState {
fn try_new() -> Result<Arc<Self>> {
let mut state = Pin::from(UniqueArc::try_new(Self {
// SAFETY: `condvar_init!` is called below.
state_changed: unsafe { CondVar::new() },
// SAFETY: `mutex_init!` is called below.
inner: unsafe {
Mutex::new(SharedStateInner { token_count: 0 })
},
})?);
// SAFETY: `state_changed` is pinned when `state` is.
let pinned = unsafe {
state.as_mut().map_unchecked_mut(|s| &mut s.state_changed)
};
kernel::condvar_init!(pinned, "SharedState::state_changed");
// SAFETY: `inner` is pinned when `state` is.
let pinned = unsafe {
state.as_mut().map_unchecked_mut(|s| &mut s.inner)
};
kernel::mutex_init!(pinned, "SharedState::inner");
Ok(state.into())
}
}
The pin-init API of this patch solves this issue by providing a
comprehensive solution comprised of macros and traits. Here is the example
from above using the pin-init API:
#[pin_data]
struct SharedState {
#[pin]
state_changed: CondVar,
#[pin]
inner: Mutex<SharedStateInner>,
}
impl SharedState {
fn new() -> impl PinInit<Self> {
pin_init!(Self {
state_changed <- new_condvar!("SharedState::state_changed"),
inner <- new_mutex!(
SharedStateInner { token_count: 0 },
"SharedState::inner",
),
})
}
}
Notably the way the macro is used here requires no `unsafe` and thus comes
with the usual Rust promise of safe code not introducing any memory
violations. Additionally it is now up to the caller of `new()` to decide
the memory location of the `SharedState`. They can choose at the moment
`Arc<T>`, `Box<T>` or the stack.
--
The API has the following architecture:
1. Initializer traits `PinInit<T, E>` and `Init<T, E>` that act like
closures.
2. Macros to create these initializer traits safely.
3. Functions to allow manually writing initializers.
The initializers (an `impl PinInit<T, E>`) receive a raw pointer pointing
to uninitialized memory and their job is to fully initialize a `T` at that
location. If initialization fails, they return an error (`E`) by value.
This way of initializing cannot be safely exposed to the user, since it
relies upon these properties outside of the control of the trait:
- the memory location (slot) needs to be valid memory,
- if initialization fails, the slot should not be read from,
- the value in the slot should be pinned, so it cannot move and the memory
cannot be deallocated until the value is dropped.
This is why using an initializer is facilitated by another trait that
ensures these requirements.
These initializers can be created manually by just supplying a closure that
fulfills the same safety requirements as `PinInit<T, E>`. But this is an
`unsafe` operation. To allow safe initializer creation, the `pin_init!` is
provided along with three other variants: `try_pin_init!`, `try_init!` and
`init!`. These take a modified struct initializer as a parameter and
generate a closure that initializes the fields in sequence.
The macros take great care in upholding the safety requirements:
- A shadowed struct type is used as the return type of the closure instead
of `()`. This is to prevent early returns, as these would prevent full
initialization.
- To ensure every field is only initialized once, a normal struct
initializer is placed in unreachable code. The type checker will emit
errors if a field is missing or specified multiple times.
- When initializing a field fails, the whole initializer will fail and
automatically drop fields that have been initialized earlier.
- Only the correct initializer type is allowed for unpinned fields. You
cannot use a `impl PinInit<T, E>` to initialize a structurally not pinned
field.
To ensure the last point, an additional macro `#[pin_data]` is needed. This
macro annotates the struct itself and the user specifies structurally
pinned and not pinned fields.
Because dropping a pinned struct is also not allowed to break the pinning
invariants, another macro attribute `#[pinned_drop]` is needed. This
macro is introduced in a following commit.
These two macros also have mechanisms to ensure the overall safety of the
API. Additionally, they utilize a combined proc-macro, declarative macro
design: first a proc-macro enables the outer attribute syntax `#[...]` and
does some important pre-parsing. Notably this prepares the generics such
that the declarative macro can handle them using token trees. Then the
actual parsing of the structure and the emission of code is handled by a
declarative macro.
For pin-projections the crates `pin-project` [4] and `pin-project-lite` [5]
had been considered, but were ultimately rejected:
- `pin-project` depends on `syn` [6] which is a very big dependency, around
50k lines of code.
- `pin-project-lite` is a more reasonable 5k lines of code, but contains a
very complex declarative macro to parse generics. On top of that it
would require modification that would need to be maintained
independently.
Link: https://rust-for-linux.com/the-safe-pinned-initialization-problem [1]
Link: https://github.com/Rust-for-Linux/linux/tree/0a04dc4ddd671efb87eef54dde0fb38e9074f4be [2]
Link: https://github.com/Rust-for-Linux/linux/blob/f509ede33fc10a07eba3da14aa00302bd4b5dddd/samples/rust/rust_miscdev.rs [3]
Link: https://crates.io/crates/pin-project [4]
Link: https://crates.io/crates/pin-project-lite [5]
Link: https://crates.io/crates/syn [6]
Co-developed-by: Gary Guo <gary@garyguo.net>
Signed-off-by: Gary Guo <gary@garyguo.net>
Signed-off-by: Benno Lossin <benno.lossin@proton.me>
Reviewed-by: Alice Ryhl <aliceryhl@google.com>
Reviewed-by: Wedson Almeida Filho <wedsonaf@gmail.com>
Reviewed-by: Andreas Hindborg <a.hindborg@samsung.com>
Link: https://lore.kernel.org/r/20230408122429.1103522-7-y86-dev@protonmail.com
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
2023-04-08 12:25:45 +00:00
|
|
|
pub mod init;
|
2026-03-19 17:07:22 -04:00
|
|
|
pub mod interop;
|
2025-01-03 17:46:02 +01:00
|
|
|
pub mod io;
|
2023-04-03 18:33:53 +09:00
|
|
|
pub mod ioctl;
|
2026-01-18 10:08:08 +00:00
|
|
|
pub mod iommu;
|
2025-08-22 08:42:32 +00:00
|
|
|
pub mod iov;
|
2025-08-11 13:03:39 -03:00
|
|
|
pub mod irq;
|
2024-10-30 16:04:24 +00:00
|
|
|
pub mod jump_label;
|
rust: support running Rust documentation tests as KUnit ones
Rust has documentation tests: these are typically examples of
usage of any item (e.g. function, struct, module...).
They are very convenient because they are just written
alongside the documentation. For instance:
/// Sums two numbers.
///
/// ```
/// assert_eq!(mymod::f(10, 20), 30);
/// ```
pub fn f(a: i32, b: i32) -> i32 {
a + b
}
In userspace, the tests are collected and run via `rustdoc`.
Using the tool as-is would be useful already, since it allows
to compile-test most tests (thus enforcing they are kept
in sync with the code they document) and run those that do not
depend on in-kernel APIs.
However, by transforming the tests into a KUnit test suite,
they can also be run inside the kernel. Moreover, the tests
get to be compiled as other Rust kernel objects instead of
targeting userspace.
On top of that, the integration with KUnit means the Rust
support gets to reuse the existing testing facilities. For
instance, the kernel log would look like:
KTAP version 1
1..1
KTAP version 1
# Subtest: rust_doctests_kernel
1..59
# rust_doctest_kernel_build_assert_rs_0.location: rust/kernel/build_assert.rs:13
ok 1 rust_doctest_kernel_build_assert_rs_0
# rust_doctest_kernel_build_assert_rs_1.location: rust/kernel/build_assert.rs:56
ok 2 rust_doctest_kernel_build_assert_rs_1
# rust_doctest_kernel_init_rs_0.location: rust/kernel/init.rs:122
ok 3 rust_doctest_kernel_init_rs_0
...
# rust_doctest_kernel_types_rs_2.location: rust/kernel/types.rs:150
ok 59 rust_doctest_kernel_types_rs_2
# rust_doctests_kernel: pass:59 fail:0 skip:0 total:59
# Totals: pass:59 fail:0 skip:0 total:59
ok 1 rust_doctests_kernel
Therefore, add support for running Rust documentation tests
in KUnit. Some other notes about the current implementation
and support follow.
The transformation is performed by a couple scripts written
as Rust hostprogs.
Tests using the `?` operator are also supported as usual, e.g.:
/// ```
/// # use kernel::{spawn_work_item, workqueue};
/// spawn_work_item!(workqueue::system(), || pr_info!("x"))?;
/// # Ok::<(), Error>(())
/// ```
The tests are also compiled with Clippy under `CLIPPY=1`, just
like normal code, thus also benefitting from extra linting.
The names of the tests are currently automatically generated.
This allows to reduce the burden for documentation writers,
while keeping them fairly stable for bisection. This is an
improvement over the `rustdoc`-generated names, which include
the line number; but ideally we would like to get `rustdoc` to
provide the Rust item path and a number (for multiple examples
in a single documented Rust item).
In order for developers to easily see from which original line
a failed doctests came from, a KTAP diagnostic line is printed
to the log, containing the location (file and line) of the
original test (i.e. instead of the location in the generated
Rust file):
# rust_doctest_kernel_types_rs_2.location: rust/kernel/types.rs:150
This line follows the syntax for declaring test metadata in the
proposed KTAP v2 spec [1], which may be used for the proposed
KUnit test attributes API [2]. Thus hopefully this will make
migration easier later on (suggested by David [3]).
The original line in that test attribute is figured out by
providing an anchor (suggested by Boqun [4]). The original file
is found by walking the filesystem, checking directory prefixes
to reduce the amount of combinations to check, and it is only
done once per file. Ambiguities are detected and reported.
A notable difference from KUnit C tests is that the Rust tests
appear to assert using the usual `assert!` and `assert_eq!`
macros from the Rust standard library (`core`). We provide
a custom version that forwards the call to KUnit instead.
Importantly, these macros do not require passing context,
unlike the KUnit C ones (i.e. `struct kunit *`). This makes
them easier to use, and readers of the documentation do not need
to care about which testing framework is used. In addition, it
may allow us to test third-party code more easily in the future.
However, a current limitation is that KUnit does not support
assertions in other tasks. Thus we presently simply print an
error to the kernel log if an assertion actually failed. This
should be revisited to properly fail the test, perhaps saving
the context somewhere else, or letting KUnit handle it.
Link: https://lore.kernel.org/lkml/20230420205734.1288498-1-rmoar@google.com/ [1]
Link: https://lore.kernel.org/linux-kselftest/20230707210947.1208717-1-rmoar@google.com/ [2]
Link: https://lore.kernel.org/rust-for-linux/CABVgOSkOLO-8v6kdAGpmYnZUb+LKOX0CtYCo-Bge7r_2YTuXDQ@mail.gmail.com/ [3]
Link: https://lore.kernel.org/rust-for-linux/ZIps86MbJF%2FiGIzd@boqun-archlinux/ [4]
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
Reviewed-by: David Gow <davidgow@google.com>
Signed-off-by: Shuah Khan <skhan@linuxfoundation.org>
2023-07-18 07:27:51 +02:00
|
|
|
#[cfg(CONFIG_KUNIT)]
|
|
|
|
|
pub mod kunit;
|
rust: list: add ListArc
The `ListArc` type can be thought of as a special reference to a
refcounted object that owns the permission to manipulate the
`next`/`prev` pointers stored in the refcounted object. By ensuring that
each object has only one `ListArc` reference, the owner of that
reference is assured exclusive access to the `next`/`prev` pointers.
When a `ListArc` is inserted into a `List`, the `List` takes ownership
of the `ListArc` reference.
There are various strategies for ensuring that a value has only one
`ListArc` reference. The simplest is to convert a `UniqueArc` into a
`ListArc`. However, the refcounted object could also keep track of
whether a `ListArc` exists using a boolean, which could allow for the
creation of new `ListArc` references from an `Arc` reference. Whatever
strategy is used, the relevant tracking is referred to as "the tracking
inside `T`", and the `ListArcSafe` trait (and its subtraits) are used to
update the tracking when a `ListArc` is created or destroyed.
Note that we allow the case where the tracking inside `T` thinks that a
`ListArc` exists, but actually, there isn't a `ListArc`. However, we do
not allow the opposite situation where a `ListArc` exists, but the
tracking thinks it doesn't. This is because the former can at most
result in us failing to create a `ListArc` when the operation could
succeed, whereas the latter can result in the creation of two `ListArc`
references. Only the latter situation can lead to memory safety issues.
This patch introduces the `impl_list_arc_safe!` macro that allows you to
implement `ListArcSafe` for types using the strategy where a `ListArc`
can only be created from a `UniqueArc`. Other strategies are introduced
in later patches.
This is part of the linked list that Rust Binder will use for many
different things. The strategy where a `ListArc` can only be created
from a `UniqueArc` is actually sufficient for most of the objects that
Rust Binder needs to insert into linked lists. Usually, these are todo
items that are created and then immediately inserted into a queue.
The const generic ID allows objects to have several prev/next pointer
pairs so that the same object can be inserted into several different
lists. You are able to have several `ListArc` references as long as they
correspond to different pointer pairs. The ID itself is purely a
compile-time concept and will not be present in the final binary. Both
the `List` and the `ListArc` will need to agree on the ID for them to
work together. Rust Binder uses this in a few places (e.g. death
recipients) where the same object can be inserted into both generic todo
lists and some other lists for tracking the status of the object.
The ID is a const generic rather than a type parameter because the
`pair_from_unique` method needs to be able to assert that the two ids
are different. There's no easy way to assert that when using types
instead of integers.
Reviewed-by: Benno Lossin <benno.lossin@proton.me>
Signed-off-by: Alice Ryhl <aliceryhl@google.com>
Link: https://lore.kernel.org/r/20240814-linked-list-v5-2-f5f5e8075da0@google.com
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
2024-08-14 08:05:21 +00:00
|
|
|
pub mod list;
|
2025-09-02 08:35:11 +00:00
|
|
|
pub mod maple_tree;
|
2024-10-01 08:22:22 +00:00
|
|
|
pub mod miscdevice;
|
2025-04-08 09:22:38 +00:00
|
|
|
pub mod mm;
|
2025-09-24 14:39:26 +02:00
|
|
|
pub mod module_param;
|
2023-12-13 09:42:08 +09:00
|
|
|
#[cfg(CONFIG_NET)]
|
|
|
|
|
pub mod net;
|
2025-11-08 11:23:47 +09:00
|
|
|
pub mod num;
|
2024-12-19 18:04:14 +01:00
|
|
|
pub mod of;
|
2023-12-18 11:15:13 +05:30
|
|
|
#[cfg(CONFIG_PM_OPP)]
|
|
|
|
|
pub mod opp;
|
rust: add abstraction for `struct page`
Adds a new struct called `Page` that wraps a pointer to `struct page`.
This struct is assumed to hold ownership over the page, so that Rust
code can allocate and manage pages directly.
The page type has various methods for reading and writing into the page.
These methods will temporarily map the page to allow the operation. All
of these methods use a helper that takes an offset and length, performs
bounds checks, and returns a pointer to the given offset in the page.
This patch only adds support for pages of order zero, as that is all
Rust Binder needs. However, it is written to make it easy to add support
for higher-order pages in the future. To do that, you would add a const
generic parameter to `Page` that specifies the order. Most of the
methods do not need to be adjusted, as the logic for dealing with
mapping multiple pages at once can be isolated to just the
`with_pointer_into_page` method.
Rust Binder needs to manage pages directly as that is how transactions
are delivered: Each process has an mmap'd region for incoming
transactions. When an incoming transaction arrives, the Binder driver
will choose a region in the mmap, allocate and map the relevant pages
manually, and copy the incoming transaction directly into the page. This
architecture allows the driver to copy transactions directly from the
address space of one process to another, without an intermediate copy
to a kernel buffer.
This code is based on Wedson's page abstractions from the old rust
branch, but it has been modified by Alice by removing the incomplete
support for higher-order pages, by introducing the `with_*` helpers
to consolidate the bounds checking logic into a single place, and
various other changes.
Co-developed-by: Wedson Almeida Filho <wedsonaf@gmail.com>
Signed-off-by: Wedson Almeida Filho <wedsonaf@gmail.com>
Reviewed-by: Andreas Hindborg <a.hindborg@samsung.com>
Reviewed-by: Trevor Gross <tmgross@umich.edu>
Reviewed-by: Benno Lossin <benno.lossin@proton.me>
Reviewed-by: Boqun Feng <boqun.feng@gmail.com>
Signed-off-by: Alice Ryhl <aliceryhl@google.com>
Link: https://lore.kernel.org/r/20240528-alice-mm-v7-4-78222c31b8f4@google.com
[ Fixed typos and added a few intra-doc links. - Miguel ]
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
2024-05-28 14:58:05 +00:00
|
|
|
pub mod page;
|
2025-01-03 17:46:01 +01:00
|
|
|
#[cfg(CONFIG_PCI)]
|
|
|
|
|
pub mod pci;
|
2024-10-02 13:38:10 +02:00
|
|
|
pub mod pid_namespace;
|
2024-12-19 18:04:16 +01:00
|
|
|
pub mod platform;
|
2022-02-11 20:25:34 +01:00
|
|
|
pub mod prelude;
|
|
|
|
|
pub mod print;
|
2025-08-21 09:20:54 +09:00
|
|
|
pub mod processor;
|
2025-09-08 22:25:54 +09:00
|
|
|
pub mod ptr;
|
2025-10-29 19:25:02 +01:00
|
|
|
#[cfg(CONFIG_RUST_PWM_ABSTRACTIONS)]
|
|
|
|
|
pub mod pwm;
|
2024-08-22 16:37:53 +00:00
|
|
|
pub mod rbtree;
|
2025-07-14 15:52:04 -03:00
|
|
|
pub mod regulator;
|
2024-12-19 18:04:08 +01:00
|
|
|
pub mod revocable;
|
2025-10-08 03:20:28 +05:30
|
|
|
pub mod safety;
|
2025-08-28 15:32:16 +02:00
|
|
|
pub mod scatterlist;
|
2024-09-15 14:31:31 +00:00
|
|
|
pub mod security;
|
rust: add seqfile abstraction
This adds a simple seq file abstraction that lets you print to a seq
file using ordinary Rust printing syntax.
An example user from Rust Binder:
pub(crate) fn full_debug_print(
&self,
m: &SeqFile,
owner_inner: &mut ProcessInner,
) -> Result<()> {
let prio = self.node_prio();
let inner = self.inner.access_mut(owner_inner);
seq_print!(
m,
" node {}: u{:016x} c{:016x} pri {}:{} hs {} hw {} cs {} cw {}",
self.debug_id,
self.ptr,
self.cookie,
prio.sched_policy,
prio.prio,
inner.strong.has_count,
inner.weak.has_count,
inner.strong.count,
inner.weak.count,
);
if !inner.refs.is_empty() {
seq_print!(m, " proc");
for node_ref in &inner.refs {
seq_print!(m, " {}", node_ref.process.task.pid());
}
}
seq_print!(m, "\n");
for t in &inner.oneway_todo {
t.debug_print_inner(m, " pending async transaction ");
}
Ok(())
}
The `SeqFile` type is marked not thread safe so that `call_printf` can
be a `&self` method. The alternative is to use `self: Pin<&mut Self>`
which is inconvenient, or to have `SeqFile` wrap a pointer instead of
wrapping the C struct directly.
Signed-off-by: Alice Ryhl <aliceryhl@google.com>
Link: https://lore.kernel.org/r/20241001-seqfile-v1-1-dfcd0fc21e96@google.com
Signed-off-by: Christian Brauner <brauner@kernel.org>
2024-10-01 09:07:02 +00:00
|
|
|
pub mod seq_file;
|
2024-09-26 14:47:51 +02:00
|
|
|
pub mod sizes;
|
2025-12-26 20:17:07 +00:00
|
|
|
#[cfg(CONFIG_SOC_BUS)]
|
|
|
|
|
pub mod soc;
|
2022-11-10 17:41:35 +01:00
|
|
|
#[doc(hidden)]
|
|
|
|
|
pub mod std_vendor;
|
2022-02-11 20:25:34 +01:00
|
|
|
pub mod str;
|
2022-12-28 06:03:40 +00:00
|
|
|
pub mod sync;
|
2023-04-11 02:45:39 -03:00
|
|
|
pub mod task;
|
2024-01-08 14:49:58 +00:00
|
|
|
pub mod time;
|
2024-10-30 16:04:25 +00:00
|
|
|
pub mod tracepoint;
|
2024-09-18 18:51:14 -04:00
|
|
|
pub mod transmute;
|
2022-11-10 17:41:39 +01:00
|
|
|
pub mod types;
|
rust: uaccess: add userspace pointers
A pointer to an area in userspace memory, which can be either read-only
or read-write.
All methods on this struct are safe: attempting to read or write on bad
addresses (either out of the bound of the slice or unmapped addresses)
will return `EFAULT`. Concurrent access, *including data races to/from
userspace memory*, is permitted, because fundamentally another userspace
thread/process could always be modifying memory at the same time (in the
same way that userspace Rust's `std::io` permits data races with the
contents of files on disk). In the presence of a race, the exact byte
values read/written are unspecified but the operation is well-defined.
Kernelspace code should validate its copy of data after completing a
read, and not expect that multiple reads of the same address will return
the same value.
These APIs are designed to make it difficult to accidentally write
TOCTOU bugs. Every time you read from a memory location, the pointer is
advanced by the length so that you cannot use that reader to read the
same memory location twice. Preventing double-fetches avoids TOCTOU
bugs. This is accomplished by taking `self` by value to prevent
obtaining multiple readers on a given `UserSlice`, and the readers only
permitting forward reads. If double-fetching a memory location is
necessary for some reason, then that is done by creating multiple
readers to the same memory location.
Constructing a `UserSlice` performs no checks on the provided address
and length, it can safely be constructed inside a kernel thread with no
current userspace process. Reads and writes wrap the kernel APIs
`copy_from_user` and `copy_to_user`, which check the memory map of the
current process and enforce that the address range is within the user
range (no additional calls to `access_ok` are needed).
This code is based on something that was originally written by Wedson on
the old rust branch. It was modified by Alice by removing the
`IoBufferReader` and `IoBufferWriter` traits, and various other changes.
Signed-off-by: Wedson Almeida Filho <wedsonaf@gmail.com>
Reviewed-by: Benno Lossin <benno.lossin@proton.me>
Reviewed-by: Trevor Gross <tmgross@umich.edu>
Reviewed-by: Boqun Feng <boqun.feng@gmail.com>
Co-developed-by: Alice Ryhl <aliceryhl@google.com>
Signed-off-by: Alice Ryhl <aliceryhl@google.com>
Link: https://lore.kernel.org/r/20240528-alice-mm-v7-1-78222c31b8f4@google.com
[ Wrapped docs to 100 and added a few intra-doc links. - Miguel ]
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
2024-05-28 14:58:02 +00:00
|
|
|
pub mod uaccess;
|
2025-10-08 14:10:28 +02:00
|
|
|
#[cfg(CONFIG_USB = "y")]
|
|
|
|
|
pub mod usb;
|
2023-08-28 10:48:02 +00:00
|
|
|
pub mod workqueue;
|
2025-04-23 09:54:38 -04:00
|
|
|
pub mod xarray;
|
2022-02-11 20:25:34 +01:00
|
|
|
|
|
|
|
|
#[doc(hidden)]
|
|
|
|
|
pub use bindings;
|
|
|
|
|
pub use macros;
|
2023-04-03 18:33:52 +09:00
|
|
|
pub use uapi;
|
2022-02-11 20:25:34 +01:00
|
|
|
|
|
|
|
|
/// Prefix to appear before log messages printed from within the `kernel` crate.
|
|
|
|
|
const __LOG_PREFIX: &[u8] = b"rust_kernel\0";
|
|
|
|
|
|
|
|
|
|
/// The top level entrypoint to implementing a kernel module.
|
|
|
|
|
///
|
|
|
|
|
/// For any teardown or cleanup operations, your type may implement [`Drop`].
|
2024-03-28 16:54:54 -03:00
|
|
|
pub trait Module: Sized + Sync + Send {
|
2022-02-11 20:25:34 +01:00
|
|
|
/// Called at module initialization time.
|
|
|
|
|
///
|
|
|
|
|
/// Use this method to perform whatever setup or registration your module
|
|
|
|
|
/// should do.
|
|
|
|
|
///
|
|
|
|
|
/// Equivalent to the `module_init` macro in the C API.
|
|
|
|
|
fn init(module: &'static ThisModule) -> error::Result<Self>;
|
|
|
|
|
}
|
|
|
|
|
|
2024-10-22 23:31:39 +02:00
|
|
|
/// A module that is pinned and initialised in-place.
|
|
|
|
|
pub trait InPlaceModule: Sync + Send {
|
|
|
|
|
/// Creates an initialiser for the module.
|
|
|
|
|
///
|
|
|
|
|
/// It is called when the module is loaded.
|
2025-03-08 11:05:09 +00:00
|
|
|
fn init(module: &'static ThisModule) -> impl pin_init::PinInit<Self, error::Error>;
|
2024-10-22 23:31:39 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
impl<T: Module> InPlaceModule for T {
|
2025-03-08 11:05:09 +00:00
|
|
|
fn init(module: &'static ThisModule) -> impl pin_init::PinInit<Self, error::Error> {
|
2024-10-22 23:31:39 +02:00
|
|
|
let initer = move |slot: *mut Self| {
|
|
|
|
|
let m = <Self as Module>::init(module)?;
|
|
|
|
|
|
|
|
|
|
// SAFETY: `slot` is valid for write per the contract with `pin_init_from_closure`.
|
|
|
|
|
unsafe { slot.write(m) };
|
|
|
|
|
Ok(())
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
// SAFETY: On success, `initer` always fully initialises an instance of `Self`.
|
2025-03-08 11:05:09 +00:00
|
|
|
unsafe { pin_init::pin_init_from_closure(initer) }
|
2024-10-22 23:31:39 +02:00
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2024-12-19 18:04:03 +01:00
|
|
|
/// Metadata attached to a [`Module`] or [`InPlaceModule`].
|
|
|
|
|
pub trait ModuleMetadata {
|
|
|
|
|
/// The name of the module as specified in the `module!` macro.
|
|
|
|
|
const NAME: &'static crate::str::CStr;
|
|
|
|
|
}
|
|
|
|
|
|
2022-02-11 20:25:34 +01:00
|
|
|
/// Equivalent to `THIS_MODULE` in the C API.
|
|
|
|
|
///
|
2024-10-21 11:58:47 +09:00
|
|
|
/// C header: [`include/linux/init.h`](srctree/include/linux/init.h)
|
2022-02-11 20:25:34 +01:00
|
|
|
pub struct ThisModule(*mut bindings::module);
|
|
|
|
|
|
|
|
|
|
// SAFETY: `THIS_MODULE` may be used from all threads within a module.
|
|
|
|
|
unsafe impl Sync for ThisModule {}
|
|
|
|
|
|
|
|
|
|
impl ThisModule {
|
|
|
|
|
/// Creates a [`ThisModule`] given the `THIS_MODULE` pointer.
|
|
|
|
|
///
|
|
|
|
|
/// # Safety
|
|
|
|
|
///
|
|
|
|
|
/// The pointer must be equal to the right `THIS_MODULE`.
|
|
|
|
|
pub const unsafe fn from_ptr(ptr: *mut bindings::module) -> ThisModule {
|
|
|
|
|
ThisModule(ptr)
|
|
|
|
|
}
|
2024-02-26 09:44:02 +00:00
|
|
|
|
|
|
|
|
/// Access the raw pointer for this module.
|
|
|
|
|
///
|
|
|
|
|
/// It is up to the user to use it correctly.
|
|
|
|
|
pub const fn as_ptr(&self) -> *mut bindings::module {
|
|
|
|
|
self.0
|
|
|
|
|
}
|
2022-02-11 20:25:34 +01:00
|
|
|
}
|
|
|
|
|
|
2025-07-26 20:07:50 +02:00
|
|
|
#[cfg(not(testlib))]
|
2022-02-11 20:25:34 +01:00
|
|
|
#[panic_handler]
|
|
|
|
|
fn panic(info: &core::panic::PanicInfo<'_>) -> ! {
|
|
|
|
|
pr_emerg!("{}\n", info);
|
|
|
|
|
// SAFETY: FFI call.
|
|
|
|
|
unsafe { bindings::BUG() };
|
|
|
|
|
}
|
2024-02-19 11:48:08 +00:00
|
|
|
|
|
|
|
|
/// Produces a pointer to an object from a pointer to one of its fields.
|
|
|
|
|
///
|
2025-06-24 15:27:56 +00:00
|
|
|
/// If you encounter a type mismatch due to the [`Opaque`] type, then use [`Opaque::cast_into`] or
|
2025-06-24 15:27:55 +00:00
|
|
|
/// [`Opaque::cast_from`] to resolve the mismatch.
|
|
|
|
|
///
|
|
|
|
|
/// [`Opaque`]: crate::types::Opaque
|
2025-06-24 15:27:56 +00:00
|
|
|
/// [`Opaque::cast_into`]: crate::types::Opaque::cast_into
|
2025-06-24 15:27:55 +00:00
|
|
|
/// [`Opaque::cast_from`]: crate::types::Opaque::cast_from
|
|
|
|
|
///
|
2024-02-19 11:48:08 +00:00
|
|
|
/// # Safety
|
|
|
|
|
///
|
|
|
|
|
/// The pointer passed to this macro, and the pointer returned by this macro, must both be in
|
|
|
|
|
/// bounds of the same allocation.
|
|
|
|
|
///
|
|
|
|
|
/// # Examples
|
|
|
|
|
///
|
|
|
|
|
/// ```
|
|
|
|
|
/// # use kernel::container_of;
|
|
|
|
|
/// struct Test {
|
|
|
|
|
/// a: u64,
|
|
|
|
|
/// b: u32,
|
|
|
|
|
/// }
|
|
|
|
|
///
|
|
|
|
|
/// let test = Test { a: 10, b: 20 };
|
2025-04-09 10:43:16 -04:00
|
|
|
/// let b_ptr: *const _ = &test.b;
|
2024-02-19 11:48:08 +00:00
|
|
|
/// // SAFETY: The pointer points at the `b` field of a `Test`, so the resulting pointer will be
|
|
|
|
|
/// // in-bounds of the same allocation as `b_ptr`.
|
|
|
|
|
/// let test_alias = unsafe { container_of!(b_ptr, Test, b) };
|
|
|
|
|
/// assert!(core::ptr::eq(&test, test_alias));
|
|
|
|
|
/// ```
|
|
|
|
|
#[macro_export]
|
|
|
|
|
macro_rules! container_of {
|
rust: check type of `$ptr` in `container_of!`
Add a compile-time check that `*$ptr` is of the type of `$type->$($f)*`.
Rename those placeholders for clarity.
Given the incorrect usage:
> diff --git a/rust/kernel/rbtree.rs b/rust/kernel/rbtree.rs
> index 8d978c896747..6a7089149878 100644
> --- a/rust/kernel/rbtree.rs
> +++ b/rust/kernel/rbtree.rs
> @@ -329,7 +329,7 @@ fn raw_entry(&mut self, key: &K) -> RawEntry<'_, K, V> {
> while !(*child_field_of_parent).is_null() {
> let curr = *child_field_of_parent;
> // SAFETY: All links fields we create are in a `Node<K, V>`.
> - let node = unsafe { container_of!(curr, Node<K, V>, links) };
> + let node = unsafe { container_of!(curr, Node<K, V>, key) };
>
> // SAFETY: `node` is a non-null node so it is valid by the type invariants.
> match key.cmp(unsafe { &(*node).key }) {
this patch produces the compilation error:
> error[E0308]: mismatched types
> --> rust/kernel/lib.rs:220:45
> |
> 220 | $crate::assert_same_type(field_ptr, (&raw const (*container_ptr).$($fields)*).cast_mut());
> | ------------------------ --------- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `*mut rb_node`, found `*mut K`
> | | |
> | | expected all arguments to be this `*mut bindings::rb_node` type because they need to match the type of this parameter
> | arguments to this function are incorrect
> |
> ::: rust/kernel/rbtree.rs:270:6
> |
> 270 | impl<K, V> RBTree<K, V>
> | - found this type parameter
> ...
> 332 | let node = unsafe { container_of!(curr, Node<K, V>, key) };
> | ------------------------------------ in this macro invocation
> |
> = note: expected raw pointer `*mut bindings::rb_node`
> found raw pointer `*mut K`
> note: function defined here
> --> rust/kernel/lib.rs:227:8
> |
> 227 | pub fn assert_same_type<T>(_: T, _: T) {}
> | ^^^^^^^^^^^^^^^^ - ---- ---- this parameter needs to match the `*mut bindings::rb_node` type of parameter #1
> | | |
> | | parameter #2 needs to match the `*mut bindings::rb_node` type of this parameter
> | parameter #1 and parameter #2 both reference this parameter `T`
> = note: this error originates in the macro `container_of` (in Nightly builds, run with -Z macro-backtrace for more info)
[ We decided to go with a variation of v1 [1] that became v4, since it
seems like the obvious approach, the error messages seem good enough
and the debug performance should be fine, given the kernel is always
built with -O2.
In the future, we may want to make the helper non-hidden, with
proper documentation, for others to use.
[1] https://lore.kernel.org/rust-for-linux/CANiq72kQWNfSV0KK6qs6oJt+aGdgY=hXg=wJcmK3zYcokY1LNw@mail.gmail.com/
- Miguel ]
Suggested-by: Alice Ryhl <aliceryhl@google.com>
Link: https://lore.kernel.org/all/CAH5fLgh6gmqGBhPMi2SKn7mCmMWfOSiS0WP5wBuGPYh9ZTAiww@mail.gmail.com/
Signed-off-by: Tamir Duberstein <tamird@gmail.com>
Reviewed-by: Benno Lossin <lossin@kernel.org>
Link: https://lore.kernel.org/r/20250529-b4-container-of-type-check-v4-1-bf3a7ad73cec@gmail.com
[ Added intra-doc link. - Miguel ]
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
2025-05-29 09:11:33 -04:00
|
|
|
($field_ptr:expr, $Container:ty, $($fields:tt)*) => {{
|
|
|
|
|
let offset: usize = ::core::mem::offset_of!($Container, $($fields)*);
|
|
|
|
|
let field_ptr = $field_ptr;
|
|
|
|
|
let container_ptr = field_ptr.byte_sub(offset).cast::<$Container>();
|
|
|
|
|
$crate::assert_same_type(field_ptr, (&raw const (*container_ptr).$($fields)*).cast_mut());
|
|
|
|
|
container_ptr
|
2024-02-19 11:48:08 +00:00
|
|
|
}}
|
|
|
|
|
}
|
2024-10-30 16:04:28 +00:00
|
|
|
|
rust: check type of `$ptr` in `container_of!`
Add a compile-time check that `*$ptr` is of the type of `$type->$($f)*`.
Rename those placeholders for clarity.
Given the incorrect usage:
> diff --git a/rust/kernel/rbtree.rs b/rust/kernel/rbtree.rs
> index 8d978c896747..6a7089149878 100644
> --- a/rust/kernel/rbtree.rs
> +++ b/rust/kernel/rbtree.rs
> @@ -329,7 +329,7 @@ fn raw_entry(&mut self, key: &K) -> RawEntry<'_, K, V> {
> while !(*child_field_of_parent).is_null() {
> let curr = *child_field_of_parent;
> // SAFETY: All links fields we create are in a `Node<K, V>`.
> - let node = unsafe { container_of!(curr, Node<K, V>, links) };
> + let node = unsafe { container_of!(curr, Node<K, V>, key) };
>
> // SAFETY: `node` is a non-null node so it is valid by the type invariants.
> match key.cmp(unsafe { &(*node).key }) {
this patch produces the compilation error:
> error[E0308]: mismatched types
> --> rust/kernel/lib.rs:220:45
> |
> 220 | $crate::assert_same_type(field_ptr, (&raw const (*container_ptr).$($fields)*).cast_mut());
> | ------------------------ --------- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `*mut rb_node`, found `*mut K`
> | | |
> | | expected all arguments to be this `*mut bindings::rb_node` type because they need to match the type of this parameter
> | arguments to this function are incorrect
> |
> ::: rust/kernel/rbtree.rs:270:6
> |
> 270 | impl<K, V> RBTree<K, V>
> | - found this type parameter
> ...
> 332 | let node = unsafe { container_of!(curr, Node<K, V>, key) };
> | ------------------------------------ in this macro invocation
> |
> = note: expected raw pointer `*mut bindings::rb_node`
> found raw pointer `*mut K`
> note: function defined here
> --> rust/kernel/lib.rs:227:8
> |
> 227 | pub fn assert_same_type<T>(_: T, _: T) {}
> | ^^^^^^^^^^^^^^^^ - ---- ---- this parameter needs to match the `*mut bindings::rb_node` type of parameter #1
> | | |
> | | parameter #2 needs to match the `*mut bindings::rb_node` type of this parameter
> | parameter #1 and parameter #2 both reference this parameter `T`
> = note: this error originates in the macro `container_of` (in Nightly builds, run with -Z macro-backtrace for more info)
[ We decided to go with a variation of v1 [1] that became v4, since it
seems like the obvious approach, the error messages seem good enough
and the debug performance should be fine, given the kernel is always
built with -O2.
In the future, we may want to make the helper non-hidden, with
proper documentation, for others to use.
[1] https://lore.kernel.org/rust-for-linux/CANiq72kQWNfSV0KK6qs6oJt+aGdgY=hXg=wJcmK3zYcokY1LNw@mail.gmail.com/
- Miguel ]
Suggested-by: Alice Ryhl <aliceryhl@google.com>
Link: https://lore.kernel.org/all/CAH5fLgh6gmqGBhPMi2SKn7mCmMWfOSiS0WP5wBuGPYh9ZTAiww@mail.gmail.com/
Signed-off-by: Tamir Duberstein <tamird@gmail.com>
Reviewed-by: Benno Lossin <lossin@kernel.org>
Link: https://lore.kernel.org/r/20250529-b4-container-of-type-check-v4-1-bf3a7ad73cec@gmail.com
[ Added intra-doc link. - Miguel ]
Signed-off-by: Miguel Ojeda <ojeda@kernel.org>
2025-05-29 09:11:33 -04:00
|
|
|
/// Helper for [`container_of!`].
|
|
|
|
|
#[doc(hidden)]
|
|
|
|
|
pub fn assert_same_type<T>(_: T, _: T) {}
|
|
|
|
|
|
2024-10-30 16:04:28 +00:00
|
|
|
/// Helper for `.rs.S` files.
|
|
|
|
|
#[doc(hidden)]
|
|
|
|
|
#[macro_export]
|
|
|
|
|
macro_rules! concat_literals {
|
|
|
|
|
($( $asm:literal )* ) => {
|
|
|
|
|
::core::concat!($($asm),*)
|
|
|
|
|
};
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// Wrapper around `asm!` configured for use in the kernel.
|
|
|
|
|
///
|
|
|
|
|
/// Uses a semicolon to avoid parsing ambiguities, even though this does not match native `asm!`
|
|
|
|
|
/// syntax.
|
|
|
|
|
// For x86, `asm!` uses intel syntax by default, but we want to use at&t syntax in the kernel.
|
|
|
|
|
#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]
|
|
|
|
|
#[macro_export]
|
|
|
|
|
macro_rules! asm {
|
|
|
|
|
($($asm:expr),* ; $($rest:tt)*) => {
|
|
|
|
|
::core::arch::asm!( $($asm)*, options(att_syntax), $($rest)* )
|
|
|
|
|
};
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/// Wrapper around `asm!` configured for use in the kernel.
|
|
|
|
|
///
|
|
|
|
|
/// Uses a semicolon to avoid parsing ambiguities, even though this does not match native `asm!`
|
|
|
|
|
/// syntax.
|
|
|
|
|
// For non-x86 arches we just pass through to `asm!`.
|
|
|
|
|
#[cfg(not(any(target_arch = "x86", target_arch = "x86_64")))]
|
|
|
|
|
#[macro_export]
|
|
|
|
|
macro_rules! asm {
|
|
|
|
|
($($asm:expr),* ; $($rest:tt)*) => {
|
|
|
|
|
::core::arch::asm!( $($asm)*, $($rest)* )
|
|
|
|
|
};
|
|
|
|
|
}
|
2025-06-15 15:14:00 -07:00
|
|
|
|
|
|
|
|
/// Gets the C string file name of a [`Location`].
|
|
|
|
|
///
|
2025-08-27 13:12:16 +00:00
|
|
|
/// If `Location::file_as_c_str()` is not available, returns a string that warns about it.
|
2025-06-15 15:14:00 -07:00
|
|
|
///
|
|
|
|
|
/// [`Location`]: core::panic::Location
|
|
|
|
|
///
|
|
|
|
|
/// # Examples
|
|
|
|
|
///
|
|
|
|
|
/// ```
|
|
|
|
|
/// # use kernel::file_from_location;
|
|
|
|
|
///
|
|
|
|
|
/// #[track_caller]
|
|
|
|
|
/// fn foo() {
|
|
|
|
|
/// let caller = core::panic::Location::caller();
|
|
|
|
|
///
|
|
|
|
|
/// // Output:
|
2025-08-27 13:12:16 +00:00
|
|
|
/// // - A path like "rust/kernel/example.rs" if `file_as_c_str()` is available.
|
|
|
|
|
/// // - "<Location::file_as_c_str() not supported>" otherwise.
|
2025-06-15 15:14:00 -07:00
|
|
|
/// let caller_file = file_from_location(caller);
|
|
|
|
|
///
|
|
|
|
|
/// // Prints out the message with caller's file name.
|
|
|
|
|
/// pr_info!("foo() called in file {caller_file:?}\n");
|
|
|
|
|
///
|
|
|
|
|
/// # if cfg!(CONFIG_RUSTC_HAS_FILE_WITH_NUL) {
|
|
|
|
|
/// # assert_eq!(Ok(caller.file()), caller_file.to_str());
|
|
|
|
|
/// # }
|
|
|
|
|
/// }
|
|
|
|
|
///
|
|
|
|
|
/// # foo();
|
|
|
|
|
/// ```
|
|
|
|
|
#[inline]
|
|
|
|
|
pub fn file_from_location<'a>(loc: &'a core::panic::Location<'a>) -> &'a core::ffi::CStr {
|
2025-08-27 13:12:16 +00:00
|
|
|
#[cfg(CONFIG_RUSTC_HAS_FILE_AS_C_STR)]
|
|
|
|
|
{
|
|
|
|
|
loc.file_as_c_str()
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#[cfg(all(CONFIG_RUSTC_HAS_FILE_WITH_NUL, not(CONFIG_RUSTC_HAS_FILE_AS_C_STR)))]
|
2025-06-15 15:14:00 -07:00
|
|
|
{
|
|
|
|
|
loc.file_with_nul()
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#[cfg(not(CONFIG_RUSTC_HAS_FILE_WITH_NUL))]
|
|
|
|
|
{
|
|
|
|
|
let _ = loc;
|
2025-08-27 13:12:16 +00:00
|
|
|
c"<Location::file_as_c_str() not supported>"
|
2025-06-15 15:14:00 -07:00
|
|
|
}
|
|
|
|
|
}
|